{"page":"<link rel=\"stylesheet\" href=\"https://lessonplanet.com/assets/packs/css/resources-572d6a42.css\" />\n<link rel=\"stylesheet\" href=\"https://lessonplanet.com/assets/packs/css/lp_boclips_stylesheets-f4d0de30.css\" media=\"all\" />\n<div data-title='Web Scraping Tutorial with Scrapy and Python for Beginners - Scrape a Website with Loading Screen' data-url='/boclips/videos/63eb89cee4dd8246199748fd' data-video-url='/boclips/videos/63eb89cee4dd8246199748fd' id='bo_player_modal'>\n<div class='boclips-resource-page modal-dialog panel-container'>\n<div class='react-notifications-root'></div>\n<div class='rp-header'>\n<div class='rp-type'>\n<i aria-hidden='true' class='fai fa-regular fa-circle-play'></i>\nVideo\n</div>\n<h1 class='rp-title' id='video-title'>\nWeb Scraping Tutorial with Scrapy and Python for Beginners - Scrape a Website with Loading Screen\n</h1>\n<div class='rp-actions'>\n<div class='mr-1'>\n<a class=\"btn btn-success\" data-posthog-event=\"Signup: LP Signup Activity\" data-posthog-location=\"body_link_boclips\" data-remote=\"true\" href=\"/subscription/new\"><span><span>Get Free Access</span><span class=\"\"> for 10 Days</span><span>!</span></span></a>\n</div>\n</div>\n</div>\n<div class='rp-body'>\n<div class='rp-info'>\n<div aria-label='Hide resource details' class='rp-hide-info' role='button' tabindex='0'>&times;</div>\n<i aria-label='Expand resource details' class='rp-expand-info fai fa-solid fa-up-right-and-down-left-from-center' role='button' tabindex='0'></i>\n<i aria-label='Compress resource details' class='rp-compress-info fai fa-solid fa-down-left-and-up-right-to-center' role='button' tabindex='0'></i>\n<div class='rp-rating'>\n<span class='resource-pool'>\n<span class='pool-label'>Publisher:</span>\n<span class='pool-name'>\n<span class='text'><a data-publisher-id=\"30356011\" href=\"/search?publisher_ids%5B%5D=30356011\">Curated Video</a></span>\n</span>\n</span>\n</div>\n<div class='rp-description'>\n<span class='short-description'>In this lesson, we will focus on waiting for a loading screen to appear and then extract the table data instead of getting an error message. We will be using Scrapy's Playwright library to dynamically render the website and extract the...</span>\n<span class='full-description hide'>In this lesson, we will focus on waiting for a loading screen to appear and then extract the table data instead of getting an error message. We will be using Scrapy's Playwright library to dynamically render the website and extract the desired information. Our step-by-step tutorial will guide you through the process of using selectors and identifying the table element to extract its content. </span>\n</div>\n<div class='action-container flex justify-between'>\n<button aria-expanded='false' aria-label='Read more description' class='rp-full-description' type='button'>\n<i class='fai fa-solid fa-align-left'></i>\n<span id='read_more'>Read More</span>\n</button>\n<div class='rp-report'>\n</div>\n</div>\n<div aria-labelledby='resource-details-heading' class='rp-info-section'>\n<h2 class='title' id='resource-details-heading'>Resource Details</h2>\n<div class='rp-resource-details clearfix'>\n<div class='detail'>\n<dl>\n<dt>Curator Rating</dt>\n<dd><span class=\"star-rating\" aria-label=\"4.0 out of 5 stars\" role=\"img\"><i class=\"fa-solid fa-star text-action\" aria-hidden=\"true\"></i><i class=\"fa-solid fa-star text-action\" aria-hidden=\"true\"></i><i class=\"fa-solid fa-star text-action\" aria-hidden=\"true\"></i><i class=\"fa-solid fa-star text-action\" aria-hidden=\"true\"></i><i class=\"fa-regular fa-star text-action\" aria-hidden=\"true\"></i></span></dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt class=\"educator-rating-title\">Educator Rating</dt><dd><div class=\"educator-rating-details\" data-path=\"/educator_ratings/rrp_data?resourceable_id=41000&amp;resourceable_type=Boclips%3A%3AVideoMetadata\"><span class=\"not-yet-rated\">Not yet Rated</span></div></dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Media Length</dt>\n<dd>5:39</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Grade</dt><dd title=\"Grade\">Higher Ed</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Subjects</dt><dd><span><a href=\"/search?grade_ids%5B%5D=259&amp;search_tab_id=1&amp;subject_ids%5B%5D=378077\">STEM</a></span></dd><dd class=\"text-muted\"><i class=\"fa-solid fa-lock mr5\"></i>3 more...</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Media Type</dt><dd><span><a href=\"/search?grade_ids%5B%5D=259&amp;search_tab_id=2&amp;type_ids%5B%5D=4543647\">Instructional Videos</a></span></dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Source:</dt>\n<div class='preview-source' data-animation='true' data-boundary='.rp-info' data-container='.rp-resource-details' data-html='false' data-title='Instructional videos for technology and computing.' data-trigger='hover focus'>\n<span>Packt -  Programming</span>\n<i aria-hidden='true' class='fa-solid fa-circle-info channel-tooltip-icon' id='channel-tooltip'></i>\n</div>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Date</dt>\n<dd>2023</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<i aria-hidden='true' class='fai fa-solid fa-language'></i>\n<dt>Language</dt><dd>English</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Audiences</dt><dd><span><a href=\"/search?audience_ids%5B%5D=371079&amp;grade_ids%5B%5D=259&amp;search_tab_id=1\">For Teacher Use</a></span></dd><dd class=\"text-muted\"><i class=\"fa-solid fa-lock mr5\"></i>2 more...</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Usage Permissions</dt><dd>Fine Print: Educational Use</dd>\n</dl>\n</div>\n</div>\n</div>\n<div aria-labelledby='additional-materials-heading' class='rp-info-section'>\n<h2 class='title' id='additional-materials-heading'>Additional Materials</h2>\n<div class='additional-material'>\n<i aria-hidden='true' class='fai fa-solid fa-lock'></i>\n<a class=\"text-muted\" title=\"Video Transcript\" data-html=\"true\" data-placement=\"bottom\" data-trigger=\"click\" data-content=\"<div class=&quot;text-center py-2&quot;><a class=&quot;bold&quot; href=&quot;/auth/users/sign_in&quot;>Sign in</a> or <a class=&quot;bold text-danger&quot; data-posthog-event=&quot;Signup: LP Signup Activity&quot; data-posthog-location=&quot;body_link_boclips&quot; data-remote=&quot;true&quot; href=&quot;/subscription/new&quot;>Join Now</a></div>\" data-title=\"Get Full Access\" data-container=\"body\" rel=\"popover\" tabindex=\"0\" href=\"/subscription/new\">Video Transcript</a>\n</div>\n<div class='additional-material'>\n<i aria-hidden='true' class='fai fa-solid fa-lock'></i>\n<a class=\"text-muted\" title=\"Video Preview\" data-html=\"true\" data-placement=\"bottom\" data-trigger=\"click\" data-content=\"<div class=&quot;text-center py-2&quot;><a class=&quot;bold&quot; href=&quot;/auth/users/sign_in&quot;>Sign in</a> or <a class=&quot;bold text-danger&quot; data-posthog-event=&quot;Signup: LP Signup Activity&quot; data-posthog-location=&quot;body_link_boclips&quot; data-remote=&quot;true&quot; href=&quot;/subscription/new&quot;>Join Now</a></div>\" data-title=\"Get Full Access\" data-container=\"body\" rel=\"popover\" tabindex=\"0\" href=\"/subscription/new\">Video Preview</a>\n</div>\n</div>\n<div aria-labelledby='concepts-heading' class='rp-info-section'>\n<h2 class='title' id='concepts-heading'>Concepts</h2>\n<div class='clearfix'>\n<div class='details-list concepts' data-identifier='Boclips::VideoDecorator63eb89cee4dd8246199748fd' data-type='concepts'>data, javascript</div>\n<div class='concepts-toggle-buttons' data-identifier='Boclips::VideoDecorator63eb89cee4dd8246199748fd'>\n<button aria-expanded='false' class='more btn-link' type='button'>\n<span>Show More</span>\n<i aria-hidden='true' class='fa-solid fa-caret-down ml5'></i>\n</button>\n<button aria-expanded='true' class='less btn-link' style='display: none;' type='button'>\n<span>Show Less</span>\n<i aria-hidden='true' class='fa-solid fa-caret-up ml5'></i>\n</button>\n</div>\n</div>\n</div>\n<div aria-labelledby='additional-tags-heading' class='rp-info-section'>\n<h2 class='title' id='additional-tags-heading'>Additional Tags</h2>\n<div class='clearfix'>\n<div class='details-list keyterms' data-identifier='Boclips::VideoDecorator63eb89cee4dd8246199748fd' data-type='keyterms'>web scraping, scraping, scrapy, python, website, dynamic website scraping, pycharm 2021.1.3, visual studio code 2022, python 3.10 and 3.8, scrapy 2.6.1, windows 10, ubuntu 20.0.2, mongodb 5.0.2, scrapy-playwright 0.0.12, json, csv, xlsx (excel), xml, loading screen, actual data, sample website, table body, spider, rendered, selector, found, inside, table, print, page, lesson, scrape, content, load, click, ahead, wait, error, check, select, loop, playwright, create, thing</div>\n<div class='keyterms-toggle-buttons' data-identifier='Boclips::VideoDecorator63eb89cee4dd8246199748fd'>\n<button aria-expanded='false' class='more btn-link' type='button'>\n<span>Show More</span>\n<i aria-hidden='true' class='fa-solid fa-caret-down ml5'></i>\n</button>\n<button aria-expanded='true' class='less btn-link' style='display: none;' type='button'>\n<span>Show Less</span>\n<i aria-hidden='true' class='fa-solid fa-caret-up ml5'></i>\n</button>\n</div>\n</div>\n</div>\n<div aria-labelledby='classroom-considerations-heading' class='rp-info-section'>\n<h2 class='title' id='classroom-considerations-heading'>Classroom Considerations</h2>\n<div class='classroom-considerations'><div class='fai fa-solid fa-bell'></div>Best For: Explaining a topic</div><div class='classroom-considerations'><div class='fai fa-solid fa-bell'></div>Video is ad-free</div> \n</div>\n<div aria-labelledby='educator-ratings-heading' class='rp-info-section'>\n<h2 class='title sr-only' id='educator-ratings-heading'>Educator Ratings</h2>\n<div id=\"educator-ratings-root\"></div><div id=\"all-educator-ratings-root\"></div><div id=\"educator-rating-form-root\"></div>\n</div>\n</div>\n<div class='rp-resource'>\n<div aria-label='Show resource details' class='rp-show-info' role='button' tabindex='0'>\n<i class='fai fa-solid fa-align-left'></i>\nShow resource details\n</div>\n<div aria-label='Video player' class='player' id='player-wrapper' role='region'>\n<div class='relative container mx-auto' id='lp-boclips-visitor-thumbnail'>\n<a class=\"block\" data-html=\"true\" data-placement=\"bottom\" data-trigger=\"click\" data-content=\"<div class=&quot;text-center py-2&quot;><a class=&quot;bold&quot; href=&quot;/auth/users/sign_in&quot;>Sign in</a> or <a class=&quot;bold text-danger&quot; data-posthog-event=&quot;Signup: LP Signup Activity&quot; data-posthog-location=&quot;body_link_boclips&quot; data-remote=&quot;true&quot; href=&quot;/subscription/new&quot;>Join Now</a></div>\" data-title=\"Get Full Access\" data-container=\"body\" rel=\"popover\" tabindex=\"0\" aria-label=\"Play video: Web Scraping Tutorial with Scrapy and Python for Beginners - Scrape a Website with Loading Screen\" href=\"/subscription/new\"><img class=\"resource-img img-thumbnail img-responsive z-10 lp-boclips-thumbnail w-full h-full lozad\" alt=\"Web Scraping Tutorial with Scrapy and Python for Beginners - Scrape a Website with Loading Screen\" title=\"Web Scraping Tutorial with Scrapy and Python for Beginners - Scrape a Website with Loading Screen\" onError=\"handleImageNotLoadedError(this)\" data-default-image=\"https://statictemp.lp.lexp.cloud/images/attachment_defaults/resource/large/missing.png\" data-src=\"https://cdnapisec.kaltura.com/p/1776261/thumbnail/entry_id/1_6zdec91j/width/250/vid_slices/3/vid_slice/1\" width=\"315\" height=\"220\" src=\"data:image/png;base64,R0lGODlhAQABAAD/ACwAAAAAAQABAAACADs\" />\n<span aria-hidden='true' class='flex justify-center items-center bg-white rounded-full w-16 h-16 absolute top-1/2 left-1/2 -mt-8 -ml-8 cursor-pointer z-0 border-2 border-primary drop-shadow-md lp-boclips-thumbnail-playBtn'>\n<i class='fa-solid fa-play text-primary text-3xl ml-1 drop-shadow-xl'></i>\n</span>\n</a></div>\n</div>\n</div>\n</div>\n</div>\n</div>\n"}