Web Scraping Tutorial For Beginners With Scrapy & Python
Last updated 6/2022
MP4 | Video: h264, 1280x720 | Audio: AAC, 44.1 KHz
Language: English | Size: 3.45 GB | Duration: 7h 33m
Learn and Master Web Scraping using Scrapy Framework and this Step-By-Step & In-Depth Guide
What you'll learn

Define the Steps Involved in Web Scraping and Creating Web Crawlers
Install and Setup Scrapy in Windows, Mac OS, Ubuntu (Linux) & Anaconda Environments
Send Request to a URL to Scrape Websites Using Scrapy Spider
Get the HTML Response From URL and Parse it for Web Scraping
Select Desired Data From Websites Using Scrapy Selector, CSS Selectors & XPath
Scrapy Crawl Spiders to Get Data From Websites And Extract it to JSON, CSV, XLSX ( Excel ) and XML Files
Use Scrapy Shell Commands to Test & Verify CSS Selectors or XPath
Export and Save Scraped Data to Online Databases Like MonogoDB Using Scrapy Item Pipelines
Define Scrapy Items to Organize Scraped Data And Load Items Using Scrapy Itemloaders with Input & Output Processors
Scrape Data From Multiple Web Pages Using Scrapy Pagination And Extract Data From HTML Tables
Login Into Websites Using Scrapy FormRequest With CSRF Tokens
Scrape Dynamic/jаvascript Rendered Websites Using Scrapy-Playwright And Interact With Web Elements, Take Screenshot of Websites or Save as PDF
Identify API Calls From a Website and Scrape Data From API Using Scrapy Request
Requirements
Python Programming
HTML Basics (+point)
Description
Web scraping is the process of scraping websites and extracting desired data from the same, and in this course, you'll learn and master web scraping using python and scrapy with a step-by-step and in-depth guide.A Step-By-Step GuideAssuming that you don't know anything about web scraping, scrapy python web scraping, or even web scraping meaning - we will start from the complete basics. In the first section, you'll learn about the web scraping process step-by-step (with infographics - no code), how to scrape data from websites and how to use scrapy for the same (i.e. scrapy meaning).After getting the basics clear and having an idea of how web scraping works, we will start web scraping using python & scrapy framework! Again, we'll move step-by-step and perform each step learned in the basics with bite-sized lessons. We'll take it slow so that it's easier for you to understand each and every step involved in scraping and extracting data from websites.Web Scraping & Scrapy EssentialsHaving built an actual web scraper, you'll get an idea of how web scraping works firsthand. Now it's crucial to cover the essential concepts of web scraping and scrapy, which we will do next.CSS Selectors to select web elementsXPath to select web elementsScrapy Shell to test & verify selectorsItems to organize extracted dataLoad Items with ItemLoaders with input & output ProcessorsExport data to JSON, CSV, XLSX (Excel) & XML file formatsSave extracted data to online databases like MongoDB using ItemPipelinesMaster Web Scraping In-DepthLearning how to scrape websites and the essentials already makes you a complete web scraper but, we'll take this even further and learn the advanced web scraping techniques to become an expert!Follow links in a webpage to another pageCrawl multiple pages and extract data i.e. PaginationScrape data using Regular Expressions (RegEx)Extract Data From HTML TablesLogin Into Websites Using Scrapy FormRequestBypass CSRF protected Login formsScrape Dynamic or jаvascript Rendered Websites using Scrapy PlaywrightInteract with web elements like fill forms, click buttons, etc.Handle Infinite Scroll websitesWait For Elements when contents/data takes time to loadTake Screenshot of websitesSave websites as PDFIdentify API calls from websites and scrape data from APIsUse middleware in a scrapy projectConfigure settings in a scrapy projectUse and Rotate User-Agents & ProxiesWeb scraping Best PracticesReal-World ProjectsAfter master web scraping, we need projects to get started! That's why you'll perform three projects as well:Champions League Table[ ESPN ]Product Tracker[ Amazon ]Scraper Application[ GUI ]Join us in this in-depth course where you'll learn about web scraping from scratch and master the process of extracting data from websites step-by-step. Check out the preview lessons to get started and learn how web scraping works! See you there~
Overview
Section 1: Introduction
Lecture 1 What is Web Scraping?
Lecture 2 How Web Scraping Works?
Lecture 3 Web Scraping With Scrapy
Section 2: Scrapy Installation
Lecture 4 Scrapy Installation for Windows
Lecture 5 Scrapy Installation for Ubuntu (Linux)
Lecture 6 Scrapy Installation for Mac
Lecture 7 Scrapy Installation for Anaconda
Lecture 8 Creating Scrapy Project
Lecture 9 Project Walkthrough
Section 3: Scrapy Spider
Lecture 10 Creating Spider
Lecture 11 Sending Request
Lecture 12 Getting the Response
Lecture 13 Scrapy CSS Selector
Lecture 14 Selecting All The Data
Lecture 15 Extracting Data
Lecture 16 Spider Overview
Section 4: CSS Selectors
Lecture 17 CSS Selectors v/s XPath : How to Select Web Elements?
Lecture 18 Tagname, Class and Id Selectors
Lecture 19 Attribute Selectors
Lecture 20 CSS Selectors Cheat Sheet
Section 5: XPath
Lecture 21 XPath Expressions
Lecture 22 XPath Attribute Selectors
Lecture 23 XPath text( ) Function
Lecture 24 XPath Cheat Sheet
Section 6: Scrapy Shell
Lecture 25 What is the Scrapy Shell and How to Use it?
Lecture 26 fetch( ) Response
Lecture 27 Shell Configuration
Section 7: Scrapy Items
Lecture 28 Structuring Data Into Scrapy Item
Lecture 29 Using Item in Spiders
Lecture 30 Define Input and Output Processors For Item Fields
Lecture 31 Loading Items with Scrapy ItemLoaders
Lecture 32 Items, Processors & ItemLoaders Overview
Section 8: Exporting Data
Lecture 33 Output Extracted Data In JSON, CSV & XML Format
Lecture 34 Overwrite Previous Output
Lecture 35 Appending Data to Previous Output
Section 9: Scrapy Item Pipeline
Lecture 36 How to use Scrapy Item Pipelines?
Lecture 37 Saving Data Locally to Excel ( XLSX ) Files
Lecture 38 Enable Item Pipelines in Settings
Lecture 39 MongoDB (Account) Setup
Lecture 40 Saving Data To MonogoDB
Section 10: Pagination
Lecture 41 Extracting Links From href Attributes
Lecture 42 Send Request to the Next Page
Lecture 43 start_requests( ) method
Section 11: Following Links
Lecture 44 How to Follow Links?
Lecture 45 How to Select Data Using Regular Expressions With Scrapy
Lecture 46 Setting Up Custom Callback Function
Lecture 47 Parse Product Details Page
Section 12: Scraping Tables
Lecture 48 HTML Tables
Lecture 49 Selecting Tables Data
Lecture 50 Extract Data From HTML Tables
Section 13: Logging Into Websites
Lecture 51 Data Hidden With Logging Forms
Lecture 52 Inspecting HTML Forms and Website Activity With Dev Tools
Lecture 53 Logging Into Websites With FormRequest
Lecture 54 CSRF Protected Login Forms
Lecture 55 Extract CSRF Values From Forms
Section 14: Scraping jаvascript Rendered Websites
Lecture 56 What are jаvascript Rendered/Dynamic Websites?
Lecture 57 scrapy-playwright Installation
Lecture 58 Setting Up Playwright in Scrapy Project
Lecture 59 Using Playwright To Render Websites
Lecture 60 Scraping Data From Dynamic Websites
Section 15: Scrapy Playwright
Lecture 61 Playwright Overview
Lecture 62 Playwright Page Object
Lecture 63 Logging In With Playwright
Lecture 64 Dynamic Websites With Loading Screens
Lecture 65 Wait For Selector/Elements Using Page Couroutines
Lecture 66 Dynamic Websites With Infinite Scroll
Lecture 67 Taking Screenshot of Websites
Lecture 68 Rendering Websites To PDF
Section 16: API Endpoints
Lecture 69 Identifying API Calls
Lecture 70 Requesting Data From API
Lecture 71 Extracting Data From API
Section 17: Settings
Lecture 72 Scrapy Project Settings
Lecture 73 Robots Text
Lecture 74 Middleware
Lecture 75 Autothrottle Extension
Section 18: User Agents & Proxies
Lecture 76 What are User Agents?
Lecture 77 User Agents With Scrapy
Lecture 78 What are Proxies?
Lecture 79 Proxies With Scrapy
Section 19: Tips & Tricks
Lecture 80 Spider Arguments
Lecture 81 Standalone Spiders
Lecture 82 Scrapy Shell With bpython
Lecture 83 Scrapy get vs extract method
Lecture 84 Logging
Section 20: Project #1: Champions League Table From ESPN.com
Lecture 85 Overview
Lecture 86 Website Visual Inspection
Lecture 87 Finding The Selectors
Lecture 88 Building The Spider: Extract Teams Data
Lecture 89 Building The Spider: Extract Teams Details
Section 21: Project #2: Amazon Product Rank
Lecture 90 Overview
Lecture 91 Scraper Visualization
Lecture 92 Finding The Selectors
Lecture 93 Building The Spider
Section 22: Project #3: Extending Scraper With GUI
Lecture 94 Scraper Application
Lecture 95 Building The GUI (Application Interface)
Lecture 96 Running the Spider From the Application
Beginner Python Developers Who Wants to Master Web Scraping,Freelancer Web Scrapers Looking To Polish Their Skills
Screenshots