Here’s how it works: first, you write some code to tell Scrapy which website(s) you want to scrape (that means “extract data from”) and what kind of information you’re looking for. For example, let’s say you want to extract all the product names and prices from a certain e-commerce site. You would create a new project in Scrapy using their handy command line tool:
# This script is used to create a new project in Scrapy and set up a spider to extract data from a website.
# First, we use the "scrapy startproject" command to create a new project called "my_spider".
scrapy startproject my_spider
# Then, we change into the newly created project directory.
cd my_spider
# Next, we create a new python file called "example.py" inside the "spiders" folder.
touch spiders/example.py
This creates a new directory called “my_spider” with some boilerplate code inside, and then opens up the file “example.py” for editing. In this file, you would write your custom scraping logic using Python:
# This script creates a new directory called "my_spider" and opens the file "example.py" for editing, where custom scraping logic can be written using Python.
# Import the scrapy library, which is used for web scraping.
import scrapy
# Import the MyItem class from the items module in the myproject package.
from myproject.items import MyItem
# Create a new class called ExampleSpider, which inherits from the scrapy.Spider class.
class ExampleSpider(scrapy.Spider):
# Set the name of the spider to "example".
name = 'example'
# Set the starting URL for the spider to "https://www.example.com".
start_urls = ['https://www.example.com']
# Define a function called parse, which takes in the response from the starting URL.
def parse(self, response):
# Use the css method to select all elements with the class "product-link" from the response.
for link in response.css('.product-link'):
# Use the follow method to navigate to the link and call the parse_item function.
yield response.follow(link, self.parse_item)
# Define a function called parse_item, which takes in the response from the product link.
def parse_item(self, response):
# Create a new instance of the MyItem class.
item = MyItem()
# Use the css method to select the element with the class "product-title" and extract the text.
item['name'] = response.css('.product-title::text').get()
# Use the css method to select the element with the class "product-price", extract the text, and convert it to a float.
item['price'] = float(response.css('.product-price::text').extract()[0].replace('$', ''))
# Yield the item to be processed by the pipeline.
yield item
This code defines a new spider called “example” that starts at the URL “https://www.example.com”. It then uses CSS selectors to find all the product links on this page, and follows each one using `response.follow()`. For each link it visits, it extracts the name and price of the item using more CSS selectors, and creates a new item object with that data.
Once you’ve written your spider code, you can run Scrapy to start scraping:
# This line imports the Scrapy library, allowing us to use its functions and methods.
import scrapy
# This class defines our spider, which will crawl through the webpage and extract data.
class ExampleSpider(scrapy.Spider):
# This line sets the name of our spider.
name = 'example'
# This line defines the starting URL for our spider to crawl.
start_urls = ['http://www.example.com/products']
# This function will be called for each URL in the start_urls list.
def parse(self, response):
# This line uses a CSS selector to extract all the product links on the page.
product_links = response.css('a.product-link::attr(href)').getall()
# This loop will iterate through each product link and follow it.
for link in product_links:
# This line uses the response.follow() method to follow the link.
yield response.follow(link, callback=self.parse_product)
# This function will be called for each product link that is followed.
def parse_product(self, response):
# This line uses CSS selectors to extract the name and price of the product.
name = response.css('h1.product-name::text').get()
price = response.css('span.product-price::text').get()
# This line creates a new item object with the extracted data.
item = {'name': name, 'price': price}
# This line yields the item, making it available for further processing.
yield item
# This line runs the spider and saves the extracted data to a JSON file.
scrapy crawl example -o output.json
This will execute the “example” spider and save its results to a file called “output.json”. And that’s it! You now have a fully automated web scraper that can extract data from any website you want, without having to manually click around or copy stuff into your clipboard. Pretty cool, huh?