Scrapy is a powerful open source web scraping framework written in Python that can be used for data mining, information processing or historical archival. It is designed to be fast, flexible and easy to use for large scale scraping projects.

It is a web scraping framework that can be used to download, process and store data in different formats like CSV, JSON and XML. It can also be used to create APIs for scraping data.

In this tutorial, we will learn how to scrape and extract data from Medium articles using Scrapy. We will also configure the scraper so that it is efficient, ethical and robust.

Getting Started with Scrapy

The first thing to do is install Scrapy, which is a python module. To do that, you need to have Python 3.3 or higher installed on your system.

You can either run the installer manually or you can just use pip to install it through your terminal. Once you have installed it, you will need to create a project that will encapsulate your spiders, utilities, and deployment configs.

Once the project is created, it will include a folder called spiders and a few files. Inside the spiders folder, you will find your spiders which are classes that define how Scrapy scrapes a site (or set of sites).

These spiders must subclass Spider to determine how to make requests and optionally follow links in the page to yield data points. They must also create a generator function to begin crawling from, and a callback function that processes responses.

Requests are the initial requests that Scrapy https://scrapy.ca/en/location/sell-your-car-montreal/ will send to web pages to get the data points they contain. They are based on HTML selectors and XPath expressions that scrape the elements of the page to get the data.

XPath expressions are powerful because they can be applied to any object on the page, even images. They are also a lot faster than a CSS selector, which can take time to compute.

They are asynchronous which means they execute multiple requests simultaneously, and if any errors happen, they do not affect the rest of the incoming requests. This improves speed efficiency and is great for web scraping where many requests must be sent to a website to obtain the desired data.

You can use a python request object to generate the requests, but it's easier and quicker to use generator functions to do this for you. These generators can be used for both requests and responses, so you can write one to yield a request and then another to process the response and yield more requests or data points.

This is a good way to keep the code tidy and it also prevents any coding errors from happening. In addition, generators allow you to reuse a lot of code between requests.

If you have any recurring tasks, you can write a generator for them so they don't have to be done again each time you scrape the same page. You can also write a pipeline for generating them, which is useful for filtering duplicate items or adding computed values to them.