Web Crawler Python


Scrapy | A Fast and Powerful Scraping and Web Crawling ...

Scrapy | A Fast and Powerful Scraping and Web Crawling …

An open source and collaborative framework for extracting the data you need from websites.
In a fast, simple, yet extensible way.
Maintained by
(formerly Scrapinghub)
many other contributors
Install the latest version of Scrapy
Scrapy 2. 5. 0
pip install scrapy
cat > <
# Deploy the spider to Zyte Scrapy Cloud
shub deploy
# Schedule the spider for execution
shub schedule blogspider
Spider blogspider scheduled, watch it running here:
# Retrieve the scraped data
shub items 26731/1/8
{“title”: “Improved Frontera: Web Crawling at Scale with Python 3 Support”}
{“title”: “How to Crawl the Web Politely with Scrapy”}…
Fast and powerful
write the rules to extract the data and let Scrapy do the rest
Easily extensible
extensible by design, plug new functionality easily without having to touch the core
Portable, Python
written in Python and runs on Linux, Windows, Mac and BSD
Healthy community
– 36. 3k stars, 8. 4k forks and 1. 8k watchers on GitHub
– 5. 1k followers on Twitter
– 14. 7k questions on StackOverflow
Web crawling with Python - ScrapingBee

Web crawling with Python – ScrapingBee

11 December, 2020
13 min read
Ari is an expert Data Engineer and a talented technical writer. He wrote the entire Scrapy integration for ScrapingBee and this awesome article.
Web crawling is a powerful technique to collect data from the web by finding all the URLs for one or multiple domains. Python has several popular web crawling libraries and frameworks.
In this article, we will first introduce different crawling strategies and use cases. Then we will build a simple web crawler from scratch in Python using two libraries: requests and Beautiful Soup. Next, we will see why it’s better to use a web crawling framework like Scrapy. Finally, we will build an example crawler with Scrapy to collect film metadata from IMDb and see how Scrapy scales to websites with several million pages.
What is a web crawler?
Web crawling and web scraping are two different but related concepts. Web crawling is a component of web scraping, the crawler logic finds URLs to be processed by the scraper code.
A web crawler starts with a list of URLs to visit, called the seed. For each URL, the crawler finds links in the HTML, filters those links based on some criteria and adds the new links to a queue. All the HTML or some specific information is extracted to be processed by a different pipeline.
Web crawling strategies
In practice, web crawlers only visit a subset of pages depending on the crawler budget, which can be a maximum number of pages per domain, depth or execution time.
Most popular websites provide a file to indicate which areas of the website are disallowed to crawl by each user agent. The opposite of the robots file is the file, that lists the pages that can be crawled.
Popular web crawler use cases include:
Search engines (Googlebot, Bingbot, Yandex Bot…) collect all the HTML for a significant part of the Web. This data is indexed to make it searchable.
SEO analytics tools on top of collecting the HTML also collect metadata like the response time, response status to detect broken pages and the links between different domains to collect backlinks.
Price monitoring tools crawl e-commerce websites to find product pages and extract metadata, notably the price. Product pages are then periodically revisited.
Common Crawl maintains an open repository of web crawl data. For example, the archive from October 2020 contains 2. 71 billion web pages.
Next, we will compare three different strategies for building a web crawler in Python. First, using only standard libraries, then third party libraries for making HTTP requests and parsing HTML and finally, a web crawling framework.
Building a simple web crawler in Python from scratch
To build a simple web crawler in Python we need at least one library to download the HTML from a URL and an HTML parsing library to extract links. Python provides standard libraries urllib for making HTTP requests and for parsing HTML. An example Python crawler built only with standard libraries can be found on Github.
The standard Python libraries for requests and HTML parsing are not very developer-friendly. Other popular libraries like requests, branded as HTTP for humans, and Beautiful Soup provide a better developer experience.
If you wan to learn more, you can check this guide about the best Python HTTP client.
You can install the two libraries locally.
A basic crawler can be built following the previous architecture diagram.
import logging
from import urljoin
import requests
from bs4 import BeautifulSoup
format=’%(asctime)s%(levelname)s:%(message)s’, )
class Crawler:
def __init__(self, urls=[]):
sited_urls = []
self. urls_to_visit = urls
def download_url(self, url):
return (url)
def get_linked_urls(self, url, html):
soup = BeautifulSoup(html, ”)
for link in nd_all(‘a’):
path = (‘href’)
if path and artswith(‘/’):
path = urljoin(url, path)
yield path
def add_url_to_visit(self, url):
if url not in sited_urls and url not in self. urls_to_visit:
def crawl(self, url):
html = wnload_url(url)
for url in t_linked_urls(url, html):
def run(self):
while self. urls_to_visit:
url = (0)
(f’Crawling: {url}’)
except Exception:
logging. exception(f’Failed to crawl: {url}’)
if __name__ == ‘__main__’:
The code above defines a Crawler class with helper methods to download_url using the requests library, get_linked_urls using the Beautiful Soup library and add_url_to_visit to filter URLs. The URLs to visit and the visited URLs are stored in two separate lists. You can run the crawler on your terminal.
The crawler logs one line for each visited URL.
2020-12-04 18:10:10, 737 INFO:Crawling: 2020-12-04 18:10:11, 599 INFO:Crawling: 2020-12-04 18:10:12, 868 INFO:Crawling: 2020-12-04 18:10:13, 526 INFO:Crawling: 2020-12-04 18:10:19, 174 INFO:Crawling: 2020-12-04 18:10:20, 624 INFO:Crawling: 2020-12-04 18:10:21, 556 INFO:Crawling: The code is very simple but there are many performance and usability issues to solve before successfully crawling a complete website.
The crawler is slow and supports no parallelism. As can be seen from the timestamps, it takes about one second to crawl each URL. Each time the crawler makes a request it waits for the request to be resolved and no work is done in between.
The download URL logic has no retry mechanism, the URL queue is not a real queue and not very efficient with a high number of URLs.
The link extraction logic doesn’t support standardizing URLs by removing URL query string parameters, doesn’t handle URLs starting with #, doesn’t support filtering URLs by domain or filtering out requests to static files.
The crawler doesn’t identify itself and ignores the file.
Next, we will see how Scrapy provides all these functionalities and makes it easy to extend for your custom crawls.
Web crawling with Scrapy
Scrapy is the most popular web scraping and crawling Python framework with 40k stars on Github. One of the advantages of Scrapy is that requests are scheduled and handled asynchronously. This means that Scrapy can send another request before the previous one is completed or do some other work in between. Scrapy can handle many concurrent requests but can also be configured to respect the websites with custom settings, as we’ll see later.
Scrapy has a multi-component architecture. Normally, you will implement at least two different classes: Spider and Pipeline. Web scraping can be thought of as an ETL where you extract data from the web and load it to your own storage. Spiders extract the data and pipelines load it into the storage. Transformation can happen both in spiders and pipelines, but I recommend that you set a custom Scrapy pipeline to transform each item independently of each other. This way, failing to process an item has no effect on other items.
On top of all that, you can add spider and downloader middlewares in between components as it can be seen in the diagram below.
Scrapy Architecture Overview [source]
If you have used Scrapy before, you know that a web scraper is defined as a class that inherits from the base Spider class and implements a parse method to handle each response. If you are new to Scrapy, you can read this article for easy scraping with Scrapy.
from scrapy. spiders import Spider
class ImdbSpider(Spider):
name = ‘imdb’
allowed_domains = [”]
start_urls = [”]
def parse(self, response):
Scrapy also provides several generic spider classes: CrawlSpider, XMLFeedSpider, CSVFeedSpider and SitemapSpider. The CrawlSpider class inherits from the base Spider class and provides an extra rules attribute to define how to crawl a website. Each rule uses a LinkExtractor to specify which links are extracted from each page. Next, we will see how to use each one of them by building a crawler for IMDb, the Internet Movie Database.
Building an example Scrapy crawler for IMDb
Before trying to crawl IMDb, I checked IMDb file to see which URL paths are allowed. The robots file only disallows 26 paths for all user-agents. Scrapy reads the file beforehand and respects it when the ROBOTSTXT_OBEY setting is set to true. This is the case for all projects generated with the Scrapy command startproject.
scrapy startproject scrapy_crawler
This command creates a new project with the default Scrapy project folder structure.
└── scrapy_crawler
└── spiders
Then you can create a spider in scrapy_crawler/spiders/ with a rule to extract all links.
from scrapy. spiders import CrawlSpider, Rule
from nkextractors import LinkExtractor
class ImdbCrawler(CrawlSpider):
rules = (Rule(LinkExtractor()), )
You can launch the crawler in the terminal.
scrapy crawl imdb –logfile
You will get lots of logs, including one log for each request. Exploring the logs I noticed that even if we set allowed_domains to only crawl web pages under, there were requests to external domains, such as
2020-12-06 12:25:18 [direct] DEBUG: Redirecting (302) to <GET > from <GET [()>
IMDb redirects from URLs paths under whitelist-offsite and whitelist to external domains. There is an open Scrapy Github issue that shows that external URLs don’t get filtered out when the OffsiteMiddleware is applied before the RedirectMiddleware. To fix this issue, we can configure the link extractor to deny URLs starting with two regular expressions.
rules = (
(”), ], )), )
Rule and LinkExtractor classes support several arguments to filter out URLs. For example, you can ignore specific URL extensions and reduce the number of duplicate URLs by sorting query strings. If you don’t find a specific argument for your use case you can pass a custom function to process_links in LinkExtractor or process_values in Rule.
For example, IMDb has two different URLs with the same content.
To limit the number of crawled URLs, we can remove all query strings from URLs with the url_query_cleaner function from the w3lib library and use it in process_links.
from import url_query_cleaner
def process_links(links):
for link in links:
= url_query_cleaner()
yield link
(”), ], ), process_links=process_links), )
Now that we have limited the number of requests to process, we can add a parse_item method to extract data from each page and pass it to a pipeline to store it. For example, we can either extract the whole to process it in a different pipeline or select the HTML metadata. To select the HTML metadata in the header tag we can code our own XPATHs but I find it better to use a library, extruct, that extracts all metadata from an HTML page. You can install it with pip install extract.
import re
import extruct
(”), ], ),
follow=True), )
def parse_item(self, response):
return {
‘metadata’: extruct. extract(,,
syntaxes=[‘opengraph’, ‘json-ld’]), }
I set the follow attribute to True so that Scrapy still follows all links from each response even if we provided a custom parse method. I also configured extruct to extract only Open Graph metadata and JSON-LD, a popular method for encoding linked data using JSON in the Web, used by IMDb. You can run the crawler and store items in JSON lines format to a file.
scrapy crawl imdb –logfile -o -t jsonlines
The output file contains one line for each crawled item. For example, the extracted Open Graph metadata for a movie taken from the tags in the HTML looks like this.
“url”: “,
“metadata”: {“opengraph”: [{
“namespace”: {“og”: “},
“properties”: [
[“og:url”, “],
[“og:image”, “],
[“og:type”, “_show”],
[“og:title”, “Peaky Blinders (TV Series 2013\u2013) – IMDb”],
[“og:site_name”, “IMDb”],
[“og:description”, “Created by Steven Knight. With Cillian Murphy, Paul Anderson, Helen McCrory, Sophie Rundle. A gangster family epic set in 1900s England, centering on a gang who sew razor blades in the peaks of their caps, and their fierce boss Tommy Shelby. “]]}]}}
The JSON-LD for a single item is too long to be included in the article, here is a sample of what Scrapy extracts from the

Useful Tools