Bs4 Tutorial

B

Tutorial: Web Scraping with Python Using Beautiful Soup

Tutorial: Web Scraping with Python Using Beautiful Soup

Published: March 30, 2021 Learn how to scrape the web with Python! The internet is an absolutely massive source of data — data that we can access using web scraping and Python! In fact, web scraping is often the only way we can access data. There is a lot of information out there that isn’t available in convenient CSV exports or easy-to-connect APIs. And websites themselves are often valuable sources of data — consider, for example, the kinds of analysis you could do if you could download every post on a web access those sorts of on-page datasets, we’ll have to use web scraping. Don’t worry if you’re still a total beginner! In this tutorial we’re going to cover how to do web scraping with Python from scratch, starting with some answers to frequently-asked, we’ll work through an actual web scraping project, focusing on weather ‘ll work together to scrape weather data from the web to support a weather before we start writing any Python, we’ve got to cover the basics! If you’re already familiar with the concept of web scraping, feel free to scroll past these questions and jump right into the tutorial! The Fundamentals of Web Scraping:What is Web Scraping in Python? Some websites offer data sets that are downloadable in CSV format, or accessible via an Application Programming Interface (API). But many websites with useful data don’t offer these convenient nsider, for example, the National Weather Service’s website. It contains up-to-date weather forecasts for every location in the US, but that weather data isn’t accessible as a CSV or via API. It has to be viewed on the NWS site:If we wanted to analyze this data, or download it for use in some other app, we wouldn’t want to painstakingly copy-paste everything. Web scraping is a technique that lets us use programming to do the heavy lifting. We’ll write some code that looks at the NWS site, grabs just the data we want to work with, and outputs it in the format we this tutorial, we’ll show you how to perform web scraping using Python 3 and the Beautiful Soup library. We’ll be scraping weather forecasts from the National Weather Service, and then analyzing them using the Pandas to be clear, lots of programming languages can be used to scrape the web! We also teach web scraping in R, for example. For this tutorial, though, we’ll be sticking with Does Web Scraping Work? When we scrape the web, we write code that sends a request to the server that’s hosting the page we specified. The server will return the source code — HTML, mostly — for the page (or pages) we far, we’re essentially doing the same thing a web browser does — sending a server request with a specific URL and asking the server to return the code for that unlike a web browser, our web scraping code won’t interpret the page’s source code and display the page visually. Instead, we’ll write some custom code that filters through the page’s source code looking for specific elements we’ve specified, and extracting whatever content we’ve instructed it to example, if we wanted to get all of the data from inside a table that was displayed on a web page, our code would be written to go through these steps in sequence:1Request the content (source code) of a specific URL from the server2Download the content that is returned3Identify the elements of the page that are part of the table we want4Extract and (if necessary) reformat those elements into a dataset we can analyze or use in whatever way we that all sounds very complicated, don’t worry! Python and Beautiful Soup have built-in features designed to make this relatively straightforward. One thing that’s important to note: from a server’s perspective, requesting a page via web scraping is the same as loading it in a web browser. When we use code to submit these requests, we might be “loading” pages much faster than a regular user, and thus quickly eating up the website owner’s server Use Python for Web Scraping? As previously mentioned, it’s possible to do web scraping with many programming ever, one of the most popular approaches is to use Python and the Beautiful Soup library, as we’ll do in this tutorial. Learning to do this with Python will mean that there are lots of tutorials, how-to videos, and bits of example code out there to help you deepen your knowledge once you’ve mastered the Beautiful Soup Web Scraping Legal? Unfortunately, there’s not a cut-and-dry answer here. Some websites explicitly allow web scraping. Others explicitly forbid it. Many websites don’t offer any clear guidance one way or the scraping any website, we should look for a terms and conditions page to see if there are explicit rules about scraping. If there are, we should follow them. If there are not, then it becomes more of a judgement member, though, that web scraping consumes server resources for the host website. If we’re just scraping one page once, that isn’t going to cause a problem. But if our code is scraping 1, 000 pages once every ten minutes, that could quickly get expensive for the website, in addition to following any and all explicit rules about web scraping posted on the site, it’s also a good idea to follow these best practices:Web Scraping Best Practices:Never scrape more frequently than you need nsider caching the content you scrape so that it’s only downloaded pauses into your code using functions like () to keep from overwhelming servers with too many requests too our case for this tutorial, the NWS’s data is public domain and its terms do not forbid web scraping, so we’re in the clear to to scrape the web with Python, right in your browser! Our interactive APIs and Web Scraping in Python skill path will help you learn the skills you need to unlock new worlds of data with Python. (No credit card required! ) The Components of a Web PageBefore we start writing code, we need to understand a little bit about the structure of a web page. We’ll use the site’s structure to write code that gets us the data we want to scrape, so understanding that structure is an important first step for any web scraping we visit a web page, our web browser makes a request to a web server. This request is called a GET request, since we’re getting files from the server. The server then sends back files that tell our browser how to render the page for us. These files will typically include:HTML — the main content of the — used to add styling to make the page look — Javascript files add interactivity to web — image formats, such as JPG and PNG, allow web pages to show our browser receives all the files, it renders the page and displays it to ’s a lot that happens behind the scenes to render a page nicely, but we don’t need to worry about most of it when we’re web scraping. When we perform web scraping, we’re interested in the main content of the web page, so we look primarily at the MLHyperText Markup Language (HTML) is the language that web pages are created in. HTML isn’t a programming language, like Python, though. It’s a markup language that tells a browser how to display content. HTML has many functions that are similar to what you might find in a word processor like Microsoft Word — it can make text bold, create paragraphs, and so you’re already familiar with HTML, feel free to jump to the next section of this tutorial. Otherwise, let’s take a quick tour through HTML so we know enough to scrape consists of elements called tags. The most basic tag is the tag. This tag tells the web browser that everything inside of it is HTML. We can make a simple HTML document just using this tag:We haven’t added any content to our page yet, so if we viewed our HTML document in a web browser, we wouldn’t see anything:Right inside an html tag, we can put two other tags: the head tag, and the body main content of the web page goes into the body tag. The head tag contains data about the title of the page, and other information that generally isn’t useful in web scraping:We still haven’t added any content to our page (that goes inside the body tag), so if we open this HTML file in a browser, we still won’t see anything:You may have noticed above that we put the head and body tags inside the html tag. In HTML, tags are nested, and can go inside other ’ll now add our first content to the page, inside a p tag. The p tag defines a paragraph, and any text inside the tag is shown as a separate paragraph:

Here’s a paragraph of text!

Here’s a second paragraph of text!

Rendered in a browser, that HTML file will look like this: Here’s a paragraph of text! Here’s a second paragraph of text! Tags have commonly used names that depend on their position in relation to other tags:child — a child is a tag inside another tag. So the two p tags above are both children of the body — a parent is the tag another tag is inside. Above, the html tag is the parent of the body biling — a sibiling is a tag that is nested inside the same parent as another tag. For example, head and body are siblings, since they’re both inside html. Both p tags are siblings, since they’re both inside can also add properties to HTML tags that change their behavior. Below, we’ll add some extra text and hyperlinks using the a tag.

Here’s a paragraph of text! Python

Here’s how this will look:In the above example, we added two a tags. a tags are links, and tell the browser to render a link to another web page. The href property of the tag determines where the link goes. a and p are extremely common html tags. Here are a few others:div — indicates a division, or area, of the page. b — bolds any text inside. i — italicizes any text — creates a — creates an input a full list of tags, look we move into actual web scraping, let’s learn about the class and id properties. These special properties give HTML elements names, and make them easier to interact with when we’re element can have multiple classes, and a class can be shared between elements. Each element can only have one id, and an id can only be used once on a page. Classes and ids are optional, and not all elements will have can add classes and ids to our example:

Here’s a paragraph of text! Learn Data Science Online

Here’s a second paragraph of text! Python

Here’s how this will look:As you can see, adding classes and ids doesn’t change how the tags are rendered at requests libraryNow that we understand the structure of a web page, it’s time to get into the fun part: scraping the content we want! The first thing we’ll need to do to scrape a web page is to download the page. We can download pages using the Python requests requests library will make a GET request to a web server, which will download the HTML contents of a given web page for us. There are several different types of requests we can make using requests, of which GET is just one. If you want to learn more, check out our API ’s try downloading a simple sample website, ll need to first import the requests library, and then download the page using the method:import requests
page = (“)
page
After running our request, we get a Response object. This object has a status_code property, which indicates if the page was downloaded successfully:A status_code of 200 means that the page downloaded successfully. We won’t fully dive into status codes here, but a status code starting with a 2 generally indicates success, and a code starting with a 4 or a 5 indicates an can print out the HTML content of the page using the content ntent



A simple example page

Here is some simple content for this page.


Parsing a page with BeautifulSoupAs you can see above, we now have downloaded an HTML can use the BeautifulSoup library to parse this document, and extract the text from the p first have to import the library, and create an instance of the BeautifulSoup class to parse our document:from bs4 import BeautifulSoup
soup = BeautifulSoup(ntent, ”)We can now print out the HTML content of the page, formatted nicely, using the prettify method on the BeautifulSoup object.
This step isn’t strictly necessary, and we won’t always bother with it, but it can be helpful to look at prettified HTML to make the structure of the and where tags are nested easier to all the tags are nested, we can move through the structure one level at a time. We can first select all the elements at the top level of the page using the children property of soup. Note that children returns a list generator, so we need to call the list function on it:list(ildren)
[‘html’, ‘n’, A simple example page

Here is some simple content for this page.

]The above tells us that there are two tags at the top level of the page — the initial tag, and the tag. There is a newline character (n) in the list as well. Let’s see what the type of each element in the list is:[type(item) for item in list(ildren)]
[ctype, vigableString, ]As we can see, all of the items are BeautifulSoup objects:The first is a Doctype object, which contains information about the type of the second is a NavigableString, which represents text found in the HTML final item is a Tag object, which contains other nested most important object type, and the one we’ll deal with most often, is the Tag Tag object allows us to navigate through an HTML document, and extract other tags and text. You can learn more about the various BeautifulSoup objects can now select the html tag and its children by taking the third item in the list:html = list(ildren)[2]Each item in the list returned by the children property is also a BeautifulSoup object, so we can also call the children method on, we can find the children inside the html tag:list(ildren)
[‘n’, A simple example page , ‘n’,

Here is some simple content for this page.

, ‘n’]As we can see above, there are two tags here, head, and body. We want to extract the text inside the p tag, so we’ll dive into the body:body = list(ildren)[3]Now, we can get the p tag by finding the children of the body tag:list(ildren)
[‘n’,

Here is some simple content for this page.

, ‘n’]We can now isolate the p tag:p = list(ildren)[1]Once we’ve isolated the tag, we can use the get_text method to extract all of the text inside the t_text()
‘Here is some simple content for this page. ‘Finding all instances of a tag at onceWhat we did above was useful for figuring out how to navigate a page, but it took a lot of commands to do something fairly simple. If we want to extract a single tag, we can instead use the find_all method, which will find all the instances of a tag on a = BeautifulSoup(ntent, ”)
nd_all(‘p’)
[

Here is some simple content for this page.

]Note that find_all returns a list, so we’ll have to loop through, or use list indexing, it to extract nd_all(‘p’)[0]. get_text()
‘Here is some simple content for this page. ‘f you instead only want to find the first instance of a tag, you can use the find method, which will return a single BeautifulSoup (‘p’)

Here is some simple content for this page.

Searching for tags by class and idWe introduced classes and ids earlier, but it probably wasn’t clear why they were asses and ids are used by CSS to determine which HTML elements to apply certain styles to. But when we’re scraping, we can also use them to specify the elements we want to illustrate this principle, we’ll work with the following page:

First paragraph.

Second paragraph.


First outer paragraph.

Second outer paragraph.
We can access the above document at the URL. Let’s first download the page and create a BeautifulSoup object:page = (“)
soup = BeautifulSoup(ntent, ”)
soup
A simple example page<br />



Now, we can use the find_all method to search for items by class or by id. In the below example, we’ll search for any p tag that has the class nd_all(‘p’, class_=’outer-text’)
[

First outer paragraph.

,

Second outer paragraph.

]In the below example, we’ll look for any tag that has the class nd_all(class_=”outer-text”)

,

]We can also search for elements by nd_all(id=”first”)
[

]Using CSS SelectorsWe can also search for items using CSS selectors. These selectors are how the CSS language allows developers to specify HTML tags to style. Here are some examples:p a — finds all a tags inside of a p p a — finds all a tags inside of a p tag inside of a body body — finds all body tags inside of an html — finds all p tags with a class of outer-text. p#first — finds all p tags with an id of — finds any p tags with a class of outer-text inside of a body can learn more about CSS selectors autifulSoup objects support searching a page via CSS selectors using the select method. We can use CSS selectors to find all the p tags in our page that are inside of a div like (“div p”)

,

]Note that the select method above returns a list of BeautifulSoup objects, just like find and wnloading weather dataWe now know enough to proceed with extracting information about the local weather from the National Weather Service website! The first step is to find the page we want to scrape. We’ll extract weather information about downtown San Francisco from this page. Specifically, let’s extract data about the extended we can see from the image, the page has information about the extended forecast for the next week, including time of day, temperature, and a brief description of the conditions. Exploring page structure with Chrome DevToolsThe first thing we’ll need to do is inspect the page using Chrome Devtools. If you’re using another browser, Firefox and Safari have can start the developer tools in Chrome by clicking View -> Developer -> Developer Tools. You should end up with a panel at the bottom of the browser like what you see below. Make sure the Elements panel is highlighted:Chrome Developer ToolsThe elements panel will show you all the HTML tags on the page, and let you navigate through them. It’s a really handy feature! By right clicking on the page near where it says “Extended Forecast”, then clicking “Inspect”, we’ll open up the tag that contains the text “Extended Forecast” in the elements panel:The extended forecast textWe can then scroll up in the elements panel to find the “outermost” element that contains all of the text that corresponds to the extended forecasts. In this case, it’s a div tag with the id seven-day-forecast:The div that contains the extended forecast we click around on the console, and explore the div, we’ll discover that each forecast item (like “Tonight”, “Thursday”, and “Thursday Night”) is contained in a div with the class to Start Scraping! We now know enough to download the page and start parsing it. In the below code, we will:Download the web page containing the a BeautifulSoup class to parse the the div with id seven-day-forecast, and assign to seven_dayInside seven_day, find each individual forecast item. Extract and print the first forecast = (“)
seven_day = (id=”seven-day-forecast”)
forecast_items = nd_all(class_=”tombstone-container”)
tonight = forecast_items[0]
print(ettify())

Tonight


Tonight: Mostly clear, with a low around 49. West northwest wind 12 to 17 mph decreasing to 6 to 11 mph after midnight. Winds could gust as high as 23 mph.

Mostly Clear

Low: 49 °F

Extracting information from the pageAs we can see, inside the forecast item tonight is all the information we want. There are four pieces of information we can extract:The name of the forecast item — in this case, description of the conditions — this is stored in the title property of img. A short description of the conditions — in this case, Mostly temperature low — in this case, 49 ’ll extract the name of the forecast item, the short description, and the temperature first, since they’re all similar:period = (class_=”period-name”). get_text()
short_desc = (class_=”short-desc”). get_text()
temp = (class_=”temp”). get_text()
print(period)
print(short_desc)
print(temp)
Low: 49 °FNow, we can extract the title attribute from the img tag. To do this, we just treat the BeautifulSoup object like a dictionary, and pass in the attribute we want as a key:img = (“img”)
desc = img[‘title’]
print(desc)
Tonight: Mostly clear, with a low around 49. Extracting all the information from the pageNow that we know how to extract each individual piece of information, we can combine our knowledge with CSS selectors and list comprehensions to extract everything at the below code, we will:Select all items with the class period-name inside an item with the class tombstone-container in a list comprehension to call the get_text method on each BeautifulSoup riod_tags = (“. tombstone-container “)
periods = [t_text() for pt in period_tags]
periods
[‘Tonight’,
‘Thursday’,
‘ThursdayNight’,
‘Friday’,
‘FridayNight’,
‘Saturday’,
‘SaturdayNight’,
‘Sunday’,
‘SundayNight’]As we can see above, our technique gets us each of the period names, in order. We can apply the same technique to get the other three fields:short_descs = [t_text() for sd in (“. tombstone-container “)]
temps = [t_text() for t in (“. tombstone-container “)]
descs = [d[“title”] for d in (“. tombstone-container img”)]print(short_descs)print(temps)print(descs)
[‘Mostly Clear’, ‘Sunny’, ‘Mostly Clear’, ‘Sunny’, ‘Slight ChanceRain’, ‘Rain Likely’, ‘Rain Likely’, ‘Rain Likely’, ‘Chance Rain’]
[‘Low: 49 °F’, ‘High: 63 °F’, ‘Low: 50 °F’, ‘High: 67 °F’, ‘Low: 57 °F’, ‘High: 64 °F’, ‘Low: 57 °F’, ‘High: 64 °F’, ‘Low: 55 °F’]
[‘Tonight: Mostly clear, with a low around 49. ‘, ‘Thursday: Sunny, with a high near 63. North wind 3 to 5 mph. ‘, ‘Thursday Night: Mostly clear, with a low around 50. Light and variable wind becoming east southeast 5 to 8 mph after midnight. ‘, ‘Friday: Sunny, with a high near 67. Southeast wind around 9 mph. ‘, ‘Friday Night: A 20 percent chance of rain after 11pm. Partly cloudy, with a low around 57. South southeast wind 13 to 15 mph, with gusts as high as 20 mph. New precipitation amounts of less than a tenth of an inch possible. ‘, ‘Saturday: Rain likely. Cloudy, with a high near 64. Chance of precipitation is 70%. New precipitation amounts between a quarter and half of an inch possible. ‘, ‘Saturday Night: Rain likely. Cloudy, with a low around 57. Chance of precipitation is 60%. ‘, ‘Sunday: Rain likely. ‘, ‘Sunday Night: A chance of rain. Mostly cloudy, with a low around 55. ‘]Combining our data into a Pandas DataframeWe can now combine the data into a Pandas DataFrame and analyze it. A DataFrame is an object that can store tabular data, making data analysis easy. If you want to learn more about Pandas, check out our free to start course order to do this, we’ll call the DataFrame class, and pass in each list of items that we have. We pass them in as part of a dictionary key will become a column in the DataFrame, and each list will become the values in the column:import pandas as pd
weather = Frame({
“period”: periods,
“short_desc”: short_descs,
“temp”: temps,
“desc”:descs})
weather
desc
period
short_desc
temp
0
Tonight: Mostly clear, with a low around 49. W…
1
Thursday: Sunny, with a high near 63. North wi…
Thursday
Sunny
High: 63 °F
2
Thursday Night: Mostly clear, with a low aroun…
ThursdayNight
Low: 50 °F
3
Friday: Sunny, with a high near 67. Southeast …
Friday
High: 67 °F
4
Friday Night: A 20 percent chance of rain afte…
FridayNight
Slight ChanceRain
Low: 57 °F
5
Saturday: Rain likely. Cloudy, with a high ne…
Saturday
Rain Likely
High: 64 °F
6
Saturday Night: Rain likely. Cloudy, with a l…
SaturdayNight
7
Sunday: Rain likely. Cloudy, with a high near…
Sunday
8
Sunday Night: A chance of rain. Mostly cloudy…
SundayNight
Chance Rain
Low: 55 °F
We can now do some analysis on the data. For example, we can use a regular expression and the method to pull out the numeric temperature values:temp_nums = weather[“temp”](“(? Pd+)”, expand=False)
weather[“temp_num”] = (‘int’)
temp_nums
0 49
1 63
2 50
3 67
4 57
5 64
6 57
7 64
8 55
Name: temp_num, dtype: objectWe could then find the mean of all the high and low temperatures:weather[“temp_num”]()
58. 444444444444443We could also only select the rows that happen at night:is_night = weather[“temp”](“Low”)
weather[“is_night”] = is_night
is_night
0 True
1 False
2 True
3 False
4 True
5 False
6 True
7 False
8 True
Name: temp, dtype: boolweather[is_night]
Name: temp, dtype: bool
temp_num
49
True
50
57
55
Next Steps For This Web Scraping ProjectIf you’ve made it this far, congratulations! You should now have a good understanding of how to scrape web pages and extract data. Of course, there’s still a lot more to learn! If you want to go further, a good next step would be to pick a site and try some web scraping on your own. Some good examples of data to scrape are:News articlesSports scoresWeather forecastsStock pricesOnline retailer pricesYou may also want to keep scraping the National Weather Service, and see what other data you can extract from the page, or about your own ternatively, if you want to take your web scraping skills to the next level, you can check out our interactive course, which covers both the basics of web scraping and using Python to connect to APIs. With those two skills under your belt, you’ll be able to collect lots of unique and interesting datasets from sites all over the web! Learn to scrape the web with Python, right in your browser! Our interactive APIs and Web Scraping in Python skill path will help you learn the skills you need to unlock new worlds of data with Python. (No credit card required! )beginner, data mining, python, python tutorials, scraping, tutorial, Tutorials, web scraping
Beautiful Soup 4.9.0 documentation - Crummy

Beautiful Soup 4.9.0 documentation – Crummy

Beautiful Soup is a
Python library for pulling data out of HTML and XML files. It works
with your favorite parser to provide idiomatic ways of navigating,
searching, and modifying the parse tree. It commonly saves programmers
hours or days of work.
These instructions illustrate all major features of Beautiful Soup 4,
with examples. I show you what the library is good for, how it works,
how to use it, how to make it do what you want, and what to do when it
violates your expectations.
This document covers Beautiful Soup version 4. 9. 3. The examples in
this documentation should work the same way in Python 2. 7 and Python
3. 8.
You might be looking for the documentation for Beautiful Soup 3.
If so, you should know that Beautiful Soup 3 is no longer being
developed and that support for it will be dropped on or after December
31, 2020. If you want to learn about the differences between Beautiful
Soup 3 and Beautiful Soup 4, see Porting code to BS4.
This documentation has been translated into other languages by
Beautiful Soup users:
这篇文档当然还有中文版.
このページは日本語で利用できます(外部リンク)
이 문서는 한국어 번역도 가능합니다.
Este documento também está disponível em Português do Brasil.
Эта документация доступна на русском языке.
Getting help¶
If you have questions about Beautiful Soup, or run into problems,
send mail to the discussion group. If
your problem involves parsing an HTML document, be sure to mention
what the diagnose() function says about
that document.
Here’s an HTML document I’ll be using as an example throughout this
document. It’s part of a story from Alice in Wonderland:
html_doc = “””The Dormouse’s story

The Dormouse’s story

Once upon a time there were three little sisters; and their names were
Elsie,
Lacie and
Tillie;
and they lived at the bottom of a well.

“””
Running the “three sisters” document through Beautiful Soup gives us a
BeautifulSoup object, which represents the document as a nested
data structure:
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc, ”)
print(ettify())
#
#
# <br /> # The Dormouse’s story<br /> #
#
#
#

#
#

#

#

# Once upon a time there were three little sisters; and their names were
#
# Elsie
#

#,
#
# Lacie
# and
#

# Tillie
#; and they lived at the bottom of a well.
#…
#
#
Here are some simple ways to navigate that data structure:
# The Dormouse’s story
# u’title’
# u’The Dormouse’s story’
# u’head’
soup. p
#

The Dormouse’s story

soup. p[‘class’]
soup. a
#
Elsie
nd_all(‘a’)
# [Elsie,
# Lacie,
# Tillie]
(id=”link3″)
# Tillie
One common task is extracting all the URLs found within a page’s tags:
for link in nd_all(‘a’):
print((‘href’))
# # #
Another common task is extracting all the text from a page:
print(t_text())
#
# Elsie,
# Lacie and
# Tillie;
# and they lived at the bottom of a well.
Does this look like what you need? If so, read on.
If you’re using a recent version of Debian or Ubuntu Linux, you can
install Beautiful Soup with the system package manager:
$ apt-get install python-bs4 (for Python 2)
$ apt-get install python3-bs4 (for Python 3)
Beautiful Soup 4 is published through PyPi, so if you can’t install it
with the system packager, you can install it with easy_install or
pip. The package name is beautifulsoup4, and the same package
works on Python 2 and Python 3. Make sure you use the right version of
pip or easy_install for your Python version (these may be named
pip3 and easy_install3 respectively if you’re using Python 3).
$ easy_install beautifulsoup4
$ pip install beautifulsoup4
(The BeautifulSoup package is not what you want. That’s
the previous major release, Beautiful Soup 3. Lots of software uses
BS3, so it’s still available, but if you’re writing new code you
should install beautifulsoup4. )
If you don’t have easy_install or pip installed, you can
download the Beautiful Soup 4 source tarball and
install it with
$ python install
If all else fails, the license for Beautiful Soup allows you to
package the entire library with your application. You can download the
tarball, copy its bs4 directory into your application’s codebase,
and use Beautiful Soup without installing it at all.
I use Python 2. 7 and Python 3. 8 to develop Beautiful Soup, but it
should work with other recent versions.
Problems after installation¶
Beautiful Soup is packaged as Python 2 code. When you install it for
use with Python 3, it’s automatically converted to Python 3 code. If
you don’t install the package, the code won’t be converted. There have
also been reports on Windows machines of the wrong version being
installed.
If you get the ImportError “No module named HTMLParser”, your
problem is that you’re running the Python 2 version of the code under
Python 3.
If you get the ImportError “No module named ”, your
problem is that you’re running the Python 3 version of the code under
Python 2.
In both cases, your best bet is to completely remove the Beautiful
Soup installation from your system (including any directory created
when you unzipped the tarball) and try the installation again.
If you get the SyntaxError “Invalid syntax” on the line
ROOT_TAG_NAME = u'[document]’, you need to convert the Python 2
code to Python 3. You can do this either by installing the package:
$ python3 install
or by manually running Python’s 2to3 conversion script on the
bs4 directory:
$ 2to3-3. 2 -w bs4
Installing a parser¶
Beautiful Soup supports the HTML parser included in Python’s standard
library, but it also supports a number of third-party Python parsers.
One is the lxml parser. Depending on your setup,
you might install lxml with one of these commands:
$ apt-get install python-lxml
$ easy_install lxml
$ pip install lxml
Another alternative is the pure-Python html5lib parser, which parses HTML the way a
web browser does. Depending on your setup, you might install html5lib
with one of these commands:
$ apt-get install python-html5lib
$ easy_install html5lib
$ pip install html5lib
This table summarizes the advantages and disadvantages of each parser library:
Parser
Typical usage
Advantages
Disadvantages
Python’s
BeautifulSoup(markup, “”)
Batteries included
Decent speed
Lenient (As of Python 2. 7. 3
and 3. 2. )
Not as fast as lxml,
less lenient than
html5lib.
lxml’s HTML parser
BeautifulSoup(markup, “lxml”)
Very fast
Lenient
External C dependency
lxml’s XML parser
BeautifulSoup(markup, “lxml-xml”)
BeautifulSoup(markup, “xml”)
The only currently supported
XML parser
html5lib
BeautifulSoup(markup, “html5lib”)
Extremely lenient
Parses pages the same way a
web browser does
Creates valid HTML5
Very slow
External Python
dependency
If you can, I recommend you install and use lxml for speed. If you’re
using a very old version of Python – earlier than 2. 3 or 3. 2 –
it’s essential that you install lxml or html5lib. Python’s built-in
HTML parser is just not very good in those old versions.
Note that if a document is invalid, different parsers will generate
different Beautiful Soup trees for it. See Differences
between parsers for details.
To parse a document, pass it into the BeautifulSoup
constructor. You can pass in a string or an open filehandle:
with open(“”) as fp:
soup = BeautifulSoup(fp, ”)
soup = BeautifulSoup(“a web page“, ”)
First, the document is converted to Unicode, and HTML entities are
converted to Unicode characters:
print(BeautifulSoup(“Sacré bleu! “, “”))
# Sacré bleu!
Beautiful Soup then parses the document using the best available
parser. It will use an HTML parser unless you specifically tell it to
use an XML parser. (See Parsing XML. )
Beautiful Soup transforms a complex HTML document into a complex tree
of Python objects. But you’ll only ever have to deal with about four
kinds of objects: Tag, NavigableString, BeautifulSoup,
and Comment.
Tag¶
A Tag object corresponds to an XML or HTML tag in the original document:
soup = BeautifulSoup(‘Extremely bold‘, ”)
tag = soup. b
type(tag)
#
Tags have a lot of attributes and methods, and I’ll cover most of them
in Navigating the tree and Searching the tree. For now, the most
important features of a tag are its name and attributes.
Name¶
Every tag has a name, accessible as
If you change a tag’s name, the change will be reflected in any HTML
markup generated by Beautiful Soup:
= “blockquote”
tag
#

Extremely bold

Attributes¶
A tag may have any number of attributes. The tag has an attribute “id” whose value is
“boldest”. You can access a tag’s attributes by treating the tag like
a dictionary:
tag = BeautifulSoup(‘bold‘, ”). b
tag[‘id’]
# ‘boldest’
You can access that dictionary directly as
# {‘id’: ‘boldest’}
You can add, remove, and modify a tag’s attributes. Again, this is
done by treating the tag as a dictionary:
tag[‘id’] = ‘verybold’
tag[‘another-attribute’] = 1
#
del tag[‘id’]
del tag[‘another-attribute’]
# bold
# KeyError: ‘id’
(‘id’)
# None
Multi-valued attributes¶
HTML 4 defines a few attributes that can have multiple values. HTML 5
removes a couple of them, but defines a few more. The most common
multi-valued attribute is class (that is, a tag can have more than
one CSS class). Others include rel, rev, accept-charset,
headers, and accesskey. Beautiful Soup presents the value(s)
of a multi-valued attribute as a list:
css_soup = BeautifulSoup(‘

‘, ”)
css_soup. p[‘class’]
# [‘body’]
css_soup = BeautifulSoup(‘

‘, ”)
# [‘body’, ‘strikeout’]
If an attribute looks like it has more than one value, but it’s not
a multi-valued attribute as defined by any version of the HTML
standard, Beautiful Soup will leave the attribute alone:
id_soup = BeautifulSoup(‘

‘, ”)
id_soup. p[‘id’]
# ‘my id’
When you turn a tag back into a string, multiple attribute values are
consolidated:
rel_soup = BeautifulSoup(‘

Back to the homepage

‘, ”)
rel_soup. a[‘rel’]
# [‘index’]
rel_soup. a[‘rel’] = [‘index’, ‘contents’]
print(rel_soup. p)
#

Back to the homepage

You can disable this by passing multi_valued_attributes=None as a
keyword argument into the BeautifulSoup constructor:
no_list_soup = BeautifulSoup(‘

‘, ”, multi_valued_attributes=None)
no_list_soup. p[‘class’]
# ‘body strikeout’
You can use get_attribute_list to get a value that’s always a
list, whether or not it’s a multi-valued atribute:
t_attribute_list(‘id’)
# [“my id”]
If you parse a document as XML, there are no multi-valued attributes:
xml_soup = BeautifulSoup(‘

‘, ‘xml’)
xml_soup. p[‘class’]
Again, you can configure this using the multi_valued_attributes argument:
class_is_multi= { ‘*’: ‘class’}
xml_soup = BeautifulSoup(‘

‘, ‘xml’, multi_valued_attributes=class_is_multi)
You probably won’t need to do this, but if you do, use the defaults as
a guide. They implement the rules described in the HTML specification:
from er import builder_registry
(‘html’). DEFAULT_CDATA_LIST_ATTRIBUTES
NavigableString¶
A string corresponds to a bit of text within a tag. Beautiful Soup
uses the NavigableString class to contain these bits of text:
# ‘Extremely bold’
type()
#
A NavigableString is just like a Python Unicode string, except
that it also supports some of the features described in Navigating
the tree and Searching the tree. You can convert a
NavigableString to a Unicode string with unicode() (in
Python 2) or str (in Python 3):
unicode_string = str()
unicode_string
type(unicode_string)
#
You can’t edit a string in place, but you can replace one string with
another, using replace_with():
(“No longer bold”)
# No longer bold
NavigableString supports most of the features described in
Navigating the tree and Searching the tree, but not all of
them. In particular, since a string can’t contain anything (the way a
tag may contain a string or another tag), strings don’t support the. contents or attributes, or the find() method.
If you want to use a NavigableString outside of Beautiful Soup,
you should call unicode() on it to turn it into a normal Python
Unicode string. If you don’t, your string will carry around a
reference to the entire Beautiful Soup parse tree, even when you’re
done using Beautiful Soup. This is a big waste of memory.
BeautifulSoup¶
The BeautifulSoup object represents the parsed document as a
whole. For most purposes, you can treat it as a Tag
object. This means it supports most of the methods described in
Navigating the tree and Searching the tree.
You can also pass a BeautifulSoup object into one of the methods
defined in Modifying the tree, just as you would a Tag. This
lets you do things like combine two parsed documents:
doc = BeautifulSoup(“INSERT FOOTER HEREHere’s the footer

“, “xml”)
(text=”INSERT FOOTER HERE”). replace_with(footer)
# ‘INSERT FOOTER HERE’
print(doc)
#
#

Here’s the footer


Since the BeautifulSoup object doesn’t correspond to an actual
HTML or XML tag, it has no name and no attributes. But sometimes it’s
useful to look at its, so it’s been given the special
“[document]”:
Here’s the “Three sisters” HTML document again:
html_doc = “””
The Dormouse’s story
I’ll use this as an example to show you how to move from one part of
a document to another.
Going down¶
Tags may contain strings and other tags. These elements are the tag’s
children. Beautiful Soup provides a lot of different attributes for
navigating and iterating over a tag’s children.
Note that Beautiful Soup strings don’t support any of these
attributes, because a string can’t have children.
Navigating using tag names¶
The simplest way to navigate the parse tree is to say the name of the
tag you want. If you want the tag, just say
# The Dormouse’s story
You can do use this trick again and again to zoom in on a certain part
of the parse tree. This code gets the first tag beneath the tag:
# The Dormouse’s story
Using a tag name as an attribute will give you only the first tag by that
name:
If you need to get all the tags, or anything more complicated
than the first tag with a certain name, you’ll need to use one of the
methods described in Searching the tree, such as find_all():
#
Tillie]. contents and. children¶
A tag’s children are available in a list called. contents:
head_tag =
head_tag
ntents
# [The Dormouse’s story]
title_tag = ntents[0]
title_tag
# [‘The Dormouse’s story’]
The BeautifulSoup object itself has children. In this case, the
tag is the child of the BeautifulSoup object. :
len(ntents)
# 1
ntents[0]
# ‘html’
A string does not have. contents, because it can’t contain
anything:
text = ntents[0]
# AttributeError: ‘NavigableString’ object has no attribute ‘contents’
Instead of getting them as a list, you can iterate over a tag’s
children using the. children generator:
for child in ildren:
print(child)
# The Dormouse’s story. descendants¶
The. children attributes only consider a tag’s
direct children. For instance, the tag has a single direct
child–the tag:<br /> But the <title> tag itself has a child: the string “The Dormouse’s<br /> story”. There’s a sense in which that string is also a child of the<br /> <head> tag. The. descendants attribute lets you iterate over all<br /> of a tag’s children, recursively: its direct children, the children of<br /> its direct children, and so on:<br /> for child in scendants:<br /> The <head> tag has only one child, but it has two descendants: the<br /> <title> tag and the <title> tag’s child. The BeautifulSoup object<br /> only has one direct child (the <html> tag), but it has a whole lot of<br /> descendants:<br /> len(list(ildren))<br /> len(list(scendants))<br /> # 26<br /> ¶<br /> If a tag has only one child, and that child is a NavigableString,<br /> the child is made available as<br /> # ‘The Dormouse’s story’<br /> If a tag’s only child is another tag, and that tag has a, then the parent tag is considered to have the same<br /> as its child:<br /> If a tag contains more than one thing, then it’s not clear what<br /> should refer to, so is defined to be<br /> None:<br /> print()<br /> # None. strings and stripped_strings¶<br /> If there’s more than one thing inside a tag, you can still look at<br /> just the strings. Use the. strings generator:<br /> for string in rings:<br /> print(repr(string))<br /> ‘\n’<br /> # “The Dormouse’s story”<br /> # ‘\n’<br /> # ‘Once upon a time there were three little sisters; and their names were\n’<br /> # ‘Elsie’<br /> # ‘, \n’<br /> # ‘Lacie’<br /> # ‘ and\n’<br /> # ‘Tillie’<br /> # ‘;\nand they lived at the bottom of a well. ‘<br /> # ‘… ‘<br /> These strings tend to have a lot of extra whitespace, which you can<br /> remove by using the. stripped_strings generator instead:<br /> for string in ripped_strings:<br /> # ‘Once upon a time there were three little sisters; and their names were’<br /> # ‘, ‘<br /> # ‘and’<br /> # ‘;\n and they lived at the bottom of a well. ‘<br /> Here, strings consisting entirely of whitespace are ignored, and<br /> whitespace at the beginning and end of strings is removed.<br /> Going up¶<br /> Continuing the “family tree” analogy, every tag and every string has a<br /> parent: the tag that contains it.<br /> You can access an element’s parent with the attribute. In<br /> the example “three sisters” document, the <head> tag is the parent<br /> of the <title> tag:<br /> title_tag =<br /> The title string itself has a parent: the <title> tag that contains<br /> it:<br /> The parent of a top-level tag like <html> is the BeautifulSoup object<br /> itself:<br /> html_tag =<br /> # <class 'autifulSoup'><br /> And the of a BeautifulSoup object is defined as None:<br /> # None. parents¶<br /> You can iterate over all of an element’s parents with. parents. This example uses. parents to travel from an <a> tag<br /> buried deep within the document, to the very top of the document:<br /> link = soup. a<br /> link<br /> for parent in rents:<br /> # p<br /> # body<br /> # html<br /> # [document]<br /> Going sideways¶<br /> Consider a simple document like this:<br /> sibling_soup = BeautifulSoup(“<a><b>text1</b><c>text2</c></b></a>“, ”)<br /> # <a><br /> # text1<br /> # <c><br /> # text2<br /> # </c><br /> The <b> tag and the <c> tag are at the same level: they’re both direct<br /> children of the same tag. We call them siblings. When a document is<br /> pretty-printed, siblings show up at the same indentation level. You<br /> can also use this relationship in the code you write.. next_sibling and. previous_sibling¶<br /> You can use. previous_sibling to navigate<br /> between page elements that are on the same level of the parse tree:<br /> xt_sibling<br /> # <c>text2</c><br /> evious_sibling<br /> # <b>text1</b><br /> The <b> tag has a. next_sibling, but no. previous_sibling,<br /> because there’s nothing before the <b> tag on the same level of the<br /> tree. For the same reason, the <c> tag has a. previous_sibling<br /> but no. next_sibling:<br /> print(evious_sibling)<br /> print(xt_sibling)<br /> The strings “text1” and “text2” are not siblings, because they don’t<br /> have the same parent:<br /> # ‘text1’<br /> In real documents, the. next_sibling or. previous_sibling of a<br /> tag will usually be a string containing whitespace. Going back to the<br /> “three sisters” document:<br /> # <a href=" class="sister" id="link1">Elsie</a><br /> # <a href=" class="sister" id="link2">Lacie</a><br /> # <a href=" class="sister" id="link3">Tillie</a><br /> You might think that the. next_sibling of the first <a> tag would<br /> be the second <a> tag. But actually, it’s a string: the comma and<br /> newline that separate the first <a> tag from the second:<br /> # ‘, \n ‘<br /> The second <a> tag is actually the. next_sibling of the comma:<br /> # <a class="sister" href=" id="link2">Lacie</a>. next_siblings and. previous_siblings¶<br /> You can iterate over a tag’s siblings with. next_siblings or. previous_siblings:<br /> for sibling in xt_siblings:<br /> print(repr(sibling))<br /> # <a class="sister" href=" id="link2">Lacie</a><br /> # ‘; and they lived at the bottom of a well. ‘<br /> for sibling in (id=”link3″). previous_siblings:<br /> Going back and forth¶<br /> Take a look at the beginning of the “three sisters” document:<br /> # <html><head><title>The Dormouse’s story
An HTML parser takes this string of characters and turns it into a
series of events: “open an tag”, “open a tag”, “open a
tag”, “add a string”, “close the <title> tag”, “open a </p> <p> tag”, and so on. Beautiful Soup offers tools for reconstructing the<br /> initial parse of the document.. next_element and. previous_element¶<br /> The. next_element attribute of a string or tag points to whatever<br /> was parsed immediately afterwards. It might be the same as. next_sibling, but it’s usually drastically different.<br /> Here’s the final <a> tag in the “three sisters” document. Its. next_sibling is a string: the conclusion of the sentence that was<br /> interrupted by the start of the <a> tag. :<br /> last_a_tag = (“a”, id=”link3″)<br /> last_a_tag<br /> But the. next_element of that <a> tag, the thing that was parsed<br /> immediately after the <a> tag, is not the rest of that sentence:<br /> it’s the word “Tillie”:<br /> xt_element<br /> That’s because in the original markup, the word “Tillie” appeared<br /> before that semicolon. The parser encountered an <a> tag, then the<br /> word “Tillie”, then the closing </a> tag, then the semicolon and rest of<br /> the sentence. The semicolon is on the same level as the <a> tag, but the<br /> word “Tillie” was encountered first.<br /> The. previous_element attribute is the exact opposite of. next_element. It points to whatever element was parsed<br /> immediately before this one:<br /> evious_element<br /> # <a class="sister" href=" id="link3">Tillie</a>. next_elements and. previous_elements¶<br /> You should get the idea by now. You can use these iterators to move<br /> forward or backward in the document as it was parsed:<br /> for element in xt_elements:<br /> print(repr(element))<br /> # </p> <p class="story">… </p> <p>Beautiful Soup defines a lot of methods for searching the parse tree,<br /> but they’re all very similar. I’m going to spend a lot of time explaining<br /> the two most popular methods: find() and find_all(). The other<br /> methods take almost exactly the same arguments, so I’ll just cover<br /> them briefly.<br /> Once again, I’ll be using the “three sisters” document as an example:<br /> By passing in a filter to an argument like find_all(), you can<br /> zoom in on the parts of the document you’re interested in.<br /> Kinds of filters¶<br /> Before talking in detail about find_all() and similar methods, I<br /> want to show examples of different filters you can pass into these<br /> methods. These filters show up again and again, throughout the<br /> search API. You can use them to filter based on a tag’s name,<br /> on its attributes, on the text of a string, or on some combination of<br /> these.<br /> A string¶<br /> The simplest filter is a string. Pass a string to a search method and<br /> Beautiful Soup will perform a match against that exact string. This<br /> code finds all the <b> tags in the document:<br /> nd_all(‘b’)<br /> # [<b>The Dormouse’s story</b>]<br /> If you pass in a byte string, Beautiful Soup will assume the string is<br /> encoded as UTF-8. You can avoid this by passing in a Unicode string instead.<br /> A regular expression¶<br /> If you pass in a regular expression object, Beautiful Soup will filter<br /> against that regular expression using its search() method. This code<br /> finds all the tags whose names start with the letter “b”; in this<br /> case, the <body> tag and the <b> tag:<br /> import re<br /> for tag in nd_all(mpile(“^b”)):<br /> # b<br /> This code finds all the tags whose names contain the letter ‘t’:<br /> for tag in nd_all(mpile(“t”)):<br /> # title<br /> A list¶<br /> If you pass in a list, Beautiful Soup will allow a string match<br /> against any item in that list. This code finds all the <a> tags<br /> and all the <b> tags:<br /> nd_all([“a”, “b”])<br /> # [<b>The Dormouse’s story</b>,<br /> # <a class="sister" href=" id="link1">Elsie</a>,<br /> True¶<br /> The value True matches everything it can. This code finds all<br /> the tags in the document, but none of the text strings:<br /> for tag in nd_all(True):<br /> # head<br /> # a<br /> A function¶<br /> If none of the other matches work for you, define a function that<br /> takes an element as its only argument. The function should return<br /> True if the argument matches, and False otherwise.<br /> Here’s a function that returns True if a tag defines the “class”<br /> attribute but doesn’t define the “id” attribute:<br /> def has_class_but_no_id(tag):<br /> return tag. has_attr(‘class’) and not tag. has_attr(‘id’)<br /> Pass this function into find_all() and you’ll pick up all the </p> <p> tags:<br /> nd_all(has_class_but_no_id)<br /> # [</p> <p class="title"><b>The Dormouse’s story</b></p> <p>,<br /> # </p> <p class="story">Once upon a time there were…bottom of a well. </p> <p>,<br /> # </p> <p class="story">… </p> <p>]<br /> This function only picks up the </p> <p> tags. It doesn’t pick up the <a><br /> tags, because those tags define both “class” and “id”. It doesn’t pick<br /> up tags like <html> and <title>, because those tags don’t define<br /> “class”.<br /> If you pass in a function to filter on a specific attribute like<br /> href, the argument passed into the function will be the attribute<br /> value, not the whole tag. Here’s a function that finds all a tags<br /> whose href attribute does not match a regular expression:<br /> def not_lacie(href):<br /> return href and not mpile(“lacie”)(href)<br /> nd_all(href=not_lacie)<br /> The function can be as complicated as you need it to be. Here’s a<br /> function that returns True if a tag is surrounded by string<br /> objects:<br /> from bs4 import NavigableString<br /> def surrounded_by_strings(tag):<br /> return (isinstance(xt_element, NavigableString)<br /> and isinstance(evious_element, NavigableString))<br /> for tag in nd_all(surrounded_by_strings):<br /> Now we’re ready to look at the search methods in detail.<br /> find_all()¶<br /> Method signature: find_all(name, attrs, recursive, string, limit, **kwargs)<br /> The find_all() method looks through a tag’s descendants and<br /> retrieves all descendants that match your filters. I gave several<br /> examples in Kinds of filters, but here are a few more:<br /> nd_all(“title”)<br /> nd_all(“p”, “title”)<br /> # [</p> <p class="title"><b>The Dormouse’s story</b></p> <p>]<br /> nd_all(“a”)<br /> nd_all(id=”link2″)<br /> # [<a class="sister" href=" id="link2">Lacie</a>]<br /> (mpile(“sisters”))<br /> Some of these should look familiar, but others are new. What does it<br /> mean to pass in a value for string, or id? Why does<br /> find_all(“p”, “title”) find a </p> <p> tag with the CSS class “title”?<br /> Let’s look at the arguments to find_all().<br /> The name argument¶<br /> Pass in a value for name and you’ll tell Beautiful Soup to only<br /> consider tags with certain names. Text strings will be ignored, as<br /> will tags whose names that don’t match.<br /> This is the simplest usage:<br /> Recall from Kinds of filters that the value to name can be a<br /> string, a regular expression, a list, a function, or the value<br /> True.<br /> The keyword arguments¶<br /> Any argument that’s not recognized will be turned into a filter on one<br /> of a tag’s attributes. If you pass in a value for an argument called id,<br /> Beautiful Soup will filter against each tag’s ‘id’ attribute:<br /> nd_all(id=’link2′)<br /> If you pass in a value for href, Beautiful Soup will filter<br /> against each tag’s ‘href’ attribute:<br /> nd_all(mpile(“elsie”))<br /> # [<a class="sister" href=" id="link1">Elsie</a>]<br /> You can filter an attribute based on a string, a regular<br /> expression, a list, a function, or the value True.<br /> This code finds all tags whose id attribute has a value,<br /> regardless of what the value is:<br /> nd_all(id=True)<br /> You can filter multiple attributes at once by passing in more than one<br /> keyword argument:<br /> nd_all(mpile(“elsie”), id=’link1′)<br /> Some attributes, like the data-* attributes in HTML 5, have names that<br /> can’t be used as the names of keyword arguments:<br /> data_soup = BeautifulSoup(‘</p> <div data-foo="value">foo! </div> <p>‘, ”)<br /> nd_all(data-foo=”value”)<br /> # SyntaxError: keyword can’t be an expression<br /> You can use these attributes in searches by putting them into a<br /> dictionary and passing the dictionary into find_all() as the<br /> attrs argument:<br /> nd_all(attrs={“data-foo”: “value”})<br /> # [</p> <div data-foo="value">foo! </div> <p>]<br /> You can’t use a keyword argument to search for HTML’s ‘name’ element,<br /> because Beautiful Soup uses the name argument to contain the name<br /> of the tag itself. Instead, you can give a value to ‘name’ in the<br /> name_soup = BeautifulSoup(‘<input name="email"/>‘, ”)<br /> nd_all(name=”email”)<br /> # []<br /> nd_all(attrs={“name”: “email”})<br /> # [<input name="email"/>]<br /> Searching by CSS class¶<br /> It’s very useful to search for a tag that has a certain CSS class, but<br /> the name of the CSS attribute, “class”, is a reserved word in<br /> Python. Using class as a keyword argument will give you a syntax<br /> error. As of Beautiful Soup 4. 1. 2, you can search by CSS class using<br /> the keyword argument class_:<br /> nd_all(“a”, class_=”sister”)<br /> As with any keyword argument, you can pass class_ a string, a regular<br /> expression, a function, or True:<br /> nd_all(mpile(“itl”))<br /> def has_six_characters(css_class):<br /> return css_class is not None and len(css_class) == 6<br /> nd_all(class_=has_six_characters)<br /> Remember that a single tag can have multiple<br /> values for its “class” attribute. When you search for a tag that<br /> matches a certain CSS class, you’re matching against any of its CSS<br /> classes:<br /> nd_all(“p”, class_=”strikeout”)<br /> # [</p> <p class="body strikeout"> <p>]<br /> nd_all(“p”, class_=”body”)<br /> You can also search for the exact string value of the class attribute:<br /> nd_all(“p”, class_=”body strikeout”)<br /> But searching for variants of the string value won’t work:<br /> nd_all(“p”, class_=”strikeout body”)<br /> If you want to search for tags that match two or more CSS classes, you<br /> should use a CSS selector:<br /> (“p. “)<br /> In older versions of Beautiful Soup, which don’t have the class_<br /> shortcut, you can use the attrs trick mentioned above. Create a<br /> dictionary whose value for “class” is the string (or regular<br /> expression, or whatever) you want to search for:<br /> nd_all(“a”, attrs={“class”: “sister”})<br /> The string argument¶<br /> With string you can search for strings instead of tags. As with<br /> name and the keyword arguments, you can pass in a string, a<br /> regular expression, a list, a function, or the value True.<br /> Here are some examples:<br /> nd_all(string=”Elsie”)<br /> # [‘Elsie’]<br /> nd_all(string=[“Tillie”, “Elsie”, “Lacie”])<br /> # [‘Elsie’, ‘Lacie’, ‘Tillie’]<br /> nd_all(mpile(“Dormouse”))<br /> # [“The Dormouse’s story”, “The Dormouse’s story”]<br /> def is_the_only_string_within_a_tag(s):<br /> “””Return True if this string is the only child of its parent tag. “””<br /> return (s ==)<br /> nd_all(string=is_the_only_string_within_a_tag)<br /> # [“The Dormouse’s story”, “The Dormouse’s story”, ‘Elsie’, ‘Lacie’, ‘Tillie’, ‘… ‘]<br /> Although string is for finding strings, you can combine it with<br /> arguments that find tags: Beautiful Soup will find all tags whose<br /> matches your value for string. This code finds the <a><br /> tags whose is “Elsie”:<br /> nd_all(“a”, string=”Elsie”)<br /> # [<a href=" class="sister" id="link1">Elsie</a>]<br /> The string argument is new in Beautiful Soup 4. 4. 0. In earlier<br /> versions it was called text:<br /> nd_all(“a”, text=”Elsie”)<br /> The limit argument¶<br /> find_all() returns all the tags and strings that match your<br /> filters. This can take a while if the document is large. If you don’t<br /> need all the results, you can pass in a number for limit. This<br /> works just like the LIMIT keyword in SQL. It tells Beautiful Soup to<br /> stop gathering results after it’s found a certain number.<br /> There are three links in the “three sisters” document, but this code<br /> only finds the first two:<br /> nd_all(“a”, limit=2)<br /> # <a class="sister" href=" id="link2">Lacie</a>]<br /> The recursive argument¶<br /> If you call nd_all(), Beautiful Soup will examine all the<br /> descendants of mytag: its children, its children’s children, and<br /> so on. If you only want Beautiful Soup to consider direct children,<br /> you can pass in recursive=False. See the differe<br /> <img decoding="async" src="" alt="Beautiful Soup: Build a Web Scraper With Python" title="Beautiful Soup: Build a Web Scraper With Python" /></p> <h2>Beautiful Soup: Build a Web Scraper With Python</h2> <p>Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Web Scraping With Beautiful Soup and Python<br /> The incredible amount of data on the Internet is a rich resource for any field of research or personal interest. To effectively harvest that data, you’ll need to become skilled at web scraping. The Python libraries requests and Beautiful Soup are powerful tools for the job. If you like to learn with hands-on examples and have a basic understanding of Python and HTML, then this tutorial is for you.<br /> In this tutorial, you’ll learn how to:<br /> Inspect the HTML structure of your target site with your browser’s developer tools<br /> Decipher data encoded in URLs<br /> Use requests and Beautiful Soup for scraping and parsing data from the Web<br /> Step through a web scraping pipeline from start to finish<br /> Build a script that fetches job offers from the Web and displays relevant information in your console<br /> Working through this project will give you the knowledge of the process and tools you need to scrape any static website out there on the World Wide Web. You can download the project source code by clicking on the link below:<br /> Let’s get started!<br /> What Is Web Scraping?<br /> Web scraping is the process of gathering information from the Internet. Even copying and pasting the lyrics of your favorite song is a form of web scraping! However, the words “web scraping” usually refer to a process that involves automation. Some websites don’t like it when automatic scrapers gather their data, while others don’t mind.<br /> If you’re scraping a page respectfully for educational purposes, then you’re unlikely to have any problems. Still, it’s a good idea to do some research on your own and make sure that you’re not violating any Terms of Service before you start a large-scale project.<br /> Reasons for Web Scraping<br /> Say you’re a surfer, both online and in real life, and you’re looking for employment. However, you’re not looking for just any job. With a surfer’s mindset, you’re waiting for the perfect opportunity to roll your way!<br /> There’s a job site that offers precisely the kinds of jobs you want. Unfortunately, a new position only pops up once in a blue moon, and the site doesn’t provide an email notification service. You think about checking up on it every day, but that doesn’t sound like the most fun and productive way to spend your time.<br /> Thankfully, the world offers other ways to apply that surfer’s mindset! Instead of looking at the job site every day, you can use Python to help automate your job search’s repetitive parts. Automated web scraping can be a solution to speed up the data collection process. You write your code once, and it will get the information you want many times and from many pages.<br /> In contrast, when you try to get the information you want manually, you might spend a lot of time clicking, scrolling, and searching, especially if you need large amounts of data from websites that are regularly updated with new content. Manual web scraping can take a lot of time and repetition.<br /> There’s so much information on the Web, and new information is constantly added. You’ll probably be interested in at least some of that data, and much of it is just out there for the taking. Whether you’re actually on the job hunt or you want to download all the lyrics of your favorite artist, automated web scraping can help you accomplish your goals.<br /> Challenges of Web Scraping<br /> The Web has grown organically out of many sources. It combines many different technologies, styles, and personalities, and it continues to grow to this day. In other words, the Web is a hot mess! Because of this, you’ll run into some challenges when scraping the Web:<br /> Variety: Every website is different. While you’ll encounter general structures that repeat themselves, each website is unique and will need personal treatment if you want to extract the relevant information.<br /> Durability: Websites constantly change. Say you’ve built a shiny new web scraper that automatically cherry-picks what you want from your resource of interest. The first time you run your script, it works flawlessly. But when you run the same script only a short while later, you run into a discouraging and lengthy stack of tracebacks!<br /> Unstable scripts are a realistic scenario, as many websites are in active development. Once the site’s structure has changed, your scraper might not be able to navigate the sitemap correctly or find the relevant information. The good news is that many changes to websites are small and incremental, so you’ll likely be able to update your scraper with only minimal adjustments.<br /> However, keep in mind that because the Internet is dynamic, the scrapers you’ll build will probably require constant maintenance. You can set up continuous integration to run scraping tests periodically to ensure that your main script doesn’t break without your knowledge.<br /> An Alternative to Web Scraping: APIs<br /> Some website providers offer application programming interfaces (APIs) that allow you to access their data in a predefined manner. With APIs, you can avoid parsing HTML. Instead, you can access the data directly using formats like JSON and XML. HTML is primarily a way to present content to users visually.<br /> When you use an API, the process is generally more stable than gathering the data through web scraping. That’s because developers create APIs to be consumed by programs rather than by human eyes.<br /> The front-end presentation of a site might change often, but such a change in the website’s design doesn’t affect its API structure. The structure of an API is usually more permanent, which means it’s a more reliable source of the site’s data.<br /> However, APIs can change as well. The challenges of both variety and durability apply to APIs just as they do to websites. Additionally, it’s much harder to inspect the structure of an API by yourself if the provided documentation lacks quality.<br /> The approach and tools you need to gather information using APIs are outside the scope of this tutorial. To learn more about it, check out API Integration in Python.<br /> Scrape the Fake Python Job Site<br /> In this tutorial, you’ll build a web scraper that fetches Python software developer job listings from the Fake Python Jobs site. It’s an example site with fake job postings that you can freely scrape to train your skills. Your web scraper will parse the HTML on the site to pick out the relevant information and filter that content for specific words.<br /> You can scrape any site on the Internet that you can look at, but the difficulty of doing so depends on the site. This tutorial offers you an introduction to web scraping to help you understand the overall process. Then, you can apply this same process for every website you’ll want to scrape.<br /> Throughout the tutorial, you’ll also encounter a few exercise blocks. You can click to expand them and challenge yourself by completing the tasks described there.<br /> Step 1: Inspect Your Data Source<br /> Before you write any Python code, you need to get to know the website that you want to scrape. That should be your first step for any web scraping project you want to tackle. You’ll need to understand the site structure to extract the information that’s relevant for you. Start by opening the site you want to scrape with your favorite browser.<br /> Explore the Website<br /> Click through the site and interact with it just like any typical job searcher would. For example, you can scroll through the main page of the website:<br /> You can see many job postings in a card format, and each of them has two buttons. If you click Apply, then you’ll see a new page that contains more detailed descriptions of the selected job. You might also notice that the URL in your browser’s address bar changes when you interact with the website.<br /> Decipher the Information in URLs<br /> A programmer can encode a lot of information in a URL. Your web scraping journey will be much easier if you first become familiar with how URLs work and what they’re made of. For example, you might find yourself on a details page that has the following URL:<br /> You can deconstruct the above URL into two main parts:<br /> The base URL represents the path to the search functionality of the website. In the example above, the base URL is The specific site location that ends with is the path to the job description’s unique resource.<br /> Any job posted on this website will use the same base URL. However, the unique resources’ location will be different depending on what specific job posting you’re viewing.<br /> URLs can hold more information than just the location of a file. Some websites use query parameters to encode values that you submit when performing a search. You can think of them as query strings that you send to the database to retrieve specific records.<br /> You’ll find query parameters at the end of a URL. For example, if you go to Indeed and search for “software developer” in “Australia” through their search bar, you’ll see that the URL changes to include these values as query parameters:<br /> The query parameters in this URL are? q=software+developer&l=Australia. Query parameters consist of three parts:<br /> Start: The beginning of the query parameters is denoted by a question mark (? ).<br /> Information: The pieces of information constituting one query parameter are encoded in key-value pairs, where related keys and values are joined together by an equals sign (key=value).<br /> Separator: Every URL can have multiple query parameters, separated by an ampersand symbol (&).<br /> Equipped with this information, you can pick apart the URL’s query parameters into two key-value pairs:<br /> q=software+developer selects the type of job.<br /> l=Australia selects the location of the job.<br /> Try to change the search parameters and observe how that affects your URL. Go ahead and enter new values in the search bar up top:<br /> Change these values to observe the changes in the URL.<br /> Next, try to change the values directly in your URL. See what happens when you paste the following URL into your browser’s address bar:<br /> If you change and submit the values in the website’s search box, then it’ll be directly reflected in the URL’s query parameters and vice versa. If you change either of them, then you’ll see different results on the website.<br /> As you can see, exploring the URLs of a site can give you insight into how to retrieve data from the website’s server.<br /> Head back to Fake Python Jobs and continue exploring it. This site is a purely static website that doesn’t operate on top of a database, which is why you won’t have to work with query parameters in this scraping tutorial.<br /> Inspect the Site Using Developer Tools<br /> Next, you’ll want to learn more about how the data is structured for display. You’ll need to understand the page structure to pick what you want from the HTML response that you’ll collect in one of the upcoming steps.<br /> Developer tools can help you understand the structure of a website. All modern browsers come with developer tools installed. In this section, you’ll see how to work with the developer tools in Chrome. The process will be very similar to other modern browsers.<br /> In Chrome on macOS, you can open up the developer tools through the menu by selecting View → Developer → Developer Tools. On Windows and Linux, you can access them by clicking the top-right menu button (⋮) and selecting More Tools → Developer Tools. You can also access your developer tools by right-clicking on the page and selecting the Inspect option or using a keyboard shortcut:<br /> Mac: Cmd+Alt+I<br /> Windows/Linux: Ctrl+Shift+I<br /> Developer tools allow you to interactively explore the site’s document object model (DOM) to better understand your source. To dig into your page’s DOM, select the Elements tab in developer tools. You’ll see a structure with clickable HTML elements. You can expand, collapse, and even edit elements right in your browser:<br /> The HTML on the right represents the structure of the page you can see on the left.<br /> You can think of the text displayed in your browser as the HTML structure of that page. If you’re interested, then you can read more about the difference between the DOM and HTML on CSS-TRICKS.<br /> When you right-click elements on the page, you can select Inspect to zoom to their location in the DOM. You can also hover over the HTML text on your right and see the corresponding elements light up on the page.<br /> Click to expand the exercise block for a specific task to practice using your developer tools:<br /> Find a single job posting. What HTML element is it wrapped in, and what other HTML elements does it contain?<br /> Play around and explore! The more you get to know the page you’re working with, the easier it will be to scrape it. However, don’t get too overwhelmed with all that HTML text. You’ll use the power of programming to step through this maze and cherry-pick the information that’s relevant to you.<br /> Step 2: Scrape HTML Content From a Page<br /> Now that you have an idea of what you’re working with, it’s time to start using Python. First, you’ll want to get the site’s HTML code into your Python script so that you can interact with it. For this task, you’ll use Python’s requests library.<br /> Create a virtual environment for your project before you install any external package. Activate your new virtual environment, then type the following command in your terminal to install the external requests library:<br /> $ python -m pip install requests<br /> Then open up a new file in your favorite text editor. All you need to retrieve the HTML are a few lines of code:<br /> import requests<br /> URL = ”<br /> page = (URL)<br /> print()<br /> This code issues an HTTP GET request to the given URL. It retrieves the HTML data that the server sends back and stores that data in a Python object.<br /> If you print the attribute of page, then you’ll notice that it looks just like the HTML that you inspected earlier with your browser’s developer tools. You successfully fetched the static site content from the Internet! You now have access to the site’s HTML from within your Python script.<br /> Static Websites<br /> The website that you’re scraping in this tutorial serves static HTML content. In this scenario, the server that hosts the site sends back HTML documents that already contain all the data that you’ll get to see as a user.<br /> When you inspected the page with developer tools earlier on, you discovered that a job posting consists of the following long and messy-looking HTML:</p> <div class="card"> <div class="card-content"> <div class="media"> <div class="media-left"> <figure class="image is-48x48"> <img src=" alt="Real Python Logo" /><br /> </figure> </div> <div class="media-content"> <h2 class="title is-5">Senior Python Developer</h2> <h3 class="subtitle is-6 company">Payne, Roberts and Davis</h3> <div class="content"> <p class="location">Stewartbury, AA</p> <p class="is-small has-text-grey"> <time datetime="2021-04-08">2021-04-08</time> </p> <footer class="card-footer"> <a href=" target="_blank" class="card-footer-item" >Learn</a ><br /> >Apply</a </footer> <p>It can be challenging to wrap your head around a long block of HTML code. To make it easier to read, you can use an HTML formatter to clean it up automatically. Good readability helps you better understand the structure of any code block. While it may or may not help improve the HTML formatting, it’s always worth a try.<br /> The HTML you’ll encounter will sometimes be confusing. Luckily, the HTML of this job board has descriptive class names on the elements that you’re interested in:<br /> class=”title is-5″ contains the title of the job posting.<br /> class=”subtitle is-6 company” contains the name of the company that offers the position.<br /> class=”location” contains the location where you’d be working.<br /> In case you ever get lost in a large pile of HTML, remember that you can always go back to your browser and use the developer tools to further explore the HTML structure interactively.<br /> By now, you’ve successfully harnessed the power and user-friendly design of Python’s requests library. With only a few lines of code, you managed to scrape static HTML content from the Web and make it available for further processing.<br /> However, there are more challenging situations that you might encounter when you’re scraping websites. Before you learn how to pick the relevant information from the HTML that you just scraped, you’ll take a quick look at two of these more challenging situations.<br /> Hidden Websites<br /> Some pages contain information that’s hidden behind a login. That means you’ll need an account to be able to scrape anything from the page. The process to make an HTTP request from your Python script is different from how you access a page from your browser. Just because you can log in to the page through your browser doesn’t mean you’ll be able to scrape it with your Python script.<br /> However, the requests library comes with the built-in capacity to handle authentication. With these techniques, you can log in to websites when making the HTTP request from your Python script and then scrape information that’s hidden behind a login. You won’t need to log in to access the job board information, which is why this tutorial won’t cover authentication.<br /> Dynamic Websites<br /> In this tutorial, you’ll learn how to scrape a static website. Static sites are straightforward to work with because the server sends you an HTML page that already contains all the page information in the response. You can parse that HTML response and immediately begin to pick out the relevant data.<br /> On the other hand, with a dynamic website, the server might not send back any HTML at all. Instead, you could receive JavaScript code as a response. This code will look completely different from what you saw when you inspected the page with your browser’s developer tools.<br /> What happens in the browser is not the same as what happens in your script. Your browser will diligently execute the JavaScript code it receives from a server and create the DOM and HTML for you locally. However, if you request a dynamic website in your Python script, then you won’t get the HTML page content.<br /> When you use requests, you only receive what the server sends back. In the case of a dynamic website, you’ll end up with some JavaScript code instead of HTML. The only way to go from the JavaScript code you received to the content that you’re interested in is to execute the code, just like your browser does. The requests library can’t do that for you, but there are other solutions that can.<br /> For example, requests-html is a project created by the author of the requests library that allows you to render JavaScript using syntax that’s similar to the syntax in requests. It also includes capabilities for parsing the data by using Beautiful Soup under the hood.<br /> You won’t go deeper into scraping dynamically-generated content in this tutorial. For now, it’s enough to remember to look into one of the options mentioned above if you need to scrape a dynamic website.<br /> Step 3: Parse HTML Code With Beautiful Soup<br /> You’ve successfully scraped some HTML from the Internet, but when you look at it, it just seems like a huge mess. There are tons of HTML elements here and there, thousands of attributes scattered around—and wasn’t there some JavaScript mixed in as well? It’s time to parse this lengthy code response with the help of Python to make it more accessible and pick out the data you want.<br /> Beautiful Soup is a Python library for parsing structured data. It allows you to interact with HTML in a similar way to how you interact with a web page using developer tools. The library exposes a couple of intuitive functions you can use to explore the HTML you received. To get started, use your terminal to install Beautiful Soup:<br /> $ python -m pip install beautifulsoup4<br /> Then, import the library in your Python script and create a Beautiful Soup object:<br /> from bs4 import BeautifulSoup<br /> soup = BeautifulSoup(ntent, “”)<br /> When you add the two highlighted lines of code, you create a Beautiful Soup object that takes ntent, which is the HTML content you scraped earlier, as its input.<br /> The second argument, “”, makes sure that you use the appropriate parser for HTML content.<br /> Find Elements by ID<br /> In an HTML web page, every element can have an id attribute assigned. As the name already suggests, that id attribute makes the element uniquely identifiable on the page. You can begin to parse your page by selecting a specific element by its ID.<br /> Switch back to developer tools and identify the HTML object that contains all the job postings. Explore by hovering over parts of the page and using right-click to Inspect.<br /> The element you’re looking for is a </p> <div> with an id attribute that has the value “ResultsContainer”. It has some other attributes as well, but below is the gist of what you’re looking for:</p> <div id="ResultsContainer"> <! -- all the job listings --><br /> Beautiful Soup allows you to find that specific HTML element by its ID:<br /> results = (id=”ResultsContainer”)<br /> For easier viewing, you can prettify any Beautiful Soup object when you print it out. If you call. prettify() on the results variable that you just assigned above, then you’ll see all the HTML contained within the </p> <div>:<br /> print(ettify())<br /> When you use the element’s ID, you can pick out one element from among the rest of the HTML. Now you can work with only this specific part of the page’s HTML. It looks like the soup just got a little thinner! However, it’s still quite dense.<br /> Find Elements by HTML Class Name<br /> You’ve seen that every job posting is wrapped in a </p> <div> element with the class card-content. Now you can work with your new object called results and select only the job postings in it. These are, after all, the parts of the HTML that you’re interested in! You can do this in one line of code:<br /> job_elements = nd_all(“div”, class_=”card-content”)<br /> Here, you call. find_all() on a Beautiful Soup object, which returns an iterable containing all the HTML for all the job listings displayed on that page.<br /> Take a look at all of them:<br /> for job_element in job_elements:<br /> print(job_element, end=”\n”*2)<br /> That’s already pretty neat, but there’s still a lot of HTML! You saw earlier that your page has descriptive class names on some elements. You can pick out those child elements from each job posting with ():<br /> title_element = (“h2″, class_=”title”)<br /> company_element = (“h3″, class_=”company”)<br /> location_element = (“p”, class_=”location”)<br /> print(title_element)<br /> print(company_element)<br /> print(location_element)<br /> Each job_element is another BeautifulSoup() object. Therefore, you can use the same methods on it as you did on its parent element, results.<br /> With this code snippet, you’re getting closer and closer to the data that you’re actually interested in. Still, there’s a lot going on with all those HTML tags and attributes floating around:<br /> Next, you’ll learn how to narrow down this output to access only the text content you’re interested in.<br /> Find Elements by Class Name and Text Content<br /> Not all of the job listings are developer jobs. Instead of printing out all the jobs listed on the website, you’ll first filter them using keywords.<br /> You know that job titles in the page are kept within </p> <h2> elements. To filter for only specific jobs, you can use the string argument:<br /> python_jobs = nd_all(“h2″, string=”Python”)<br /> This code finds all </p> <h2> elements where the contained string matches “Python” exactly. Note that you’re directly calling the method on your first results variable. If you go ahead and print() the output of the above code snippet to your console, then you might be disappointed because it’ll be empty:<br /> >>>>>> print(python_jobs)<br /> []<br /> There was a Python job in the search results, so why is it not showing up?<br /> When you use string= as you did above, your program looks for that string exactly. Any differences in the spelling, capitalization, or whitespace will prevent the element from matching. In the next section, you’ll find a way to make your search string more general.<br /> Pass a Function to a Beautiful Soup Method<br /> In addition to strings, you can sometimes pass functions as arguments to Beautiful Soup methods. You can change the previous line of code to use a function instead:<br /> python_jobs = nd_all(<br /> “h2”, string=lambda text: “python” in ())<br /> Now you’re passing an anonymous function to the string= argument. The lambda function looks at the text of each </p> <h2> element, converts it to lowercase, and checks whether the substring “python” is found anywhere. You can check whether you managed to identify all the Python jobs with this approach:<br /> >>>>>> print(len(python_jobs))<br /> 10<br /> Your program has found 10 matching job posts that include the word “python” in their job title!<br /> Finding elements depending on their text content is a powerful way to filter your HTML response for specific information. Beautiful Soup allows you to use either exact strings or functions as arguments for filtering text in Beautiful Soup objects.<br /> However, when you try to run your scraper to print out the information of the filtered Python jobs, you’ll run into an error:<br /> AttributeError: ‘NoneType’ object has no attribute ‘text’<br /> This message is a common error that you’ll run into a lot when you’re scraping information from the Internet. Inspect the HTML of an element in your python_jobs list. What does it look like? Where do you think the error is coming from?<br /> Identify Error Conditions<br /> When you look at a single element in python_jobs, you’ll see that it consists of only the </p> <h2> element that contains the job title:<br /> When you revisit the code you used to select the items, you’ll see that that’s what you targeted. You filtered for only the </p> <h2> title elements of the job postings that contain the word “python”. As you can see, these elements don’t include the rest of the information about the job.<br /> The error message you received earlier was related to this:<br /> You tried to find the job title, the company name, and the job’s location in each element in python_jobs, but each element contains only the job title text.<br /> Your diligent parsing library still looks for the other ones, too, and returns None because it can’t find them. Then, print() fails with the shown error message when you try to extract the attribute from one of these None objects.<br /> The text you’re looking for is nested in sibling elements of the </p> <h2> elements your filter returned. Beautiful Soup can help you to select sibling, child, and parent elements of each Beautiful Soup object.<br /> Access Parent Elements<br /> One way to get access to all the information you need is to step up in the hierarchy of the DOM starting from the </p> <h2> elements that you identified. Take another look at the HTML of a single job posting. Find the </p> <h2> element that contains the job title as well as its closest parent element that contains all the information that you’re interested in:<br /> The </p> <div> element with the card-content class contains all the information you want. It’s a third-level parent of the </p> <h2> title element that you found using your filter.<br /> With this information in mind, you can now use the elements in python_jobs and fetch their great-grandparent elements instead to get access to all the information you want:<br /> python_job_elements = [<br /> for h2_element in python_jobs]<br /> You added a list comprehension that operates on each of the </p> <h2> title elements in python_jobs that you got by filtering with the lambda expression. You’re selecting the parent element of the parent element of the parent element of each </p> <h2> title element. That’s three generations up!<br /> When you were looking at the HTML of a single job posting, you identified that this specific parent element with the class name card-content contains all the information you need.<br /> Now you can adapt the code in your for loop to iterate over the parent elements instead:<br /> for job_element in python_job_elements:<br /> # — snip —<br /> When you run your script another time, you’ll see that your code once again has access to all the relevant information. That’s because you’re now looping over the </p> <div class="card-content"> elements instead of just the </p> <h2> title elements.<br /> Using the attribute that each Beautiful Soup object comes with gives you an intuitive way of stepping through your DOM structure and addressing the elements you need. You can also access child elements and sibling elements in a similar manner. Read up on navigating the tree for more information.<br /> Keep Practicing<br /> If you’ve written the code alongside this tutorial, then you can run your script as is, and you’ll see the fake job information pop up in your terminal. Your next step is to tackle a real-life job board! To keep practicing your new skills, revisit the web scraping process using any or all of the following sites:<br /> PythonJobs<br /> Remote(dot)co<br /> Indeed<br /> The linked websites return their search results as static HTML responses, similar to the Fake Python job board. Therefore, you can scrape them using only requests and Beautiful Soup.<br /> Start going through this tutorial again from the top using one of these other sites. You’ll see that each website’s structure is different and that you’ll need to rebuild the code in a slightly different way to fetch the data you want. Tackling this challenge is a great way to practice the concepts that you just learned. While it might make you sweat every so often, your coding skills will be stronger for it!<br /> During your second attempt, you can also explore additional features of Beautiful Soup. Use the documentation as your guidebook and inspiration. Extra practice will help you become more proficient at web scraping using Python, requests, and Beautiful Soup.<br /> To wrap up your journey into web scraping, you could then give your code a final makeover and create a command-line interface (CLI) app that scrapes one of the job boards and filters the results by a keyword that you can input on each execution. Your CLI tool could allow you to search for specific types of jobs or jobs in particular locations.<br /> If you’re interested in learning how to adapt your script as a command-line interface, then check out How to Build Command-Line Interfaces in Python With argparse.<br /> Conclusion<br /> The requests library gives you a user-friendly way to fetch static HTML from the Internet using Python. You can then parse the HTML with another package called Beautiful Soup. Both packages are trusted and helpful companions for your web scraping adventures. You’ll find that Beautiful Soup will cater to most of your parsing needs, including navigation and advanced searching.<br /> In this tutorial, you learned how to scrape data from the Web using Python, requests, and Beautiful Soup. You built a script that fetches job postings from the Internet and went through the complete web scraping process from start to finish.<br /> You learned how to:<br /> Decipher the data encoded in URLs<br /> Download the page’s HTML content using Python’s requests library<br /> Parse the downloaded HTML with Beautiful Soup to extract relevant information<br /> With this broad pipeline in mind and two powerful libraries in your tool kit, you can go out and see what other websites you can scrape. Have fun, and always remember to be respectful and use your programming skills responsibly.<br /> You can download the source code for the sample script that you built in this tutorial by clicking the link below:<br /> Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: Web Scraping With Beautiful Soup and Python</p> <h2>Frequently Asked Questions about bs4 tutorial</h2> <h3>What is bs4 used for?</h3> <p>Beautiful Soup is a Python library for pulling data out of HTML and XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work.Feb 24, 2020</p> <h3>How do I use BeautifulSoup to scrape?</h3> <p>Implementing Web Scraping in Python with BeautifulSoupSteps involved in web scraping:Step 1: Installing the required third-party libraries.Step 2: Accessing the HTML content from webpage.Step 3: Parsing the HTML content.Step 4: Searching and navigating through the parse tree.May 15, 2021</p> <h3>How do I run BeautifulSoup?</h3> <p>using a Python script like this:from bs4 import BeautifulSoup import csv soup = BeautifulSoup (open(“43rd-congress.html”), features=”lxml”) final_link = soup. p. … from bs4 import BeautifulSoup soup = BeautifulSoup(open(“43rd-congress.html”), features=”lxml”) print(soup. prettify()) … python soupexample.py.Dec 30, 2012</p> </div> <div class="meks_ess layout-1-1 rectangle no-labels solid"><a href="#" class="meks_ess-item socicon-facebook" data-url="http://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fproxywatcher.com%2Fbs4-tutorial%2F&t=Bs4%20Tutorial"><span>Facebook</span></a><a href="#" class="meks_ess-item socicon-twitter" data-url="http://twitter.com/intent/tweet?url=https%3A%2F%2Fproxywatcher.com%2Fbs4-tutorial%2F&text=Bs4%20Tutorial"><span>Twitter</span></a></div> </article> </div> <div class="section-head"><h3 class="section-title h6">About the author</h3></div> <div class="section-content typology-author"> <div class="container"> <div class="col-lg-2"> <img alt='' src='https://secure.gravatar.com/avatar/2ac98697d1de45adf19ef225667a6136?s=100&d=mm&r=g' srcset='https://secure.gravatar.com/avatar/2ac98697d1de45adf19ef225667a6136?s=200&d=mm&r=g 2x' class='avatar avatar-100 photo' height='100' width='100' loading='lazy' decoding='async'/> </div> <div class="col-lg-10"> <h5 class="typology-author-box-title">proxyreview</h5> <div class="typology-author-desc"> <p>If you 're a SEO / IM geek like us then you'll love our updates and our website. Follow us for the latest news in the world of web automation tools & proxy servers!</p> </div> <div class="typology-author-links"> <a class="typology-button-social hover-on" href="https://proxywatcher.com/author/proxyreview/">View all posts</a><a href="http://proxywatcher.com" target="_blank" class="typology-icon-social hover-on fa fa-link"></a><a href="https://www.facebook.com/proxywatchercom-1689228624692822" target="_blank" class="typology-icon-social hover-on fa fa-facebook"></a><a href="https://twitter.com/TheProxywatcher" target="_blank" class="typology-icon-social hover-on fa fa-twitter"></a> </div> </div> </div> </div> <div class="typology-ad typology-ad-bottom"><!-- Google tag (gtag.js) --> <script async src="https://www.googletagmanager.com/gtag/js?id=G-6QFKZ2KN67"></script> <script> window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'G-6QFKZ2KN67'); </script></div> </div> <div class="typology-section typology-section-related"> <div class="section-head"><h3 class="section-title h6">Read more</h3></div> <div class="section-content section-content-c"> <div class="typology-posts"> <article class="typology-post typology-layout-c col-lg-6 text-center post-image-off post-38368 post type-post status-publish format-standard has-post-thumbnail hentry category-proxy tag-craigslist-account-for-sale tag-craigslist-chicago-account tag-craigslist-homepage tag-craigslist-login tag-craigslist-my-account-not-working tag-how-to-place-an-ad-on-craigslist-with-pictures tag-post-ad-on-craigslist-for-free tag-sell-on-craigslist"> <header class="entry-header"> <h2 class="entry-title h4"><a href="https://proxywatcher.com/create-new-craigslist-account/">Create New Craigslist Account</a></h2> <div class="entry-meta"><div class="meta-item meta-date"><span class="updated">11 months ago</span></div></div> <div class="post-letter">C</div> </header> </article> <article class="typology-post typology-layout-c col-lg-6 text-center post-image-off post-41004 post type-post status-publish format-standard has-post-thumbnail hentry category-proxy tag-bittorrent-proxy tag-qbittorrent-proxy tag-torguard-proxy-list tag-torguard-proxy-qbittorrent tag-utorrent-proxy-connect-error tag-utorrent-proxy-free-download tag-utorrent-proxy-list tag-utorrent-proxy-site-kickass"> <header class="entry-header"> <h2 class="entry-title h4"><a href="https://proxywatcher.com/utorrent-proxy/">Utorrent Proxy</a></h2> <div class="entry-meta"><div class="meta-item meta-date"><span class="updated">11 months ago</span></div></div> <div class="post-letter">U</div> </header> </article> <article class="typology-post typology-layout-c col-lg-6 text-center post-image-off post-36208 post type-post status-publish format-standard has-post-thumbnail hentry category-proxy tag-crawl-website-for-all-urls tag-free-web-crawler tag-google-crawler tag-how-to-make-a-web-crawler-in-python tag-types-of-web-crawlers tag-web-crawler-is-an-example-of tag-web-crawler-tool tag-web-crawling-vs-web-scraping"> <header class="entry-header"> <h2 class="entry-title h4"><a href="https://proxywatcher.com/how-to-crawl-the-web/">How To Crawl The Web</a></h2> <div class="entry-meta"><div class="meta-item meta-date"><span class="updated">11 months ago</span></div></div> <div class="post-letter">H</div> </header> </article> <article class="typology-post typology-layout-c col-lg-6 text-center post-image-off post-43551 post type-post status-publish format-standard has-post-thumbnail hentry category-proxy tag-best-vpn-to-hide-ip-address tag-free-vpn-to-hide-ip-address tag-hide-my-ip-address-free-online tag-how-to-hide-ip-address-free tag-how-to-hide-ip-address-on-android tag-how-to-hide-my-ip-address-without-vpn tag-how-to-hide-your-ip-address-on-iphone tag-how-to-protect-your-ip-address-from-hackers"> <header class="entry-header"> <h2 class="entry-title h4"><a href="https://proxywatcher.com/how-to-protect-my-ip-address-from-tracking/">How To Protect My Ip Address From Tracking</a></h2> <div class="entry-meta"><div class="meta-item meta-date"><span class="updated">11 months ago</span></div></div> <div class="post-letter">H</div> </header> </article> </div> </div> </div> <div id="typology-single-sticky" class="typology-single-sticky"> <div class="typology-sticky-content meta"> <div class="typology-flex-center"> <div class="typology-sticky-author typology-sticky-l"> <img alt='' src='https://secure.gravatar.com/avatar/2ac98697d1de45adf19ef225667a6136?s=50&d=mm&r=g' srcset='https://secure.gravatar.com/avatar/2ac98697d1de45adf19ef225667a6136?s=100&d=mm&r=g 2x' class='avatar avatar-50 photo' height='50' width='50' loading='lazy' decoding='async'/> <span class="sticky-author-title"> <a href="https://proxywatcher.com/author/proxyreview/">By proxyreview</a> <span class="sticky-author-date">November 16, 2021</span> </span> </div> <div class="typology-sticky-c"> </div> <div class="typology-sticky-comments typology-sticky-r"> </div> </div> </div> <div class="typology-sticky-content prev-next"> <nav class="typology-prev-next-nav typology-flex-center"> <div class="typology-prev-link typology-sticky-l"> <a href="https://proxywatcher.com/craigslist-mobile-view/"> <span class="typology-pn-ico"><i class="fa fa-chevron-left"></i></span> <span class="typology-pn-link">Craigslist Mobile View</span> </a> </div> <a href="javascript: void(0);" class="typology-sticky-to-top typology-sticky-c"> <span class="typology-top-ico"><i class="fa fa-chevron-up"></i></span> <span class="typology-top-link">To Top</span> </a> <div class="typology-next-link typology-sticky-r"> <a href="https://proxywatcher.com/time-parser/"> <span class="typology-pn-ico"><i class="fa fa-chevron-right"></i></span> <span class="typology-pn-link">Time Parser</span> </a> </div> </nav> </div> </div> <footer id="typology-footer" class="typology-footer"> <div class="container"> </div> </footer> </div> <div class="typology-sidebar"> <div class="typology-sidebar-header"> <div class="typology-sidebar-header-wrapper"> <div class="typology-site-branding"> <span class="site-title h4"><a href="https://proxywatcher.com/" rel="home"><img class="typology-logo" src="https://proxywatcher.com/wp-content/uploads/2021/09/logo_white.png" alt="Proxywatcher.com"></a></span> </div> <span class="typology-sidebar-close"><i class="fa fa-times" aria-hidden="true"></i></span> </div> </div> <div class="widget typology-responsive-menu"> </div> <div id="recent-posts-4" class="widget clearfix widget_recent_entries"> <h4 class="widget-title h5">Recent Posts</h4> <ul> <li> <a href="https://proxywatcher.com/create-new-craigslist-account/">Create New Craigslist Account</a> </li> <li> <a href="https://proxywatcher.com/utorrent-proxy/">Utorrent Proxy</a> </li> <li> <a href="https://proxywatcher.com/how-to-crawl-the-web/">How To Crawl The Web</a> </li> <li> <a href="https://proxywatcher.com/how-to-protect-my-ip-address-from-tracking/">How To Protect My Ip Address From Tracking</a> </li> <li> <a href="https://proxywatcher.com/android-9-proxy-settings/">Android 9 Proxy Settings</a> </li> </ul> </div><div id="text-4" class="widget clearfix widget_text"><h4 class="widget-title h5">Useful Tools</h4> <div class="textwidget"><a href="http://proxycompass.com">Proxy Package Finder</a></div> </div> </div> <div class="typology-sidebar-overlay"></div> <link rel='stylesheet' id='aal_style-css' href='https://proxywatcher.com/wp-content/plugins/wp-auto-affiliate-links/css/style.css?ver=6.2' type='text/css' media='all' /> <script type='text/javascript' id='wp-postviews-cache-js-extra'> /* <![CDATA[ */ var viewsCacheL10n = {"admin_ajax_url":"https:\/\/proxywatcher.com\/wp-admin\/admin-ajax.php","post_id":"31818"}; /* ]]> */ </script> <script type='text/javascript' src='https://proxywatcher.com/wp-content/plugins/wp-postviews/postviews-cache.js?ver=1.68' id='wp-postviews-cache-js'></script> <script type='text/javascript' src='https://proxywatcher.com/wp-includes/js/imagesloaded.min.js?ver=4.1.4' id='imagesloaded-js'></script> <script type='text/javascript' id='typology-main-js-extra'> /* <![CDATA[ */ var typology_js_settings = {"rtl_mode":"","header_sticky":"1","logo":"https:\/\/proxywatcher.com\/wp-content\/uploads\/2021\/09\/logo_white.png","logo_retina":"https:\/\/proxywatcher.com\/wp-content\/uploads\/2021\/09\/logo_white_retina.png","use_gallery":"1","slider_autoplay":"0","cover_video_image_fallback":""}; /* ]]> */ </script> <script type='text/javascript' src='https://proxywatcher.com/wp-content/themes/typology/assets/js/min.js?ver=1.7.2' id='typology-main-js'></script> <script type='text/javascript' src='https://proxywatcher.com/wp-content/plugins/meks-easy-social-share/assets/js/main.js?ver=1.2.9' id='meks_ess-main-js'></script> <script>!function(){window.advanced_ads_ready_queue=window.advanced_ads_ready_queue||[],advanced_ads_ready_queue.push=window.advanced_ads_ready;for(var d=0,a=advanced_ads_ready_queue.length;d<a;d++)advanced_ads_ready(advanced_ads_ready_queue[d])}();</script> </body> </html>