Python Parse Html Table

Parse HTML table to Python list? – Stack Overflow

I’d like to take an HTML table and parse through it to get a list of dictionaries. Each list element would be a dictionary corresponding to a row in the table.
If, for example, I had an HTML table with three columns (marked by header tags), “Event”, “Start Date”, and “End Date” and that table had 5 entries, I would like to parse through that table to get back a list of length 5 where each element is a dictionary with keys “Event”, “Start Date”, and “End Date”.
Thanks for the help!
asked Jun 12 ’11 at 22:46
You should use some HTML parsing library like lxml:
from lxml import etree
s = “””

Event	Start Date	End Date
a	b	c
d	e	f
g	h	i

“””
table = (s)(“body/table”)
rows = iter(table)
headers = [ for col in next(rows)]
for row in rows:
values = [ for col in row]
print dict(zip(headers, values))
prints
{‘End Date’: ‘c’, ‘Start Date’: ‘b’, ‘Event’: ‘a’}
{‘End Date’: ‘f’, ‘Start Date’: ‘e’, ‘Event’: ‘d’}
{‘End Date’: ‘i’, ‘Start Date’: ‘h’, ‘Event’: ‘g’}
answered Jun 12 ’11 at 22:59
Sven MarnachSven Marnach501k111 gold badges880 silver badges790 bronze badges
8
Hands down the easiest way to parse a HTML table is to use ad_html() – it accepts both URLs and HTML.
import pandas as pd
url = r”
tables = ad_html(url) # Returns list of all tables on page
sp500_table = tables[0] # Select table of interest
Only downside is that read_html() doesn’t preserve hyperlinks.
answered Jul 14 ’17 at 23:48
zeluspzelusp2, 8582 gold badges26 silver badges54 bronze badges
7
Sven Marnach excellent solution is directly translatable into ElementTree which is part of recent Python distributions:
from import ElementTree as ET
table = (s)
print(dict(zip(headers, values)))
same output as Sven Marnach’s answer…
Hugo23. 9k6 gold badges70 silver badges88 bronze badges
answered Sep 6 ’11 at 6:46
3
If the HTML is not XML you can’t do it with etree. But even then, you don’t have to use an external library for parsing a HTML table. In python 3 you can reach your goal with HTMLParser from I’ve the code of the simple derived HTMLParser class here in a github repo.
You can use that class (here named HTMLTableParser) the following way:
import quest
from html_table_parser import HTMLTableParser
target = ”
# get website content
req = quest(url=target)
f = quest. urlopen(req)
xhtml = ()(‘utf-8’)
# instantiate the parser and feed it
p = HTMLTableParser()
(xhtml)
print()
The output of this is a list of 2D-lists representing tables. It looks maybe like this:
[[[‘ ‘, ‘ Anmelden ‘]],
[[‘Land’, ‘Code’, ‘Für Kunden von’],
[‘Vereinigte Staaten’, ‘40404’, ‘(beliebig)’],
[‘Kanada’, ‘21212’, ‘(beliebig)’],…
[‘3424486444’, ‘Vodafone’],
[‘ Zeige SMS-Kurzwahlen für andere Länder ‘]]]
answered Mar 11 ’14 at 8:31
schmijosschmijos6, 9603 gold badges45 silver badges52 bronze badges
2
Not the answer you’re looking for? Browse other questions tagged python html or ask your own question.

Parsing HTML Tables in Python with BeautifulSoup and pandas

Something that seems daunting at first when switching from R to Python is replacing all the ready-made functions R has. For example, R has a nice CSV reader out of the box. Python users will eventually find pandas, but what about other R libraries like their HTML Table Reader from the xml package? That’s very helpful for scraping web pages, but in Python it might take a little more work. So in this post, we’re going to write a brief but robust HTML table parser.
Our parser is going to be built on top of the Python package BeautifulSoup. It’s a convenient package and easy to use. Our use will focus on the “find_all” function, but before we start parsing, you need to understand the basics of HTML terminology.
An HTML object consists of a few fundamental pieces: a tag. The format that defines a tag is

and it could have attributes which consistes of a property and a value. A tag we are interested in is the table tag, which defined a table in a website. This table tag has many elements. An element is a component of the page which typically contains content. For a table in HTML, they consist of rows designated by elements within the tr tags, and then column content inside the td tags. A typical example is

Hello!

Table

It turns out that most sites keep data you’d like to scrape in tables, and so we’re going to learn to parse them.
Parsing a Table in BeautifulSoup
To parse the table, we are going to use the Python library BeautifulSoup. It constructs a tree from the HTML and gives you an API to access different elements of the webpage.
Let’s say we already have our table object returned from BeautifulSoup. To parse the table, we’d like to grab a row, take the data from its columns, and then move on to the next row ad nauseam. In the next bit of code, we define a website that is simply the HTML for a table. We load it into BeautifulSoup and parse it, returning a pandas data frame of the contents.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import pandas as pd
from bs4 import BeautifulSoup
html_string = ”’
”’
soup = BeautifulSoup(html_string, ‘lxml’) # Parse the HTML as a string
table = nd_all(‘table’)[0] # Grab the first table
new_table = Frame(columns=range(0, 2), index = [0]) # I know the size
row_marker = 0
for row in nd_all(‘tr’):
column_marker = 0
columns = nd_all(‘td’)
for column in columns:
[row_marker, column_marker] = t_text()
column_marker += 1
new_table
As you can see, we grab all the tr elements from the table, followed by grabbing the td elements one at a time. We use the “get_text()” method from the td element (called a column in each iteration) and put it into our python object representing a table (it will eventually be a pandas dataframe).
Now, that we have our plan to parse a table, we probably need to figure out how to get to that point. That’s actually easier! We’re going to use the requests package in Python.
import requests
url = ”
response = (url)
[:100] # Access the HTML with the text property
‘\r\n\n\n\n\n