Selenium Parser


How can I parse a website using Selenium ... - Stack Overflow

How can I parse a website using Selenium … – Stack Overflow

New to programming and figured out how to navigate to where I need to go using Selenium. I’d like to parse the data now but not sure where to start. Can someone hold my hand a sec and point me in the right direction?
Any help appreciated –
asked Dec 19 ’12 at 20:06
Assuming you are on the page you want to parse, Selenium stores the source HTML in the driver’s page_source attribute. You would then load the page_source into BeautifulSoup as follows:
In [8]: from bs4 import BeautifulSoup
In [9]: from selenium import webdriver
In [10]: driver = refox()
In [11]: (”)
In [12]: html = ge_source
In [13]: soup = BeautifulSoup(html)
In [14]: for tag in nd_all(‘title’):…. : print…. :…. :
Hacker News
answered Dec 19 ’12 at 20:19
RocketDonkeyRocketDonkey33. 7k7 gold badges75 silver badges83 bronze badges
As your question isn’t particularly concrete, here’s a simple example. To do something more useful read the BS docs. You will also find plenty of examples of selenium (and BS)usage here in SO.
from selenium import webdriver
from bs4 import BeautifulSoup
#do something useful
#prints all the links with corresponding text
for link in nd_all(‘a’):
print (‘href’, None), t_text()
answered Dec 19 ’12 at 20:18
Are you sure you want to use Selenium? For this reasons I used PyQt4, it’s very powerful, and you can do what ever you want.
I can give you a sample code, that I just wrote, just change url and you good to go:
#! /usr/bin/env python2. 7
from import *
from PyQt4. QtWebKit import *
import sys, signal
class Browser(QWebView):
def __init__(self):
QWebView. __init__(self)
nnect(self. _progress)
nnect(self. _loadFinished)
= (). currentFrame()
def _progress(self, progress):
print str(progress) + “%”
def _loadFinished(self):
print “Load Finished”
html = unicode(())(‘utf-8’)
soup = BeautifulSoup(html)
print ettify()
if __name__ == “__main__”:
app = QApplication()
br = Browser()
url = QUrl(‘web site that can contain ‘)
if (, G_DFL):
(app. exec_())
app. exec_()
answered Dec 19 ’12 at 20:14
VorVor28. 8k39 gold badges123 silver badges186 bronze badges
Not the answer you’re looking for? Browse other questions tagged python selenium beautifulsoup or ask your own question.
Web Scraping with Python using Selenium and Beautiful Soup

Web Scraping with Python using Selenium and Beautiful Soup

Beautiful Soup
Web Scraping
Web scraping is a powerful tool which can be used to retrieve structured data from websites if an API is not available. This article shows you how to get started.
In this article, I want to show you how to download and parse structured data from the text of a website. It is a gentle introduction those who have wanted to deal to get started with Web Scrapping and building their own datasets, but did not know where to dig in.
What Is Web Scraping? Web scraping is the practice of automatically retrieving the content of user-facing web pages, analyzing them, and extracting/structuring useful information. Sometimes this is necessary if you need to retrieve data from a resource (perhaps a government website), but a URL for fetching a machine readable format (such as JSON or XML) is not available. Using a scraper, it is possible to “crawl” the pages of the site, read the HTML source, and extract the pieces of information in a more machine friendly format. Everything that I’ve described can certainly be done manually, but the power of scrapers is that they allow you to automate the process. Mature scraping programs, such as Selenium, allow for you to automate all aspects of data collection. This might include specifying a URL to visit, elements in the page to interact with, and even ways to simulate user behavior. Once the specifics of the “session” are defined, you are then free to run the script hundreds, thousands, or millions of though scraping is extremely powerful, this is not to say it is an easy task. Websites come in many different forms, and many websites need love and care in order to find and retrieve information. Further, there are many parsing libraries available, with large variability in terms of functionality and capabilities. In this article, we will look at two of the most popular parsing tools and talk about how they are used.
Static and Dynamic Scraping, What’s the Difference? First, though, let’s talk about the two major types of scraping: static and scraping ignores JavaScript. It pulls web pages from the server without using a browser. You get exactly what you see in the “page source” and then you cut and parse it. If the content you’re looking for is available without JavaScript, then using the popular Beautiful Soup library is an ideal way to scrape static sites. However, if the content is something like iframe comments, you need dynamic scraping. Dynamic scraping uses an actual browser (or headless browser) and allows you to read content that has been generated or modified via JavaScript. Then it asks the DOM to fetch the content you are looking for. Also sometimes you need to automate the browser by simulating the user to get the content you want. For such a task, you need to use Selenium WebDriver.
Static Scraping With Beautiful SoupIn this section, we will look at how you can scrape static content. We’ll introduce the Python library “Beatiful Soup, ” discuss what it is used for, and a brief description of how to use autiful Soup is a Python library that uses an HTML/XML parser and turns the web page/html/xml into a tree of tags, elements, attributes, and values. Once parsed, a document consists of four types of objects: Tag, NavigableString, BeautifulSoup, and Comment. The tree provides methods / properties of the BeautifulSoup object which facilitates iterating through the content to retrieve information of 0: Install the LibraryTo get started with Beautiful Soup, run the following commands in terminal. It is also recommended to use a virtual environment to keep your system “clean”.
pip install lxml
pip install requests
pip install beautifulsoup4
Step 1: Retrieve Data From a Target WebsiteGo to the code editor and import the libraries:
from bs4 import BeautifulSoup
import requests
To get acquainted with the scraping process, we will use and try to parse the prices of, let’s create a variable for our URL:
url = ”
Now let’s send a GET request to the site and save the received data in the page variable:
page = (url)
Before parsing, let’s check the connection to ensure that the website returned content rather than an error:
The code returned us a status of 200, which means that we are successfully connected and everything is in can now use BeautifulSoup4 and pass our page to it.
soup = BeautifulSoup (, “”)
Step 2: Inspect the Contents of the Page and Extract DataWe can look at the html-code of our page:
Let’s get back to our ‘s start by writing a function that will make a request for the specified link and return a response:
def get_html(url):
response = (url)
The page displays 48 products, all of them are located in the

  • tag with the s-item class. Let’s write a function that will find them.
    def get_all_items(html):
    soup = BeautifulSoup(html, ‘lxml’)
    items = (“ul”, {“class”: “b-list__items_nofooter”}). findAll(“li”, {“class”: “s-item”})
    return items
    While the page has a lot of different types of data, we will only parse the name of the product and its price. To do this, we will write a function that will accept one product as input. Using the h3 tag with the b-s-item__title class, get the name of the product and find the price in the span with the s-item__price class
    def get_item_data(item):
    title = ({“h3”: “b-s-item__title”})
    title = ”
    price = (“span”, {“class”: “s-item__price”})
    price = ”
    data = {‘title’: title,
    ‘price’: price}
    return data
    Next, we will write another function with which we will write the data that we received to a file with the csv extension:
    def write_csv(i, data):
    with open(”, ‘a’) as f:
    writer = (f)
    writer. writerow((data[‘title’],
    print(i, data[‘title’], ‘parsed’)
    Putting all the pieces together in a main function, we can create a simple program which will connect to Ebay and retrieve the first five pages of laptop data:
    def main():
    for page in range(1, 5): # count of pages to parse
    all_items = get_all_items(get_html(url + ‘? _pgn={}'(page)))
    for i, item in enumerate(all_items):
    data = get_item_data(item)
    write_csv(i, data)
    if __name__ == ‘__main__’:
    Once the program finishes running, we will receive a file with the names of goods and their prices:
    An example of the result obtained
    The example above is the most basic information needed to familiarize yourself with the Beautiful Soup library. Most real-world scraping problems are more complex, and require that you retrieve a large amount of information within a fairly short time frame. If that is the case, Beautiful Soup can be used alongside the multiprocessing library, which allows you to run multiple threads (each of which can retrieve and parse data in parallel), which will significantly increase the parsing speed.
    Dynamic Scraping With Selenium WebDriverIn this section, let’s do some dynamic nsider a website that sells game cards. Unlike Ebay, which sends fully formatted HTML, the card game website uses a single page architecture which dynamically renders the data via JavaScript templates. In this case, if you attempt to parse the data using Beautiful Soup, your parser won’t find any data. The information first must be rendered by this type of application, you can use Selenium to get prices for cards. Selenium is a library which will interface with the browser, allow for the site to render, and then allow you to retrieve the data from the browser’s DOM. If you need to, you can script the browser to click on various links to load HTML partials that can also be parsed to get additional 0: Set Up Your ProgramHere are the required imports.
    import csv
    from import By
    from import WebDriverWait
    from pport import expected_conditions as EC
    from selenium import webdriver
    from import ChromeDriverManager
    get_page_data() function accepts driver and url. It uses the get() method of the driver to fetch the url. This is similar to (), but the difference is that the driver object manages the live view of the 1: Determine How to Parse the Dynamic Content of the PageFor the website in our example, the next step is to select the country and the number of prices on the that we expect the page elements to be present as the content is loaded dynamically and will not necessarily be immediately available. To prevent the program from moving on without finding and retrieving the data, we can use we look for the block where the prices are located using the product-listing class:
    def get_page_data(driver, url):
    btn_country = WebDriverWait(driver, 20)(
    esence_of_element_located((ASS_NAME, “btn-link”)))
    select_country = WebDriverWait(driver, 20)(
    (, “/html/body/div[5]/div[4]/div/div/div[2]/form/select/option[208]”)))
    save_country = WebDriverWait(driver, 20)(
    (, “/html/body/div[5]/div[4]/div/div/div[3]/button[1]”)))
    line_count = WebDriverWait(driver, 20)(
    (, “/html/body/div[5]/section[3]/div[2]/section/div[1]/div[3]/select”)))
    select_count = WebDriverWait(driver, 20)(
    (, “/html/body/div[5]/section[3]/div[2]/section/div[1]/div[3]/select/option[3]”)))
    lines = nd_elements_by_class_name(‘product-listing’)
    return lines
    Then we will find the seller, card information and price. To do this, we will again use the class of elements by which they are located (in this way, Beautiful Soup and Selenium are very similar):
    def get_prices(line):
    seller = nd_element_by_class_name(‘seller__name’)()
    seller = ”
    listing_condition = nd_element_by_class_name(
    listing_condition = ”
    price = nd_element_by_class_name(‘product-listing__price’)
    price =
    price = float(place(‘$’, ”). replace(‘, ‘, ”))
    data = {‘seller’: seller,
    ‘listing_condition’: listing_condition,
    Once we’ve found the data we are interested in, we can create a csv file from the data we received:
    def write_csv(data):
    writer. writerow((data[‘seller’],
    driver = (ChromeDriverManager(). install())
    lines = get_page_data(driver, url)
    for line in lines:
    Here is the result we got:
    ConclusionWeb scraping is a useful tool for retrieving information from web applications in the absence of an API. Using tools such as requests, BeautifulSoup, and Selenium it is possible to build tools for fetch significant amounts of data and convert it to a more convenient format for scraping!
    Unable to find related content
    RISE TO THE NEXT LEVEL | Keep up to date by subscribing to Oak-Tree.
    Python + Selenium parser - gists · GitHub

    Python + Selenium parser – gists · GitHub

    from selenium import webdriver
    from import Keys
    from import By
    from import WebDriverWait
    from pport import expected_conditions as EC
    driver = refox()
    #Ждем 10 сек. пока загрузится элемент на странице (в данном случае

    ), можно добавлять несколько для уверенности
    WebDriverWait(driver, 10)(esence_of_element_located((By. CSS_SELECTOR, “”)))
    #Записываем body страницы в переменную
    pageBody = nd_element_by_css_selector(‘body’). get_attribute(“outerHTML”)
    text_file = open(“”, “w”)

    Frequently Asked Questions about selenium parser

    About the author


    If you 're a SEO / IM geek like us then you'll love our updates and our website. Follow us for the latest news in the world of web automation tools & proxy servers!