Web Scraping with Selenium in Python (Step by Step)

Web scraping with Selenium in Python means driving a real browser from code, so you can read pages that build their content with JavaScript. Install it with pip install selenium, start a browser, load the page, and find elements:

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Edge()          # or webdriver.Chrome() / webdriver.Firefox()
driver.get("https://quotes.toscrape.com/js/")
quotes = driver.find_elements(By.CSS_SELECTOR, "div.quote")

Every example ran headless Microsoft Edge from Python 3.12.5 with Selenium 4.50.0 against quotes.toscrape.com, a practice site built for scraping.

We’ll see why Requests fails on JavaScript pages, wait for content properly, click through pages, fill in a form, and take a browser screenshot, plus a few rules for scraping responsibly.

Why use Selenium for web scraping?

The /js/ version of the practice site sends an almost empty page and adds the quotes with JavaScript. Requests only sees that empty page; Selenium runs the JavaScript like a real browser:

import requests
from selenium import webdriver
from selenium.webdriver.common.by import By

URL = "https://quotes.toscrape.com/js/"           # quotes are added by JavaScript

html = requests.get(URL, timeout=20).text
print("requests sees", html.count('class="quote"'), "quotes")

options = webdriver.EdgeOptions()
options.add_argument("--headless=new")             # no visible window
driver = webdriver.Edge(options=options)
driver.get(URL)
quotes = driver.find_elements(By.CSS_SELECTOR, "div.quote")
print("Selenium sees", len(quotes), "quotes")
print(quotes[0].find_element(By.CSS_SELECTOR, "small.author").text)
driver.quit()

Output:

requests sees 0 quotes
Selenium sees 10 quotes
Albert Einstein
Command Prompt output of Python requests finding 0 quotes on a JavaScript page while Selenium with headless Edge finds 10
Same URL: 0 quotes with Requests, 10 with Selenium.

Selenium Manager, built into Selenium 4, downloads the matching browser driver for you, so there’s no manual driver setup. If the page doesn’t need JavaScript, Requests and Beautiful Soup are faster; see web scraping with Python.

Selenium waits and clicking through pages

Your pages load in stages, so don’t grab elements the instant a page opens. WebDriverWait pauses until an element appears, and .click() presses the Next button:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.EdgeOptions()
options.add_argument("--headless=new")
driver = webdriver.Edge(options=options)
wait = WebDriverWait(driver, 10)

authors = []
driver.get("https://quotes.toscrape.com/js/")
for page in range(1, 4):
    wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "div.quote")))   # wait for JS to render
    for q in driver.find_elements(By.CSS_SELECTOR, "div.quote"):
        authors.append(q.find_element(By.CSS_SELECTOR, "small.author").text)
    print(f"page {page}: {len(authors)} quotes so far")
    driver.find_element(By.CSS_SELECTOR, "li.next > a").click()             # go to the next page

driver.quit()
print(len(set(authors)), "different authors, e.g.", sorted(set(authors))[:4])

Output:

page 1: 10 quotes so far
page 2: 20 quotes so far
page 3: 30 quotes so far
20 different authors, e.g. ['Albert Einstein', 'Allen Saunders', 'André Gide', 'Bob Marley']
Command Prompt output of a Python Selenium scraper waiting for quotes and clicking Next through three JavaScript-rendered pages
Three pages, ten quotes each, collected by clicking the site’s own Next link.

I always use explicit waits instead of time.sleep(): they move on as soon as the element is ready and fail clearly if it never appears.

How do you fill in a form with Selenium?

You find each field, type with send_keys(), and click the submit button. The practice site’s login page accepts any test username:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.EdgeOptions()
options.add_argument("--headless=new")
options.add_argument("--window-size=1000,700")
driver = webdriver.Edge(options=options)

driver.get("https://quotes.toscrape.com/login")      # practice form; accepts any test username
driver.find_element(By.ID, "username").send_keys("demo_user")
driver.find_element(By.ID, "password").send_keys("not-a-real-password")
driver.find_element(By.CSS_SELECTOR, "input[type=submit]").click()

WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.LINK_TEXT, "Logout")))
print("logged in:", bool(driver.find_elements(By.LINK_TEXT, "Logout")))
print("page title:", driver.title)
driver.save_screenshot("selenium_after_login.png")    # a real screenshot of the page
driver.quit()

Output:

logged in: True
page title: Quotes to Scrape
Command Prompt output of Python Selenium filling in a practice login form, confirming the Logout link and the page title
Selenium typed into both fields, submitted, and waited for the Logout link to confirm success.

And here’s the screenshot Selenium saved from the headless browser after logging in:

Screenshot captured by Python Selenium from headless Microsoft Edge showing the Quotes to Scrape page after logging in
driver.save_screenshot() captures exactly what the headless browser rendered.

I’d only automate logins on sites you’re allowed to, and never put real passwords in a script; read them from environment variables instead.

Selenium locators cheat sheet

Find byCode
CSS selectordriver.find_element(By.CSS_SELECTOR, "div.quote")
IDdriver.find_element(By.ID, "username")
Link textdriver.find_element(By.LINK_TEXT, "Logout")
XPathdriver.find_element(By.XPATH, "//li[@class='next']/a")
All matchesdriver.find_elements(...) (plural returns a list)

Scraping responsibly with Selenium

  • Check the site’s terms and robots.txt, and prefer an official API when there is one.
  • Slow down: real browsers are heavy, and many fast requests can burden a small site.
  • Always call driver.quit(), ideally in a try/finally, so browsers don’t pile up in the background.
  • Don’t collect personal data you don’t have a right to.

For lighter, faster scraping of static pages, compare the options in Python libraries for web scraping.

More scraping and automation guides:

Frequently asked questions

What is web scraping with Selenium in Python?

It’s using Selenium to control a real browser from Python so you can load JavaScript-heavy pages, click, type and then read the content.

Do I need to download a driver for Selenium?

Not with Selenium 4.6 and newer. Selenium Manager finds or downloads the right driver automatically.

How do I run Selenium without opening a window?

Add the headless option: options.add_argument("--headless=new"), then pass options=options to the driver.

Why does Selenium say no such element?

The element may not have loaded yet. Use WebDriverWait with an expected condition, and double-check the selector.

Is Selenium better than Beautiful Soup?

For JavaScript pages and interaction, yes. For static pages, Requests plus Beautiful Soup is much faster and lighter.

Where is Selenium documented?