How to Use Selenium with Python

How to use Selenium with Python?

You mean Selenium WebDriver?
Huh....

Prerequisite: Install Python based on your OS

Install with following command

pip install -U selenium

And use this module in your code

from selenium import webdriver

You can also use many of the following as required

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException

Here is an updated answer

I would recommend you to run script without IDE... Here is my approach

1. USE IDE to find xpath of object / element
2. And use find_element_by_xpath().click()

An example below shows login page automation

#ScriptName : Login.py
#---------------------
from selenium import webdriver

#Following are optional required
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException

baseurl = "http://www.mywebsite.com/login.php"
username = "admin"
password = "admin"

xpaths = { 'usernameTxtBox' : "//input[@name='username']",
           'passwordTxtBox' : "//input[@name='password']",
           'submitButton' :   "//input[@name='login']"
         }

mydriver = webdriver.Firefox()
mydriver.get(baseurl)
mydriver.maximize_window()

#Clear Username TextBox if already allowed "Remember Me" 
mydriver.find_element_by_xpath(xpaths['usernameTxtBox']).clear()

#Write Username in Username TextBox
mydriver.find_element_by_xpath(xpaths['usernameTxtBox']).send_keys(username)

#Clear Password TextBox if already allowed "Remember Me" 
mydriver.find_element_by_xpath(xpaths['passwordTxtBox']).clear()

#Write Password in password TextBox
mydriver.find_element_by_xpath(xpaths['passwordTxtBox']).send_keys(password)

#Click Login button
mydriver.find_element_by_xpath(xpaths['submitButton']).click()

There is an another way that you can find xpath of any object -

1. Install Firebug and Firepath addons in firefox
2. Open URL in Firefox
3. Press F12 to open Firepath developer instance
4. Select Firepath in below browser pane and chose select by "xpath"
5. Move cursor of the mouse to element on webpage
6. in the xpath textbox you will get xpath of an object/element.
7. Copy Paste xpath to the script.

Run script -

python Login.py

You can also use a CSS selector instead of xpath. CSS selectors are slightly faster than xpath in most cases, and are usually preferred over xpath (if there isn't an ID attribute on the elements you're interacting with).

Firepath can also capture the object's locator as a CSS selector if you move your cursor to the object. You'll have to update your code to use the equivalent find by CSS selector method instead -

find_element_by_css_selector(css_selector) 

How to use selenium on Repl.it?

You need to go to replit.nix , click on hidden files and you are going to see it,
you need to import pkgs.chromium and pkgs.chromedriver into deps

{ pkgs }: {
deps = [
    pkgs.python38Full
    pkgs.chromium
    pkgs.chromedriver
];

don't forget to refresh the page

How to Use Selenium to Click Button/Link on Website in Python

driver.get() does not wait for AJAX requests to complete before returning. The element you are trying to click is created via an AJAX request and does not yet exist at the time you are trying to click it.

To get around this, you must first wait for the element to be created, as in the following code.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.Firefox(executable_path='xxx/geckodriver.exe'))
driver.get('https://mymarketnews.ams.usda.gov/viewReport/2837')
WebDriverWait(driver, timeout=10).until(EC.presence_of_element_located((By.ID, '2020s')))
elt = driver.find_element(By.XPATH, '//*[@id="2020s"]/a')
elt.click()

Two other things to note as you continue to work on this project are

• CSS selectors have a limitation where they cannot be used to find an element with an ID starting with a number.
• You need to click the <a> element directly, not the <li> element surrounding it.

How to search in a webpage using selenium despite using correct search id

The Search field is not visible by default, to send a character sequence to the search field you need to click() on the search icon first inducing WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

• Using CSS_SELECTOR:

  driver.get("https://www.gartner.com/en")
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#onetrust-accept-btn-handler"))).click()
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "span.nav-icon.gcom-icon-search"))).click()
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#searchString"))).send_keys("Risk Management Solution" + Keys.RETURN)
  

• Using XPATH:

  driver.get("https://www.gartner.com/en")
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@id='onetrust-accept-btn-handler']"))).click()
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[@class='nav-icon gcom-icon-search']"))).click()
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='searchString']"))).send_keys("Risk Management Solution" + Keys.RETURN)
  

• Note: You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

• Browser Snapshot:

How to use if statement with selenium python?

You can tackle this situation with find_elements as well.

find_elements

will return a list of web element if found, if not then it won't throw any error instead size of the list will be zero.

try:
    if len(driver.find_elements(By.NAME, "selectAllCurrentMPNs")) > 0:
        #follow some steps...
    else:
        #do something when `selectAllCurrentMPNs` web element is not present.
except:
    pass

How to call a method within a Python class using Selenium and Python

Please note - Never ever provide credentials

What happens?

You mentioned you have to perform a login and selenium is a good choice, but what you are doing is to call parse() that only performce a call via requests. So if you take a look into your soup you wont find what you are looking for.

How to fix?

Perform your selenium actions and walk to the website you want to scrape. In next step push your driver.page_source into BeautifulSoup and find your elements:

soup = BeautifulSoup(driver.page_source,'html.parser')
items = soup.find_all('a', class_='Antineoplastic')

print(items)

If your selection is right, you will get your result.

EDIT

Concerning your comments a clue where you can end up, for debugging steps between, you should ask as new question with focused examples:

import requests
from bs4 import BeautifulSoup
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

driver = webdriver.Chrome()
 
driver.get("http://way2drug.com/passonline/")
driver.set_window_size(1920, 1030)
driver.find_element(By.CSS_SELECTOR, "#registration img").click()
driver.find_element(By.NAME, "user_login").click()
driver.find_element(By.NAME, "user_login").send_keys("MY USER")
driver.find_element(By.ID, "page1").click()
driver.find_element(By.NAME, "user_password").send_keys("MY PASS")
driver.find_element(By.ID, "register").click()
driver.find_element(By.ID, "myHeader1").click()
driver.find_element(By.ID, "smiles").click()
driver.find_element(By.ID, "smi").click()
driver.find_element(By.ID, "smi").send_keys("CC1(C)C(O)CC[C@@]2(C)C1CC[C@]3(C)C2CCC4[C@@]3(C)CC[C@]5(C(O)=O)C4[C@H](C)C(C)=CC5")
driver.find_element(By.CSS_SELECTOR, "#myContent4 input:nth-child(4)").click()

soup = BeautifulSoup(driver.page_source,'html.parser')
items = soup.find_all('a', class_='Antineoplastic')

print(items)

driver.quit()

Selenium, Python - Use For-loop in finding values in a column, but resulted in duplicating first values instead

You are traversing the whole page instead of just the rows. And since you are searching for single element in the loop, it returns the first td element of the page. It should be

for i, col in enumerate(rows):
  name = col.find_element(By.TAG_NAME,'td').text
  print(f'Row: {i}, Name: {name}')

---

Related Topics