Loop Over List of Elements for Find_Element_By_Xpath() by Selenium and Webdriver

Loop over list of elements for find_element_by_xpath() by Selenium and Webdriver

Try this:

name_list = ["Kate", "David"]

for name in name_list:
    xpath = "//*[contains(text(), '" + str(name) + "')]" # simplified
    print xpath
    list = driver.find_elements_by_xpath(xpath) # locate all elements by xpath
    if len(list) > 0: # if list is not empty, click on element
        list[0].click() # click on the first element in the list
    time.sleep(5)

This will prevent from throwing

common.exceptions.NoSuchElementException

Note: also make sure, that you using the correct xPath.

Looping through to find element by xpath and then click on each element found

You need to format the xpath string.

By doing

"//table/tbody/tr[i]/td/a"

it is literally putting tr[i] into the xpath, not tr[1] etc.,

Instead of

driver.find_element_by_xpath("//table/tbody/tr[i]/td/a").click()

do

x_path = "//table/tbody/tr[{0}]/td/a".format(i)
driver.find_element_by_xpath(x_path).click()

This replaces the {1} in the string with the value you provide in the format function, so in this case it will replace {1} with 1, 2 etc. and will make your xpath selector work as expected.

How to loop to find child elements in Selenium xpath?

Basically you have quite a mess inside a loop.
So in your case you loop over reviews and for each step of the loop you are getting first name of the reviewer all the time.

In xpath if you start search with // it will search from the beginning of the document.

What you would like to do here would be:

for review in driver.find_elements_by_xpath('//div[@id="cm_cr-review_list"]/div[@data-hook="review"]'):
    if review.find_element_by_xpath('.//span[@class="a-profile-name"]') :
        name_row.append(review.find_element_by_xpath('.//span[@class="a-profile-name"]').text)
    else:
        name_row.append("None")

As you can see .// would start the search from within the review HTML.

Second thing is you have a loop to go over pages inside the loop for checking the reviews, and then another loop again for reviews.

Basically you could right away find all names that you need with only one search with xpath.

Here is working example:

import csv
import pandas as pd
from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException


driver = webdriver.Chrome('chromedriver')
# Set driver to wait up to 3 seconds for the elements to appear if we try to find them
driver.implicitly_wait(3)

url = "https://www.amazon.com/Arti-Cipes-Electric-Cosmetic-Automatic-Machine-Spinner/product-reviews/B08QMKN8Z8/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews"
driver.get(url)
    
item_name = driver.find_element_by_xpath('//div[@class="a-row product-title"]/h1/a').text
    
name_row = []

while True:
    try:
        # Get all name rows
        name_rows = driver.find_elements_by_xpath('.//div[@id="cm_cr-review_list"]//span[@class="a-profile-name"]')
        # Get text from each span element
        names = [row.text for row in name_rows]
        name_row.extend(names)
        
        next_page_button = driver.find_element_by_xpath('//li[@class="a-last"]/a')
        url = next_page_button.get_attribute('href')
        driver.get(url)
        
    except NoSuchElementException:
        break
        
df = pd.DataFrame({'Names': name_row})
df.to_csv(item_name + '.csv', index=False)

driver.close()

There is also room for optimisation because you could use only requests and lxml library since there is no real need to use selenium but this is another scope of the problem

Iterating over List[] takes and composing a dict only uses first list element

Solved it.

Problem was that I forgot to add a . to the XPath query in the get_data() function.
This SO post describes the same problem: Iterating through elements get repeating result on Selenium on Python

If I dont't add a ., the XPath will search from the top of the DOM and return always the same item.

Thank you!

How do i iterate through a webelements list with Python and Selenium?

It was in my face all along,this will print the text from each element, glad i could find out

PlajePariuri = driver.find_elements_by_class_name('KambiBC-bet-offer-category KambiBC-collapsible-container KambiBC-expanded KambiBC-bet-offer-category--hidden KambiBC-bet-offer-category--fade-in')


    for items2 in PlajePariuri:

        NumePlaje = items2.find_element_by_class_name('KambiBC-bet-offer-category__title js-bet-offer-category-title')

        print (NumePlaje.text)

Looping through XPATH SELENIUM PYTHON

If you want to use your existing code then please correct the below line. Just changed the [%b] to [%d] at the end of the xpath string.

Old code:

Teamname= driver.find_element_by_xpath("//*[@id='tab-3959-3']/div/div[1]/div[1]/div/div/div/div[2]/div/div[%b]" %(b))

Updated Code:

Teamname= driver.find_element_by_xpath("//*[@id='tab-3959-3']/div/div[1]/div[1]/div/div/div/div[2]/div/div[%d]" %(b))

Here is the refracted code. I haven't got a chance to test this.

    #click on Games
    driver.find_element_by_css("ul.nav.navbar-nav a[data-value='Games']").click()
    #click on Game Logs
    driver.find_element_by_css_selector("ul.dropdown-menu a[data-value='Game Logs']").click()
    #switch to Teams tab
    driver.find_element_by_css_selector("ul.nav.nav-tabs a[data-value='Teams']").click()
    #click the teams listbox
    teamNames = driver.find_element_by_xpath("//div[@class='tab-pane active' and @data-value='Teams']//label[.='Team:']//parent::div//div[@class='selectize-dropdown-content']").click()
    #get the list of team names
    teams = driver.find_elements_by_xpath("//div[@class='tab-pane active' and @data-value='Teams']//label[.='Team:']//parent::div//div[@class='selectize-dropdown-content']//div[@class='option']")

    # get the list of seasons
    seasons = driver.find_elements_by_xpath("//div[@class='tab-pane active' and @data-value='Teams']//select[@id='game_logs_teams_season']/option")
    # iterate through each team
    for team in teams:
        team.click()
        # iterate through each season
        for season in seasons:
            seanson.click()

---

Related Topics