Staleelementreferenceexception on Python Selenium

StaleElementReferenceException: Message: stale element reference: element is not attached to the page document error when using Selenium and Python

To print the value of the href attribute you have to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

  • Using CSS_SELECTOR:

    driver.get("https://www.jumia.ug/always/")
    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[aria-label='newsletter_popup_close-cta']"))).click()
    print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.core[data-brand='Always'][href]")))])
  • Using XPATH:

    driver.get("https://www.jumia.ug/always/")
    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@aria-label='newsletter_popup_close-cta']"))).click()
    print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//a[@class='core' and @data-brand='Always'][@href]")))])
  • Console Output:

    ['https://www.jumia.ug/always-maxi-long-7s-15736010.html', 'https://www.jumia.ug/always-roxanne-maxi-duo-long-16s-15736015.html', 'https://www.jumia.ug/always-ultra-long-s3-8s-15736020.html', 'https://www.jumia.ug/always-roxanne-dreamz-maxi-ex-long-8s-15736016.html', 'https://www.jumia.ug/stainless-steel-thermos-flask-3-litres-silver-always-mpg51110.html', 'https://www.jumia.ug/always-roxanne-dreamz-maxi-ex-long-16s-15736017.html', 'https://www.jumia.ug/always-ultra-new-12s-vp-sup-pr-african-15736026.html', 'https://www.jumia.ug/travel-vacuum-thermo-cup-450ml-red-always-mpg59272.html', 'https://www.jumia.ug/pressing-flask-3litres-silver-always-mpg49765.html', 'https://www.jumia.ug/stainless-steel-travel-mug-silver-always-mpg69987.html', 'https://www.jumia.ug/unbreakable-2.5-litres-vaccum-flask-silver-always-mpg54323.html', 'https://www.jumia.ug/stainless-steel-vacuum-flask-pressing-3.5l-silver-always-mpg56188.html', 'https://www.jumia.ug/always-dailies-flexistyle-slim-panty-liners-breathable-flexible-with-fresh-scent-pack-of-26-10968170.html', 'https://www.jumia.ug/stainless-steel-vacuum-flask-500ml-black-always-mpg48945.html', 'https://www.jumia.ug/always-flask-pressing-colour-silver-16418821.html', 'https://www.jumia.ug/vaccum-travel-flask-450ml-blue-always-mpg73899.html', 'https://www.jumia.ug/500mls-vacuum-hot-cold-bottle-flask-sliver-always-mpg54381.html', 'https://www.jumia.ug/always-ultra-lw-14s-vp-sup-pr-african-15736025.html', 'https://www.jumia.ug/pressing-2.0-litres-unbreakable-vaccum-jar-flask-stainless-steel-always-mpg56170.html', 'https://www.jumia.ug/stainless-steel-travel-cap-black-always-mpg65745.html', 'https://www.jumia.ug/always-stainless-steel-travel-mug-0.48l-navy-blue-8224366.html', 'https://www.jumia.ug/always-thermal-flask-cup-silver-black-20572534.html', 'https://www.jumia.ug/stainless-vacuum-travel-mug-450ml-black-always-mpg56189.html', 'https://www.jumia.ug/0.5l-insulated-stainless-steel-travel-mug-colour-varies-always-mpg73677.html', 'https://www.jumia.ug/stainless-steel-travel-mug-0.5l-gold-always-mpg67096.html', 'https://www.jumia.ug/ultra-platinum-long-7-pads-always-mpg59273.html', 'https://www.jumia.ug/life-travel-flask-450ml-red-always-mpg72907.html', 'https://www.jumia.ug/always-xtra-protection-feminine-panty-liners-extra-long-92-pieces-us-4766946.html', 'https://www.jumia.ug/always-zzz-overnight-pads-for-women-size-6-with-wings-for-worry-free-nights-20ct-14336082.html', 'https://www.jumia.ug/portable-vacuum-thermos-flask-bottle-0.5ltr-silver-always-mpg48676.html', 'https://www.jumia.ug/stainless-steel-vacuum-flask-700ml-silver-always-mpg59271.html', 'https://www.jumia.ug/always-vacuum-hot-cold-bottle-flask-500mls-always-mpg46382.html', 'https://www.jumia.ug/stainlesss-steel-side-pressing-vaccum-flask-3litres-silver-always-mpg65748.html', 'https://www.jumia.ug/always-stainless-steel-thermos-flask-jug-silver-always-mpg42506.html', 'https://www.jumia.ug/always-450ml-thermal-flask-maroon-5494832.html', 'https://www.jumia.ug/always-food-flask-18949481.html', 'https://www.jumia.ug/always-portable-vacuum-1.0ltr-thermos-flask-bottle-silver-8032566.html', 'https://www.jumia.ug/always-stainless-vaccum-flask-3-litre-silver-black-always-mpg46820.html', 'https://www.jumia.ug/stainless-steel-vaccum-flask-3.5l-silver-always-mpg48943.html', 'https://www.jumia.ug/always-stainless-steel-vacuum-flask-silver-5196899.html']
  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC

Python Selenium StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

Since I have no credentials to enter that site I can only guess.

So my guess is: that site uses dynamic DOM.

It means that after the initial appearance the web elements continue re-build, so even after Selenium caught some specific element to be clickable or visible this element may instantly continue changing so that the initial element reference becomes stale.

You can try treat this problem by 2 ways:

  1. Set first wait condition, after that set a delay and after that another wait condition. Something like this:
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver,60)
wait.until(EC.element_to_be_clickable((By.XPATH,"//input[@id='ContentPlaceHolder1_rcmbCapacityTranch_Input']")))
time.sleep(2)
wait.until(EC.element_to_be_clickable((By.XPATH,"//input[@id='ContentPlaceHolder1_rcmbCapacityTranch_Input']"))).click()

  1. After you caught the element try clicking it. In case Stale element exception is raised wait until the element is clickable again and try clicking it again until click is succeed. With such loop you will normally make a click during 2-3 iterations.

    So please try this:
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver,60)
succeed = False
while !succeed:
try:
wait.until(EC.element_to_be_clickable((By.XPATH,"//input[@id='ContentPlaceHolder1_rcmbCapacityTranch_Input']"))).click()
succeed = True
except:
pass

StaleElementReferenceException issue while scraping with Selenium

There are many ways to handle stale element reference.

One is like try to re-click on the web element in a while loop.

Your link_text also looks wrong, Please use the below xpath :

# click cookies popup
driver.find_element_by_xpath('//*[(@id = "description")]//*[contains(concat( " ", @class, " " ), concat( " ", "tc-open-privacy-center", " " ))]').click()

time.sleep(10)

# click show more button until no more results to load
while True:
try:
more_button = wait.until(EC.visibility_of_element_located((By.XPATH, "//a[starts-with(@onclick,'tagDeClick') and contains(@href,'/offres/emploi.rechercheoffre:afficherplusderesultats')]")))
ActionChains(driver).move_to_element(more_button).perform()
attempts = 0
while attempts < 2 :
try:
more_button.click()
break
except StaleElementReferenceException as exception:
print(exception.msg)
attempts = attempts + 1

except TimeoutException:
break

time.sleep(10)

print(driver.page_source)
print("Complete")

time.sleep(10)

Output :

stale element reference: element is not attached to the page document
(Session info: chrome=94.0.4606.81)

If you see this in logs, and you do not wish to see this, you will have to comment print(exception.msg).

Imports :

from selenium.webdriver.common.action_chains import ActionChains

Python Selenium Error: Message: stale element reference: element is not attached to the page document

You can use the normal click method, store the element in the list, get the element index and click on the index.

you can update your code with the below one

video_text = driver.find_elements_by_class_name("entry-thumbnails-link")
WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'entry-thumbnails-link')))
video_text[0].click()

The complete code will look like this

Vids = WebDriverWait(driver, 10).until(EC.visibility_of_any_elements_located((By.CLASS_NAME, 'entry-thumbnails-link')))
for title in Vids:
time.sleep(3)
actions = ActionChains(driver)
video_text = driver.find_elements_by_class_name("entry-thumbnails-link")
WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'entry-thumbnails-link')))
video_text[0].click()
time.sleep(5)
VidUrl = driver.current_url
VidTitle = driver.find_element_by_xpath('//*[@id="post-69331"]/h1/a').text
try:
VidTags = driver.find_elements_by_class_name('tags')
for tag in VidTags:
VidTag = tag.find_element_by_tag_name('a').text

except NoSuchElementException or StaleElementReferenceException:
pass

with open('data.csv', 'w', newline='', encoding="utf-8") as f:
fieldnames = ['Title', 'Tags', 'URL']
thewriter = csv.DictWriter(f, fieldnames=fieldnames)

thewriter.writeheader()
thewriter.writerow({'Title': VidTitle, 'Tags': VidTag, 'URL': VidUrl})
driver.get(URL)
print('done')

Also try not to use these many sleep().

StaleElementReferenceException in Python

For each click on the "Next" button -- you should find that button and click on it.

Or do something like this:

max_attemps = 10

while True:

next = self.driver.find_element_by_css_selector(".next>a")

if next is not None:

break

else:

time.sleep(0.5)
max_attemps -= 1

if max_attemps == 0:

self.fail("Cannot find element.")

And after this code does click action.

PS: Also try to add just time.sleep(x) after fiding element and then do click action.

StaleElementReferenceException on Python Selenium while using for loop

You have to define this

 driver.find_elements_by_xpath('//a[@class="sc-dBAPYN kcrxQo"]')

again in loop.

Code :

number_of_titles = len(driver.find_elements_by_xpath('//a[@class="sc-dBAPYN kcrxQo"]'))

#this is block of code which is error prone
j = 0
for i in range(number_of_titles):
time.sleep(5)
titles = driver.find_elements_by_xpath('//a[@class="sc-dBAPYN kcrxQo"]')
driver.execute_script("window.scrollTo(0, 0);")
element = titles[j].find_element_by_xpath('.//div//h4')
driver.execute_script("arguments[0].click();", element)
time.sleep(3)
j = j +1
name_of_rests = driver.find_element_by_css_selector('#root > div > main > div > section.sc-kxynE.jzTfFZ > section > section > div > div > div > h1').text
res_list.append(name_of_rests)

driver.back()

**Python selenium error** :Message: stale element reference: element is not attached to the page document

The web element tweets found before the for loop becomes inaccessible inside the for loop at the end for some reason.

So I tried initiating tweets again inside the for loop and it works now. It seems to be a temporary fix, but works.

The last block of my code now :

tweet_contents = []
for _ in range(5):
tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]")
time.sleep(1)
for tweet in tweets:
print(tweet.text)
tweet_contents += [tweet.text]
tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]") # initialising tweets variable again inside for loop
body.send_keys(Keys.PAGE_DOWN)
time.sleep(0.4)

tweet_contents = sorted(set(tweet_contents), key=tweet_contents.index) # To remove duplicates, whilst preserving the order of tweets

How to fix StaleElementReferenceException on Python Selenium?

Solution:

all_available_options_nivel_ensino = create_list_nivel_ensino()

all_available_options_unidades = create_list_unidades()

for nivel_ensino in all_available_options_nivel_ensino:
select_element_nivel_ensino = driver.find_element(By.ID, "formTurma:inputNivel")
select_object_nivel_ensino = Select(select_element_nivel_ensino)
select_object_nivel_ensino.select_by_value(nivel_ensino)
for unidades in all_available_options_unidades:
select_element_unidades = driver.find_element(By.ID, "formTurma:inputDepto")
select_object_unidades = Select(select_element_unidades)
select_object_unidades.select_by_value(unidades)
driver.find_element(By.NAME, "formTurma:j_id_jsp_1370969402_11").click()


Related Topics



Leave a reply



Submit