StaleElementReferenceException: Message: stale element reference: element is not attached to the page document error when using Selenium and Python
To print the value of the href attribute you have to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get("https://www.jumia.ug/always/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[aria-label='newsletter_popup_close-cta']"))).click()
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.core[data-brand='Always'][href]")))])Using XPATH:
driver.get("https://www.jumia.ug/always/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@aria-label='newsletter_popup_close-cta']"))).click()
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//a[@class='core' and @data-brand='Always'][@href]")))])Console Output:
['https://www.jumia.ug/always-maxi-long-7s-15736010.html', 'https://www.jumia.ug/always-roxanne-maxi-duo-long-16s-15736015.html', 'https://www.jumia.ug/always-ultra-long-s3-8s-15736020.html', 'https://www.jumia.ug/always-roxanne-dreamz-maxi-ex-long-8s-15736016.html', 'https://www.jumia.ug/stainless-steel-thermos-flask-3-litres-silver-always-mpg51110.html', 'https://www.jumia.ug/always-roxanne-dreamz-maxi-ex-long-16s-15736017.html', 'https://www.jumia.ug/always-ultra-new-12s-vp-sup-pr-african-15736026.html', 'https://www.jumia.ug/travel-vacuum-thermo-cup-450ml-red-always-mpg59272.html', 'https://www.jumia.ug/pressing-flask-3litres-silver-always-mpg49765.html', 'https://www.jumia.ug/stainless-steel-travel-mug-silver-always-mpg69987.html', 'https://www.jumia.ug/unbreakable-2.5-litres-vaccum-flask-silver-always-mpg54323.html', 'https://www.jumia.ug/stainless-steel-vacuum-flask-pressing-3.5l-silver-always-mpg56188.html', 'https://www.jumia.ug/always-dailies-flexistyle-slim-panty-liners-breathable-flexible-with-fresh-scent-pack-of-26-10968170.html', 'https://www.jumia.ug/stainless-steel-vacuum-flask-500ml-black-always-mpg48945.html', 'https://www.jumia.ug/always-flask-pressing-colour-silver-16418821.html', 'https://www.jumia.ug/vaccum-travel-flask-450ml-blue-always-mpg73899.html', 'https://www.jumia.ug/500mls-vacuum-hot-cold-bottle-flask-sliver-always-mpg54381.html', 'https://www.jumia.ug/always-ultra-lw-14s-vp-sup-pr-african-15736025.html', 'https://www.jumia.ug/pressing-2.0-litres-unbreakable-vaccum-jar-flask-stainless-steel-always-mpg56170.html', 'https://www.jumia.ug/stainless-steel-travel-cap-black-always-mpg65745.html', 'https://www.jumia.ug/always-stainless-steel-travel-mug-0.48l-navy-blue-8224366.html', 'https://www.jumia.ug/always-thermal-flask-cup-silver-black-20572534.html', 'https://www.jumia.ug/stainless-vacuum-travel-mug-450ml-black-always-mpg56189.html', 'https://www.jumia.ug/0.5l-insulated-stainless-steel-travel-mug-colour-varies-always-mpg73677.html', 'https://www.jumia.ug/stainless-steel-travel-mug-0.5l-gold-always-mpg67096.html', 'https://www.jumia.ug/ultra-platinum-long-7-pads-always-mpg59273.html', 'https://www.jumia.ug/life-travel-flask-450ml-red-always-mpg72907.html', 'https://www.jumia.ug/always-xtra-protection-feminine-panty-liners-extra-long-92-pieces-us-4766946.html', 'https://www.jumia.ug/always-zzz-overnight-pads-for-women-size-6-with-wings-for-worry-free-nights-20ct-14336082.html', 'https://www.jumia.ug/portable-vacuum-thermos-flask-bottle-0.5ltr-silver-always-mpg48676.html', 'https://www.jumia.ug/stainless-steel-vacuum-flask-700ml-silver-always-mpg59271.html', 'https://www.jumia.ug/always-vacuum-hot-cold-bottle-flask-500mls-always-mpg46382.html', 'https://www.jumia.ug/stainlesss-steel-side-pressing-vaccum-flask-3litres-silver-always-mpg65748.html', 'https://www.jumia.ug/always-stainless-steel-thermos-flask-jug-silver-always-mpg42506.html', 'https://www.jumia.ug/always-450ml-thermal-flask-maroon-5494832.html', 'https://www.jumia.ug/always-food-flask-18949481.html', 'https://www.jumia.ug/always-portable-vacuum-1.0ltr-thermos-flask-bottle-silver-8032566.html', 'https://www.jumia.ug/always-stainless-vaccum-flask-3-litre-silver-black-always-mpg46820.html', 'https://www.jumia.ug/stainless-steel-vaccum-flask-3.5l-silver-always-mpg48943.html', 'https://www.jumia.ug/always-stainless-steel-vacuum-flask-silver-5196899.html']
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Python Selenium StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
Since I have no credentials to enter that site I can only guess.
So my guess is: that site uses dynamic DOM.
It means that after the initial appearance the web elements continue re-build, so even after Selenium caught some specific element to be clickable or visible this element may instantly continue changing so that the initial element reference becomes stale.
You can try treat this problem by 2 ways:
- Set first
wait
condition, after that set a delay and after that anotherwait
condition. Something like this:
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver,60)
wait.until(EC.element_to_be_clickable((By.XPATH,"//input[@id='ContentPlaceHolder1_rcmbCapacityTranch_Input']")))
time.sleep(2)
wait.until(EC.element_to_be_clickable((By.XPATH,"//input[@id='ContentPlaceHolder1_rcmbCapacityTranch_Input']"))).click()
- After you caught the element try clicking it. In case Stale element exception is raised wait until the element is clickable again and try clicking it again until click is succeed. With such loop you will normally make a click during 2-3 iterations.
So please try this:
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver,60)
succeed = False
while !succeed:
try:
wait.until(EC.element_to_be_clickable((By.XPATH,"//input[@id='ContentPlaceHolder1_rcmbCapacityTranch_Input']"))).click()
succeed = True
except:
pass
StaleElementReferenceException issue while scraping with Selenium
There are many ways to handle stale element reference.
One is like try to re-click on the web element in a while loop.
Your link_text also looks wrong, Please use the below xpath :
# click cookies popup
driver.find_element_by_xpath('//*[(@id = "description")]//*[contains(concat( " ", @class, " " ), concat( " ", "tc-open-privacy-center", " " ))]').click()
time.sleep(10)
# click show more button until no more results to load
while True:
try:
more_button = wait.until(EC.visibility_of_element_located((By.XPATH, "//a[starts-with(@onclick,'tagDeClick') and contains(@href,'/offres/emploi.rechercheoffre:afficherplusderesultats')]")))
ActionChains(driver).move_to_element(more_button).perform()
attempts = 0
while attempts < 2 :
try:
more_button.click()
break
except StaleElementReferenceException as exception:
print(exception.msg)
attempts = attempts + 1
except TimeoutException:
break
time.sleep(10)
print(driver.page_source)
print("Complete")
time.sleep(10)
Output :
stale element reference: element is not attached to the page document
(Session info: chrome=94.0.4606.81)
If you see this in logs
, and you do not wish to see this, you will have to comment print(exception.msg)
.
Imports :
from selenium.webdriver.common.action_chains import ActionChains
Python Selenium Error: Message: stale element reference: element is not attached to the page document
You can use the normal click method, store the element in the list, get the element index and click on the index.
you can update your code with the below one
video_text = driver.find_elements_by_class_name("entry-thumbnails-link")
WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'entry-thumbnails-link')))
video_text[0].click()
The complete code will look like this
Vids = WebDriverWait(driver, 10).until(EC.visibility_of_any_elements_located((By.CLASS_NAME, 'entry-thumbnails-link')))
for title in Vids:
time.sleep(3)
actions = ActionChains(driver)
video_text = driver.find_elements_by_class_name("entry-thumbnails-link")
WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'entry-thumbnails-link')))
video_text[0].click()
time.sleep(5)
VidUrl = driver.current_url
VidTitle = driver.find_element_by_xpath('//*[@id="post-69331"]/h1/a').text
try:
VidTags = driver.find_elements_by_class_name('tags')
for tag in VidTags:
VidTag = tag.find_element_by_tag_name('a').text
except NoSuchElementException or StaleElementReferenceException:
pass
with open('data.csv', 'w', newline='', encoding="utf-8") as f:
fieldnames = ['Title', 'Tags', 'URL']
thewriter = csv.DictWriter(f, fieldnames=fieldnames)
thewriter.writeheader()
thewriter.writerow({'Title': VidTitle, 'Tags': VidTag, 'URL': VidUrl})
driver.get(URL)
print('done')
Also try not to use these many sleep().
StaleElementReferenceException in Python
For each click on the "Next" button -- you should find that button and click on it.
Or do something like this:
max_attemps = 10
while True:
next = self.driver.find_element_by_css_selector(".next>a")
if next is not None:
break
else:
time.sleep(0.5)
max_attemps -= 1
if max_attemps == 0:
self.fail("Cannot find element.")
And after this code does click action.
PS: Also try to add just time.sleep(x)
after fiding element and then do click action.
StaleElementReferenceException on Python Selenium while using for loop
You have to define this
driver.find_elements_by_xpath('//a[@class="sc-dBAPYN kcrxQo"]')
again in loop.
Code :
number_of_titles = len(driver.find_elements_by_xpath('//a[@class="sc-dBAPYN kcrxQo"]'))
#this is block of code which is error prone
j = 0
for i in range(number_of_titles):
time.sleep(5)
titles = driver.find_elements_by_xpath('//a[@class="sc-dBAPYN kcrxQo"]')
driver.execute_script("window.scrollTo(0, 0);")
element = titles[j].find_element_by_xpath('.//div//h4')
driver.execute_script("arguments[0].click();", element)
time.sleep(3)
j = j +1
name_of_rests = driver.find_element_by_css_selector('#root > div > main > div > section.sc-kxynE.jzTfFZ > section > section > div > div > div > h1').text
res_list.append(name_of_rests)
driver.back()
**Python selenium error** :Message: stale element reference: element is not attached to the page document
The web element tweets
found before the for loop becomes inaccessible inside the for loop at the end for some reason.
So I tried initiating tweets
again inside the for loop and it works now. It seems to be a temporary fix, but works.
The last block of my code now :
tweet_contents = []
for _ in range(5):
tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]")
time.sleep(1)
for tweet in tweets:
print(tweet.text)
tweet_contents += [tweet.text]
tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]") # initialising tweets variable again inside for loop
body.send_keys(Keys.PAGE_DOWN)
time.sleep(0.4)
tweet_contents = sorted(set(tweet_contents), key=tweet_contents.index) # To remove duplicates, whilst preserving the order of tweets
How to fix StaleElementReferenceException on Python Selenium?
Solution:
all_available_options_nivel_ensino = create_list_nivel_ensino()
all_available_options_unidades = create_list_unidades()
for nivel_ensino in all_available_options_nivel_ensino:
select_element_nivel_ensino = driver.find_element(By.ID, "formTurma:inputNivel")
select_object_nivel_ensino = Select(select_element_nivel_ensino)
select_object_nivel_ensino.select_by_value(nivel_ensino)
for unidades in all_available_options_unidades:
select_element_unidades = driver.find_element(By.ID, "formTurma:inputDepto")
select_object_unidades = Select(select_element_unidades)
select_object_unidades.select_by_value(unidades)
driver.find_element(By.NAME, "formTurma:j_id_jsp_1370969402_11").click()
Related Topics
Convert Rgb Color to English Color Name, Like 'Green' With Python
Cqlsh Connection Error: 'Ref() Does Not Take Keyword Arguments'
How to Iterate Through Two Lists in Parallel
Convert Columns into Rows With Pandas
Error "Microsoft Visual C++ 14.0 Is Required (Unable to Find Vcvarsall.Bat)"
If/Else in a List Comprehension
How to Post Json Data With Python Requests
How to Terminate a Python Subprocess Launched With Shell=True
Django [Errno 13] Permission Denied: '/Var/Www/Media/Animals/User_Uploads'
Get the Olson Tz Name For the Local Timezone
Why Is Using 'Eval' a Bad Practice
I'M Getting an Indentationerror. How to Fix It
How to Access Object Attribute Given String Corresponding to Name of That Attribute
Are a Wsgi Server and Http Server Required to Serve a Flask App