How to Make Sure If Some HTML Elements Are Loaded for Selenium + Python

How can I make sure if some HTML elements are loaded for Selenium + Python?

As per your code trials you are using ChromeDriver and Chrome Browser to Automate the steps. As you have configured set_page_load_timeout(10) the timeout exception is raised as the page havn't completely loaded within the timeframe configured through set_page_load_timeout(). But as you have invoked print(self.page_source) the elements rendered within the partially rendered HTML DOM are retrieved.

Now about your individual queries :

  • How can I make sure some elements are already loaded? : An ideal testcase would have a definite step e.g. to validate the presence of an element, to validate the visibility of an element or to validate the interactability (while clicking) of element. From this perspective verifying the elements are already loaded may not include the desired element. Hence instead of such a broader search criteria you need to narrow down your search criteria to some thing definite e.g.

    • Page Title
    • Page Heading
    • Presence of an Alert
    • Attribute of an element
    • Presence of an element
    • Presence of a group of elements
    • Visibility of an element
    • Visibility of a group of elements
    • Clickablity of an element
    • StalenessOf of an element
    • FrameToBeAvailableAndSwitchToIt

Implementing these narrowed down search criteria can save a lot of execution time with the help of WebDriverWait inconjunction with expected_conditions.

  • How can I make sure all elements are loaded? : Again, our tests should be focused only on the element/elements with which we need to interact with and leaveout verifying the status/condition of other elements which are not of our interest.

  • Now, following the two points mentioned above these are the 3 most commonly used usecases :

    • presence_of_element_located : An expectation for checking that an element is present on the DOM of a page. This does not necessarily mean that the element is visible.
    • visibility_of_element_located : An expectation for checking that an element is present on the DOM of a page and visible. Visibility means that the element is not only displayed but also has a height and width that is greater than 0.
    • element_to_be_clickable : An Expectation for checking an element is visible and enabled such that you can click it.
  • As per the usecase you mentioned you can make a List of all the <th> elements visible within the DOM Tree whilst waiting for a configurable amount of time inducing WebDriverWait as follows :

    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC

    headerList = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@class='fth1_' and @id='fth1_']/thead/tr//th")))

Note : The xpath used in this illustration is a sample xpath used only for demonstration purpose.

Wait until page is loaded with Selenium WebDriver for Python

The webdriver will wait for a page to load by default via .get() method.

As you may be looking for some specific element as @user227215 said, you should use WebDriverWait to wait for an element located in your page:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException

browser = webdriver.Firefox()
browser.get("url")
delay = 3 # seconds
try:
myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'IdOfMyElement')))
print "Page is ready!"
except TimeoutException:
print "Loading took too much time!"

I have used it for checking alerts. You can use any other type methods to find the locator.

EDIT 1:

I should mention that the webdriver will wait for a page to load by default. It does not wait for loading inside frames or for ajax requests. It means when you use .get('url'), your browser will wait until the page is completely loaded and then go to the next command in the code. But when you are posting an ajax request, webdriver does not wait and it's your responsibility to wait an appropriate amount of time for the page or a part of page to load; so there is a module named expected_conditions.

Checking if an element exists with Python Selenium

A) Yes. The easiest way to check if an element exists is to simply call find_element inside a try/catch.

B) Yes, I always try to identify elements without using their text for two reasons:

  1. the text is more likely to change and;
  2. if it is important to you, you won't be able to run your tests against localized builds.

The solution is either:

  1. You can use XPath to find a parent or ancestor element that has an ID or some other unique identifier and then find its child/descendant that matches or;
  2. you could request an ID or name or some other unique identifier for the link itself.

For the follow-up questions, using try/catch is how you can tell if an element exists or not and good examples of waits can be found here: http://seleniumhq.org/docs/04_webdriver_advanced.html

Python Selenium wait for several elements to load

Keeping in mind comments of Mr.E. and Arran I made my list traversal fully on CSS selectors. The tricky part was about my own list structure and marks (changing classes, etc.), as well as about creating required selectors on the fly and keeping them in memory during traversal.

I disposed waiting for several elements by searching for anything that is not loading state. You may use ":nth-child" selector as well like here:

#in for loop with enumerate for i    
selector.append(' > li:nth-child(%i)' % (i + 1)) # identify child <li> by its order pos

This is my hard-commented code solution for example:

def parse_crippled_shifted_list(driver, frame, selector, level=1, parent_id=0, path=None):
"""
Traversal of html list of special structure (you can't know if element has sub list unless you enter it).
Supports start from remembered list element.

Nested lists have classes "closed" and "last closed" when closed and "open" and "last open" when opened (on <li>).
Elements themselves have classes "leaf" and "last leaf" in both cases.
Nested lists situate in <li> element as <ul> list. Each <ul> appears after clicking <a> in each <li>.
If you click <a> of leaf, page in another frame will load.

driver - WebDriver; frame - frame of the list; selector - selector to current list (<ul>);
level - level of depth, just for console output formatting, parent_id - id of parent category (in DB),
path - remained path in categories (ORM objects) to target category to start with.
"""

# Add current level list elements
# This method selects all but loading. Just what is needed to exclude.
selector.append(' > li > a:not([class=loading])')

# Wait for child list to load
try:
query = WebDriverWait(driver, WAIT_LONG_TIME).until(
EC.presence_of_all_elements_located((By.CSS_SELECTOR, ''.join(selector))))

except TimeoutException:
print "%s timed out" % ''.join(selector)

else:
# List is loaded
del selector[-1] # selector correction: delete last part aimed to get loaded content
selector.append(' > li')

children = driver.find_elements_by_css_selector(''.join(selector)) # fetch list elements

# Walk the whole list
for i, child in enumerate(children):

del selector[-1] # delete non-unique li tag selector
if selector[-1] != ' > ul' and selector[-1] != 'ul.ltr':
del selector[-1]

selector.append(' > li:nth-child(%i)' % (i + 1)) # identify child <li> by its order pos
selector.append(' > a') # add 'li > a' reference to click

child_link = driver.find_element_by_css_selector(''.join(selector))

# If we parse freely further (no need to start from remembered position)
if not path:
# Open child
try:
double_click(driver, child_link)
except InvalidElementStateException:
print "\n\nERROR\n", InvalidElementStateException.message(), '\n\n'
else:
# Determine its type
del selector[-1] # delete changed and already useless link reference
# If <li> is category, it would have <ul> as child now and class="open"
# Check by class is priority, because <li> exists for sure.
current_li = driver.find_element_by_css_selector(''.join(selector))

# Category case - BRANCH
if current_li.get_attribute('class') == 'open' or current_li.get_attribute('class') == 'last open':
new_parent_id = process_category_case(child_link, parent_id, level) # add category to DB
selector.append(' > ul') # forward to nested list
# Wait for nested list to load
try:
query = WebDriverWait(driver, WAIT_LONG_TIME).until(
EC.presence_of_all_elements_located((By.CSS_SELECTOR, ''.join(selector))))

except TimeoutException:
print "\t" * level, "%s timed out (%i secs). Failed to load nested list." %\
''.join(selector), WAIT_LONG_TIME
# Parse nested list
else:
parse_crippled_shifted_list(driver, frame, selector, level + 1, new_parent_id)

# Page case - LEAF
elif current_li.get_attribute('class') == 'leaf' or current_li.get_attribute('class') == 'last leaf':
process_page_case(driver, child_link, level)
else:
raise Exception('Damn! Alien class: %s' % current_li.get_attribute('class'))

# If it's required to continue from specified category
else:
# Check if it's required category
if child_link.text == path[0].name:
# Open required category
try:
double_click(driver, child_link)

except InvalidElementStateException:
print "\n\nERROR\n", InvalidElementStateException.msg, '\n\n'

else:
# This element of list must be always category (have nested list)
del selector[-1] # delete changed and already useless link reference
# If <li> is category, it would have <ul> as child now and class="open"
# Check by class is priority, because <li> exists for sure.
current_li = driver.find_element_by_css_selector(''.join(selector))

# Category case - BRANCH
if current_li.get_attribute('class') == 'open' or current_li.get_attribute('class') == 'last open':
selector.append(' > ul') # forward to nested list
# Wait for nested list to load
try:
query = WebDriverWait(driver, WAIT_LONG_TIME).until(
EC.presence_of_all_elements_located((By.CSS_SELECTOR, ''.join(selector))))

except TimeoutException:
print "\t" * level, "%s timed out (%i secs). Failed to load nested list." %\
''.join(selector), WAIT_LONG_TIME
# Process this nested list
else:
last = path.pop(0)
if len(path) > 0: # If more to parse
print "\t" * level, "Going deeper to: %s" % ''.join(selector)
parse_crippled_shifted_list(driver, frame, selector, level + 1,
parent_id=last.id, path=path)
else: # Current is required
print "\t" * level, "Returning target category: ", ''.join(selector)
path = None
parse_crippled_shifted_list(driver, frame, selector, level + 1, last.id, path=None)

# Page case - LEAF
elif current_li.get_attribute('class') == 'leaf':
pass
else:
print "dummy"

del selector[-2:]


Related Topics



Leave a reply



Submit