Don't wait for a page to load using Selenium in Python
ChromeDriver 77.0 (which supports Chrome version 77) now supports eager
as pageLoadStrategy.
Resolved issue 1902: Support eager page load strategy [Pri-2]
As you question mentions of click on elements and scrape data before the page has fully loaded
in this case we can take help of an attribute pageLoadStrategy
. When Selenium loads a page/url by default it follows a default configuration with pageLoadStrategy
set to normal
. Selenium can start executing the next line of code from different Document readiness state
. Currently Selenium supports 3 different Document readiness state
which we can configure through the pageLoadStrategy
as follows:
none
(undefined)eager
(page becomes interactive)normal
(complete page load)
Here is the code block to configure the pageLoadStrategy
:
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
binary = r'C:\Program Files\Mozilla Firefox\firefox.exe'
caps = DesiredCapabilities().FIREFOX
# caps["pageLoadStrategy"] = "normal" # complete
caps["pageLoadStrategy"] = "eager" # interactive
# caps["pageLoadStrategy"] = "none" # undefined
driver = webdriver.Firefox(capabilities=caps, firefox_binary=binary, executable_path="C:\\Utility\\BrowserDrivers\\geckodriver.exe")
driver.get("https://google.com")
How to make Selenium not wait till full page load, which has a slow script?
When Selenium loads a page/url by default it follows a default configuration with pageLoadStrategy
set to normal
. To make Selenium not to wait for full page load we can configure the pageLoadStrategy
. pageLoadStrategy
supports 3 different values as follows:
normal
(full page load)eager
(interactive)none
Here is the code block to configure the pageLoadStrategy
:
Firefox :
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
caps = DesiredCapabilities().FIREFOX
caps["pageLoadStrategy"] = "normal" # complete
#caps["pageLoadStrategy"] = "eager" # interactive
#caps["pageLoadStrategy"] = "none"
driver = webdriver.Firefox(desired_capabilities=caps, executable_path=r'C:\path\to\geckodriver.exe')
driver.get("http://google.com")Chrome :
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
caps = DesiredCapabilities().CHROME
caps["pageLoadStrategy"] = "normal" # complete
#caps["pageLoadStrategy"] = "eager" # interactive
#caps["pageLoadStrategy"] = "none"
driver = webdriver.Chrome(desired_capabilities=caps, executable_path=r'C:\path\to\chromedriver.exe')
driver.get("http://google.com")
Note :
pageLoadStrategy
valuesnormal
,eager
andnone
is a requirement as per WebDriver W3C Editor's Draft butpageLoadStrategy
value aseager
is still a WIP (Work In Progress) within ChromeDriver implementation. You can find a detailed discussion in “Eager” Page Load Strategy workaround for Chromedriver Selenium in Python
Wait until page is loaded with Selenium WebDriver for Python
The webdriver
will wait for a page to load by default via .get()
method.
As you may be looking for some specific element as @user227215 said, you should use WebDriverWait
to wait for an element located in your page:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
browser = webdriver.Firefox()
browser.get("url")
delay = 3 # seconds
try:
myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'IdOfMyElement')))
print "Page is ready!"
except TimeoutException:
print "Loading took too much time!"
I have used it for checking alerts. You can use any other type methods to find the locator.
EDIT 1:
I should mention that the webdriver
will wait for a page to load by default. It does not wait for loading inside frames or for ajax requests. It means when you use .get('url')
, your browser will wait until the page is completely loaded and then go to the next command in the code. But when you are posting an ajax request, webdriver
does not wait and it's your responsibility to wait an appropriate amount of time for the page or a part of page to load; so there is a module named expected_conditions
.
python selenium: does not wait until page is loaded after a click() command
Fundamentally if the page is being generated by JavaScript then there is no way to tell when it has "finished" loading.
However. I typically try to retrieve a page element, catch the exception and keep on trying until a timeout is reached. I might also check that the element is visible not merely present.
There must be some DOM element whose visibility you can use to test that a page has "finished" loading. I typically have a wait_for_element_visibility
, and wait_for_element_presence
functions in my test cases.
def wait_for_visibility(self, selector, timeout_seconds=10):
retries = timeout_seconds
while retries:
try:
element = self.get_via_css(selector)
if element.is_displayed():
return element
except (exceptions.NoSuchElementException,
exceptions.StaleElementReferenceException):
if retries <= 0:
raise
else:
pass
retries = retries - 1
time.sleep(pause_interval)
raise exceptions.ElementNotVisibleException(
"Element %s not visible despite waiting for %s seconds" % (
selector, timeout_seconds)
)
get_via_css
is one of my own functions but I hope it's fairly clear what it is doing.
Related Topics
Python Equivalent of Ruby's Each_Slice(Count)
Using Beautiful Soup to Convert CSS Attributes to Individual HTML Attributes
Best Way to Set Entry Background Color in Python Gtk3 and Set Back to Default
Does Ruby Support Conditional Regular Expressions
Urllib2 in Python Equivalent for Ruby
What Are the Python Equivalents to Ruby's Bundler/Perl's Carton
Ruby Equivalent of Python's "Dir"
R, Python: Install Packages on Rpy2
Fama MACbeth Regression in Python (Pandas or Statsmodels)
List Comprehension in Haskell, Python and Ruby
Combine a Folder of Text Files into a CSV with Each Content in a Cell
Install Rpy2 on Windows7 64Bit for Python 2.7
Alternative Way to Split a List into Groups of N