Make Headless Browser Stop Loading Page

Some websites dont fully load/render in selenium headless mode

from selenium import webdriver
from time import sleep

options = webdriver.ChromeOptions()
options.add_argument("--window-size=1920,1080")
options.add_argument("--headless")
options.add_argument("--disable-gpu")
options.add_argument(
"user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36")
browser = webdriver.Chrome(options=options)

some websites uses user-agent to detect whether the browser is in headless mode or not as headless browser uses a different user-agent than normal browser. So explicitly set user agent.

Headless browser detection

stop page loading with Selenium Webdriver

https://www.selenium.dev/documentation/en/webdriver/page_loading_strategy/#:~:text=Defines%20the%20current%20session's%20page,loading%20takes%20lot%20of%20time.

    chromeOptions.setPageLoadStrategy(PageLoadStrategy.EAGER);
WebDriver driver = new ChromeDriver(chromeOptions);

use page loading strategy Eager to wait only till initial html is loaded , you can also use none , but make sure you have explicit/implicit waits for elements if timing issue comes up

In python its working wierdly only the diseried capability is working

from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

caps = DesiredCapabilities().CHROME
# caps["pageLoadStrategy"] = "normal" # Waits for full page load
caps["pageLoadStrategy"] = "none"

options = Options()

driver = webdriver.Chrome(desired_capabilities=caps, options=options)

url = 'https://www.gm-trucks.com/'

driver.get(url)
print(driver.title)
print("hi")
input()

Or :

options = Options()

options.set_capability("pageLoadStrategy", "none")
driver = webdriver.Chrome(options=options)

Update

The documentation is updated as per selenium 4.0.0-alpha-7

so use above solution or update to selenium v4 for future protection

  pip install selenium==4.0.0.a7

Bug

https://github.com/SeleniumHQ/seleniumhq.github.io/issues/627

Headless Chrome with Selenium not loading the web-page elements correctly

On some more research, I discovered adding

chrome_options.add_argument('--lang=en_US') 

this code snippet to my program does the trick.

Headless chrome does not support all incoming languages and thus some pages do not respond well to that. Supporting the language that page outputs in, loads the page correctly.



Related Topics



Leave a reply



Submit