Webdriver for Firefox: Browser Starts W/ Empty Page, Hangs for 2 Min, Restarts, Then Test Runs. Why

Selenium::WebDriver::Error with Firefox & Chrome

The problems identified, in the updates were not the root cause.

The root cause was that firefox/chrome browser ports were not closed and held open.

After looking at htop, Polkitd was seen to be taking up 16.5gb of ram!

This was caused by a memory leak in Polkitd.

After checking the issues it was confirmed that Polkitd memory leak is a know issue.

The issue has been fixed but only in later distributions of linux debian and not for Wheezy.

After restarting Polkitd, and rerunning the tests in parallel they worked!

This explains why the first time I created a new linux user with a clean profile the parallel test issues were still occuring. - Memory leaks are unpredictable.

It also explains why another computer did not run into the issue.

And why the second time I created a new user the parallel tests worked!
Phew, that took a lot of effort!

Polkitd was uninstalled as it was not needed for any printers or other software that we run.

Overall, if anyone else has the locking issue, it would be helpful to follow some of the process detection that I have done as some of the issues are common to all OS.

GEB :One class Test cases are not opening in separate browser

Actually an error on your web page should not be a problem if the next feature opens the initial URL it needs afresh. Also cookies should not be a problem because you can delete them manually or automatically. Over-using @Stepwise is a source of problems for many users, too. You should avoid it whenever possible. I even see you use it in a specification with only one feature (or maybe you only showed me one and actually there are more). Using many browsers is a huge resource waste and just makes your tests slow. The Book of Geb (manual) is an excellent source of information.

Take a look at implicit lifecycle with regard to clearCookies() andclearWebStorage(). Auto-clearing cookies and/or web storage might also be helpful. The same chapter also explains that if you just use CachingDriverFactory.clearCacheAndQuitDriver() at the end of a feature method (or in cleanup() if you need it in every feature), the next method will get a new browser instance automatically.

Of course you can quit the browser via quit() (quits the browser) and close() (closes current browser window) if you want to start the new browser manually. But implicit driver management is easier to use IMO, so I am just mentioning it for completeness' sake.

---

Now for a very special case: You can even use something like resetBrowser(); CachingDriverFactory.clearCacheAndQuitDriver() in the middle of a single feature method like this:

package de.scrum_master.testing

import geb.driver.CachingDriverFactory
import geb.spock.GebReportingSpec

class RestartBrowserIT extends GebReportingSpec {
  def "Search web site Scrum-Master.de"() {
    when:_ "download page is opened"
    go "https://scrum-master.de"
    report "welcome page"

    then:_ "expected text is found on page"
    $("h2").text().startsWith("Herzlich Willkommen bei Scrum-Master.de")

    when:_ "browser is reset"
    resetBrowser()
    CachingDriverFactory.clearCacheAndQuitDriver()

    and:_ "download page is opened again in new browser"
    go "https://scrum-master.de/Downloads"
    report "download page"

    then:_ "expected text is found on page"
    $("h2").text().startsWith("Scrum on a Page")
  }
}

Then on the console you see something like this:

download page is opened
07:49:49.105 [main] INFO  i.g.bonigarcia.wdm.WebDriverManager - Using chromedriver 83.0.4103.39 (since Google Chrome 83 is installed in your machine)
07:49:49.146 [main] INFO  i.g.bonigarcia.wdm.WebDriverManager - Exporting webdriver.chrome.driver as C:\Users\alexa\.m2\repository\webdriver\chromedriver\win32\83.0.4103.39\chromedriver.exe
Starting ChromeDriver 83.0.4103.39 (ccbf011cb2d2b19b506d844400483861342c20cd-refs/branch-heads/4103@{#416}) on port 3302
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
Jul 03, 2020 7:49:53 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFORMATION: Detected dialect: W3C
expected text is found on page
browser is reset
download page is opened again in new browser
07:49:57.387 [main] INFO  i.g.bonigarcia.wdm.WebDriverManager - Using chromedriver 83.0.4103.39 (since Google Chrome 83 is installed in your machine)
07:49:57.413 [main] INFO  i.g.bonigarcia.wdm.WebDriverManager - Exporting webdriver.chrome.driver as C:\Users\alexa\.m2\repository\webdriver\chromedriver\win32\83.0.4103.39\chromedriver.exe
Starting ChromeDriver 83.0.4103.39 (ccbf011cb2d2b19b506d844400483861342c20cd-refs/branch-heads/4103@{#416}) on port 27426
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
Jul 03, 2020 7:50:00 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFORMATION: Detected dialect: W3C
expected text is found on page

---

P.S.: Just in case you are wondering why I write when:_ "label" instead of just when: "label", I use this SpockConfig.groovy file in order to help me print labels in my test, as you can also see in the console log above:

import spock.lang.Specification

/**
 * Use like this in order to print Spock/Geb labels:
 *   given:_ "foo"
 *   when:_ "bar"
 *   then:_ "zot"
 */
class LabelPrinter {
  def _(def message) {
    println message
    true
  }
}

Specification.mixin LabelPrinter

The Spock configuration file should be in src/main/resources/SpockConfig.groovy if you use a standard Maven directory layout.

Selenium with Firefox: FirefoxProfile stops working when used a second time

Probably when you quit() then it removes profile - and you would have to create again profile.

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

# --- functions ---

def sele_scrape(url):

    profile = webdriver.FirefoxProfile()
    #profile.add_extension(extension='extension/1/path')
    #profile.add_extension(extension='extension/2/path')
    
    options = Options()
    #options.headless = True
    
    driver = webdriver.Firefox(firefox_profile=profile, options=options)#,executable_path='path/to/geckodriver')
    
    driver.get(url)
    source = driver.page_source
    driver.quit()

# --- main ---

sele_scrape('https://stackoverflow.com')
sele_scrape('https://stackoverflow.com')

---

Code works correctly if I use close() instead of quit()

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

# --- functions ---

def sele_scrape(url, profile, options):
   
    driver = webdriver.Firefox(firefox_profile=profile, options=options)#,executable_path='path/to/geckodriver')
    
    driver.get(url)
    source = driver.page_source
    driver.close()

# --- main ---

profile = webdriver.FirefoxProfile()
#profile.add_extension(extension='extension/1/path')
#profile.add_extension(extension='extension/2/path')

options = Options()
#options.headless = True

sele_scrape('https://stackoverflow.com', profile, options)
sele_scrape('https://stackoverflow.com', profile, options)

---

Frankly, if profile should be reused then I would create driver only once and I would skip close().

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

# --- functions ---

def sele_scrape(url, driver):
   
    driver.get(url)
    source = driver.page_source

# --- main ---

profile = webdriver.FirefoxProfile()
#profile.add_extension(extension='extension/1/path')
#profile.add_extension(extension='extension/2/path')

options = Options()
#options.headless = True

driver = webdriver.Firefox(firefox_profile=profile, options=options)#,executable_path='path/to/geckodriver')

sele_scrape('https://stackoverflow.com', driver)
sele_scrape('https://stackoverflow.com', driver)

driver.close()

---

Related Topics