MaxRetryError: HTTPConnectionPool: Max retries exceeded (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused')))
This error message...
MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=51379): Max retries exceeded with url: /session/2e64d2a1-3c7f-4221-96fe-9d0b1c102195/window (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused')))
...implies that the call to self.driver.close()
method failed raising MaxRetryError.
A couple of things:
First and foremost as per the discussion max-retries-exceeded exceptions are confusing the traceback is somewhat misleading. Requests wraps the exception for the users convenience. The original exception is part of the message displayed.
Requests never retries (it sets the
retries=0
for urllib3'sHTTPConnectionPool
), so the error would have been much more canonical without the MaxRetryError and HTTPConnectionPool keywords. So an ideal Traceback would have been:ConnectionError(<class 'socket.error'>: [Errno 1111] Connection refused)
But again @sigmavirus24 in his comment mentioned ...wrapping these exceptions make for a great API but a poor debugging experience...
Moving forward the plan was to traverse as far downwards as possible to the lowest level exception and use that instead.
Finally this issue was fixed by rewording some exceptions which has nothing to do with the actual connection refused error.
Solution
Even before self.driver.close()
within tearDown(self)
is invoked, the try{} block within test_select_in_sina(self)
includes finally{} where you have invoked driver.quit()
which is used to call the /shutdown endpoint and subsequently the web driver & the client instances are destroyed completely closing all the pages/tabs/windows. Hence no more connection exists.
You can find a couple of relevant detailed discussion in:
- PhantomJS web driver stays in memory
- Selenium : How to stop geckodriver process impacting PC memory, without calling
driver.quit()?
In such a situation when you invoke self.driver.close()
the python client is unable to locate any active connection to initiate a clousure. Hence you see the error.
So a simple solution would be to remove the line driver.quit()
i.e. remove the finally
block.
tl; dr
As per the Release Notes of Selenium 3.14.1:
* Fix ability to set timeout for urllib3 (#6286)
The Merge is: repair urllib3 can't set timeout!
Conclusion
Once you upgrade to Selenium 3.14.1 you will be able to set the timeout and see canonical Tracebacks and would be able to take required action.
References
A couple of relevent references:
- Adding max_retries as an argument
- Removed the bundled charade and urllib3.
- Third party libraries committed verbatim
How to address urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=58408): Max retries exceeded with url
This error message...
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=58408): Max retries exceeded with url: /session/4b3cb270d1b5b867257dcb1cee49b368/url (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001D5B378FA60>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))
...implies that the failed to establish a new connection raising MaxRetryError as no connection could be made.
A couple of things:
First and foremost as per the discussion max-retries-exceeded exceptions are confusing the traceback is somewhat misleading. Requests wraps the exception for the users convenience. The original exception is part of the message displayed.
Requests never retries (it sets the
retries=0
for urllib3'sHTTPConnectionPool
), so the error would have been much more canonical without the MaxRetryError and HTTPConnectionPool keywords. So an ideal Traceback would have been:ConnectionError(<class 'socket.error'>: [Errno 1111] Connection refused)
Root Cause and Solution
Once you have initiated the webdriver and web client session, next within def search(st)
you are invoking get()
o access an url and in the subsequent lines you are also invoking browser.quit()
which is used to call the /shutdown
endpoint and subsequently the webdriver & the web-client instances are destroyed completely closing all the pages/tabs/windows. Hence no more connection exists.
You can find a couple of relevant detailed discussion in:
- PhantomJS web driver stays in memory
- Selenium : How to stop geckodriver process impacting PC memory, without calling
driver.quit()?
In such a situation in the next iteration (due to the for
loop) when browser.get()
is invoked there are no active connections. hence you see the error.
So a simple solution would be to remove the line browser.quit()
and invoke browser.get(url)
within the same browsing context.
Conclusion
Once you upgrade to Selenium 3.14.1 you will be able to set the timeout and see canonical Tracebacks and would be able to take required action.
References
You can find a relevant detailed discussion in:
- MaxRetryError: HTTPConnectionPool: Max retries exceeded (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused')))
tl; dr
A couple of relevent discussions:
- Adding max_retries as an argument
- Removed the bundled charade and urllib3.
- Third party libraries committed verbatim
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=59587): Max retries exceeded with url using Selenium GeckoDriver Firefox
This error message...
MaxRetryError(_pool, url, error or ResponseError(cause))urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=59587): Max retries exceeded with url: /session/b38be2fe-6d92-464f-a096-c43183aef6a8/element (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000173145EF520>: Failed to establish a new connection: [WinError 10061] No connections could be made because the target machine actively refused them'))
...implies that the GeckoDriver was unable to initiate/spawn a new Browsing Context i.e. firefox session.
Root cause
The root cause of this error can be either of the following:
- This error may surface if have closed the Browsing Context manually with brute force when the driver have already initiated a lookout for element/elements.
- There is a possibility that the application you are trying to access is throttling the requests from your system/machine/ip-address/network.
- There is also a possibility that the application have identified the Selenium driven GeckoDriver initiated firefox Browsing Context as a bot and is denying any access.
Solution
Ensure that:
To evade the detection as a bot, pass the argument
--disable-blink-features=AutomationControlled
as follows:from selenium.webdriver.firefox.options import Options
options = Options()
options.add_argument('--disable-blink-features=AutomationControlled')Always invoke
driver.quit()
withintearDown(){}
method to close & destroy the WebDriver and Web Client instances gracefully.Induce WebDriverWait to synchronize the fast moving WebDriver along with the Browsing Context.
Related Topics
Pandas Groupby Multiple Fields Then Diff
How to Uninstall Anaconda Completely from MACos
Why Are Empty Strings Returned in Split() Results
How to Put Individual Tags for a Matplotlib Scatter Plot
Given a Url to a Text File, What Is the Simplest Way to Read the Contents of the Text File
How to Interpret Conda Package Conflicts
Class Method Decorator with Self Arguments
Thread Starts Running Before Calling Thread.Start
How to Find the Number of Arguments of a Python Function
Differencebetween Slice Assignment That Slices the Whole List and Direct Assignment
Python Function Attributes - Uses and Abuses
List to Dictionary Conversion with Multiple Values Per Key
How Could I Use Requests in Asyncio
Isprime Function for Python Language