Selenium webdriver: Modifying navigator.webdriver flag to prevent selenium detection
First the update 1execute_cdp_cmd()
: With the availability of execute_cdp_cmd(cmd, cmd_args)
command now you can easily execute google-chrome-devtools commands using Selenium. Using this feature you can modify the navigator.webdriver
easily to prevent Selenium from getting detected.
Preventing Detection 2
To prevent Selenium driven WebDriver getting detected a niche approach would include either / all of the below mentioned steps:
Adding the argument --disable-blink-features=AutomationControlled
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--disable-blink-features=AutomationControlled')
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.website.com")
You can find a relevant detailed discussion in Selenium can't open a second page
Rotating the user-agent through
execute_cdp_cmd()
command as follows:#Setting up Chrome/83.0.4103.53 as useragent
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})Change the property value of the
navigator
for webdriver to undefineddriver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
Exclude the collection of
enable-automation
switchesoptions.add_experimental_option("excludeSwitches", ["enable-automation"])
Turn-off
useAutomationExtension
options.add_experimental_option('useAutomationExtension', False)
Sample Code 3
Clubbing up all the steps mentioned above and effective code block will be:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
print(driver.execute_script("return navigator.userAgent;"))
driver.get('https://www.httpbin.org/headers')
History
As per the W3C Editor's Draft the current implementation strictly mentions:
The
webdriver-active
flag is set totrue
when the user agent is under remote control which is initially set tofalse
.
Further,
Navigator includes NavigatorAutomationInformation;
It is to be noted that:
The
NavigatorAutomationInformation
interface should not be exposed on WorkerNavigator.
The NavigatorAutomationInformation
interface is defined as:
interface mixin NavigatorAutomationInformation {
readonly attribute boolean webdriver;
};
which returns true
if webdriver-active
flag is set, false otherwise.
Finally, the navigator.webdriver
defines a standard way for co-operating user agents to inform the document that it is controlled by WebDriver, so that alternate code paths can be triggered during automation.
Caution: Altering/tweaking the above mentioned parameters may block the navigation and get the WebDriver instance detected.
Update (6-Nov-2019)
As of the current implementation an ideal way to access a web page without getting detected would be to use the ChromeOptions()
class to add a couple of arguments to:
- Exclude the collection of
enable-automation
switches - Turn-off
useAutomationExtension
through an instance of ChromeOptions
as follows:
Java Example:
System.setProperty("webdriver.chrome.driver", "C:\\Utility\\BrowserDrivers\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.setExperimentalOption("excludeSwitches", Collections.singletonList("enable-automation"));
options.setExperimentalOption("useAutomationExtension", false);
WebDriver driver = new ChromeDriver(options);
driver.get("https://www.google.com/");Python Example
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\path\to\chromedriver.exe')
driver.get("https://www.google.com/")Ruby Example
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument("--disable-blink-features=AutomationControlled")
driver = Selenium::WebDriver.for :chrome, options: options
Legends
1: Applies to Selenium's Python clients only.
2: Applies to Selenium's Python clients only.
3: Applies to Selenium's Python clients only.
How do you disable "navigator.webdriver" in chromedriver?
After looking at the chrome source code for a good while, I finally figured out what flag needs to be used to disable that navigator.webdriver nuisance.
Here is the flag that needs to be used when starting chromedriver. It completely disables "webdriver=true" from even showing up. :)
--disable-blink-features=AutomationControlled
Also here's some more features that can be enabled/disabled.
Features
Enable Flag:
--enable-blink-features=example1,example2,example3
Disable Flag
--disable-blink-features=example1,example2,example3
Hopefully this helps someone who has also come across this problem. Thanks for looking
chromedriver window.navigator.webdriver flag is true with chrome v80+
The presence of the window.navigator.webdriver
property is mandated by the W3C WebDriver Specification. It is both a security feature and a feature to allow better testability of web pages. If you are attempting to automate a page with a WebDriver-powered bot, and you do not own the page being automated, and you are attempting to hide the fact that you’re using a bot from the page’s owners, that is exactly the kind of thing that the property is intended to expose.
It looks like the developers of Chrome and chromedriver have now closed the security loophole they had allowed in previous versions. I suppose it’s possible you could build your own hacked version of the browser and the driver to not set that property when browsing sites with WebDriver, but that would be the only way.
As an editorial comment, attempting to hide browsing by a WebDriver-powered bot is most often a fool’s errand. There are multiple ways of detecting such a bot beyond the navigator.webdriver
property, and it would be nearly impossible to mask them all. Even if it appeared to work for awhile, eventually, the site in question will come up with more detection mechanisms. Moreover, such activity more than likely violates the Terms of Service of nearly any site one would care to attempt this for. It would be far better to find another mechanism for accomplishing your goal.
Related Topics
Illegal Character When Trying to Compile Java Code
How to Make a Multipart/Form-Data Post Request Using Java
How to Call Methods Simultaneously in Java
Enter Array Without Knowing Its Size
Java: How to Check If Object Is Null
How to Test Code Dependent on Environment Variables Using Junit
How to Check Json in Response Body With Mockmvc
Java Coding Converted to Pseudo Code
Java Socket Api: How to Tell If a Connection Has Been Closed
Null Pointer on an Autowired Bean Which Is Not Mocked by Mockito
Spring Rest - Create Zip File and Send It to the Client
Java.Io.Filenotfoundexception: (Access Is Denied)
How to Return a Zip File to the Browser Via the Response Outputstream
Return Json Object from a Spring-Boot Rest Controller
Javax.Xml.Bind.Unmarshalexception: Unexpected Element (Uri:"", Local:"Group")