Unable to locate elements on webpage with headless chrome
If it's an issue with SSL certs, you can start Chrome without certs using a command line flag (assuming that's how you're starting it). I believe the switch is --allow-running-insecure-content
, and I located that from this list found here.
Headless chrome cannot detect elements(selenium)
I took your code and modified a bit and executed at my end and here is the execution result:
Code Block:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.headless = True
options.add_argument('window-size=1400,600')
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://moneyforward.com/users/sign_in')
print(driver.page_source)
driver.save_screenshot('./save_screenshot_method.png') #Capture the screen
driver.quit()Console Output:
<html><head></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">Forbidden</pre></body></html>
Browser Snapshot:
Analysis
It seems ChromeDriver driven google-chrome-headless is getting detected and being Access Denied the message Forbidden is shown.
Solution
As a solution you can adopt some strategies so ChromeDriver driven Chrome Browsing context doesn't gets detected and you can find a couple of detailed discussions in:
- Selenium webdriver: Modifying navigator.webdriver flag to prevent selenium detection
- Selenium “selenium.common.exceptions.NoSuchElementException” when using Chrome
Selenium Python - "No such element: Unable to locate element" locating an input element within an iframe with headless chrome
The <input>
element with the placeholder as Apple ID is within an iframe so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get("https://appstoreconnect.apple.com/login")
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe#aid-auth-widget-iFrame")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#account_name_text_field"))).send_keys("mastaofthepasta")Using XPATH:
driver.get("https://appstoreconnect.apple.com/login")
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[@id='aid-auth-widget-iFrame']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='account_name_text_field']"))).send_keys("mastaofthepasta")
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ECBrowser Snapshot:
Reference
You can find a couple of relevant discussions in:
- Switch to an iframe through Selenium and python
- selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
- selenium in python : NoSuchElementException: Message: no such element: Unable to locate element
Update
When you use google-chrome-headless and print the page_source the output on the console is:
<html><head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>Apple</center>
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
</body></html>
403 Forbidden
The HTTP 403 Forbidden
response status code indicates that the server understands the request but refuses to authorize it.
This status is similar to 401
, but for the 403 Forbidden status code re-authenticating makes no difference. The access is permanently forbidden and tied to the application logic, such as insufficient rights to a resource.
Conclusion
ChromeDriver initiated Chrome Browser gets detected as a bot and further navigation is blocked.
References
You can find a couple of relevant detailed discussions in:
- Website denies get request using Selenium
- How to fix 403 response when using HttpURLConnection in Selenium since the links are opening manually without any issue
Python Selenium: Can't find element by xpath when browser is headless
I had once some trouble with the versions of such tools. I fixed that by ensuring I had the latest version of Chrome and webdriver installed. Maybe it's a similar issue?!
But apart of that your selector depends on a lot elements. So imagining that the html looks a bit different in headless mode could lead to your issue.
I would try to get a less strict selector like the following: //a[starts-with(@href,'/member/general/login?')]
If that is not working try to dump the html into a file from within the headless mode. Just to see what the headless browser sees and trying build a fancy selector with that.
Related Topics
How to Locate the Index With in a Nested List Python
Splitting Dictionary Items into Smaller Dictionaries Based on Condition
How to Convert List into String With Quotes in Python
Python Not Working in the Command Line of Git Bash
Python: Import Cx_Oracle Importerror: No Module Named Cx_Oracle Error Is Thown
Python:Compare Two CSV Files and Print Out Differences
How to Resolve Modulenotfounderror: No Module Named 'Google.Colab'
How to Get Current Cpu and Ram Usage in Python
Why Does the Session Cookie Work When Serving from a Domain But Not When Using an Ip
Error! C:\File\Example.Db Is Not Utf-8 Encoded Ipython Notebook
Remove Last Few Characters in Pyspark Dataframe Column
Making a Discord Bot Change Playing Status Every 10 Seconds
Changing Presence Discord Status
How to Create an Automatically Updating Gui Using Tkinter
Python Comparing List Values to Keys in List of Dicts
Python Regex - Finding Phone Number
How to Make Tkinter Frames in a Loop and Update Object Values