Selenium Python - No such element: Unable to locate element locating an input element within an iframe with headless chrome
The <input>
element with the placeholder as Apple ID is within an iframe so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get("https://appstoreconnect.apple.com/login")
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe#aid-auth-widget-iFrame")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#account_name_text_field"))).send_keys("mastaofthepasta")Using XPATH:
driver.get("https://appstoreconnect.apple.com/login")
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[@id='aid-auth-widget-iFrame']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='account_name_text_field']"))).send_keys("mastaofthepasta")
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ECBrowser Snapshot:
Reference
You can find a couple of relevant discussions in:
- Switch to an iframe through Selenium and python
- selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
- selenium in python : NoSuchElementException: Message: no such element: Unable to locate element
Update
When you use google-chrome-headless and print the page_source the output on the console is:
<html><head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>Apple</center>
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
</body></html>
403 Forbidden
The HTTP 403 Forbidden
response status code indicates that the server understands the request but refuses to authorize it.
This status is similar to 401
, but for the 403 Forbidden status code re-authenticating makes no difference. The access is permanently forbidden and tied to the application logic, such as insufficient rights to a resource.
Conclusion
ChromeDriver initiated Chrome Browser gets detected as a bot and further navigation is blocked.
References
You can find a couple of relevant detailed discussions in:
- Website denies get request using Selenium
- How to fix 403 response when using HttpURLConnection in Selenium since the links are opening manually without any issue
Unable to locate elements on webpage with headless chrome
I had the same problem. You could take screenshots to understand whats wrong.
driver.get_screenshot_as_file("screenshot.png")
A few reasons why selenium works when run normally but stops working in headless mode -
1)It might have switched to a mobile template. Can be fixed by changing the window size.
chrome_options.add_argument("--window-size=1920,1080")
2)If its a blank page (screenshot), it might be due to an invalid SSL certificate.(see @Marcel_Wilson post) It should be fixed by -
chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_argument('--allow-running-insecure-content')
3)Its possible that the website blocks 'headless' mode. (Your screenshots might show errors which you cannot recreate in normal mode)
You could try this-
user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.50 Safari/537.36'
options.add_argument(f'user-agent={user_agent}')
However, the above code won't work if the website has a more robust blocking method. You can find more about this here https://intoli.com/blog/making-chrome-headless-undetectable/.
Headless chrome cannot detect elements(selenium)
I took your code and modified a bit and executed at my end and here is the execution result:
Code Block:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.headless = True
options.add_argument('window-size=1400,600')
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://moneyforward.com/users/sign_in')
print(driver.page_source)
driver.save_screenshot('./save_screenshot_method.png') #Capture the screen
driver.quit()Console Output:
<html><head></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">Forbidden</pre></body></html>
Browser Snapshot:
Analysis
It seems ChromeDriver driven google-chrome-headless is getting detected and being Access Denied the message Forbidden is shown.
Solution
As a solution you can adopt some strategies so ChromeDriver driven Chrome Browsing context doesn't gets detected and you can find a couple of detailed discussions in:
- Selenium webdriver: Modifying navigator.webdriver flag to prevent selenium detection
- Selenium “selenium.common.exceptions.NoSuchElementException” when using Chrome
no such element: Unable to locate element using chromedriver and Selenium in production environment
It's report that the element not found error after you supplying the login , so I think the login failed and the page redirected to somewhere. You can use screenshot option to take a screenshot of the page and then see which page the driver load.
driver.save_screenshot("path to save screen.jpeg")
Also you can save the raw html code and inspect the same page.
Webdriver Screenshot
Using Selenium in Python to save a webpage on Firefox
Selenium locate element fails when Chrome in background
You should use either one of the below locators before jumping to XPath.
- ID
- name
- classname
- linkText
- partialLinkText
- tagName
- css selector
- xpath
Also, make sure that you should not be using absolute xpath
, rather it should be relative xpath
.
So, Instead of these
loginuser = WebDriverWait(driver, 50).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="layers"]/div/div/div/div/div/div/div[2]/div[2]/div/div/div[2]/div[2]/div[1]/div/div[5]/label/div/div[2]/div/input')))
loginuser.send_keys("Username")
sleep(5)
loginuser.send_keys(Keys.RETURN)
loginuser = WebDriverWait(driver, 50).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="layers"]/div/div/div/div/div/div/div[2]/div[2]/div/div/div[2]/div[2]/div[1]/div/div[3]/div/label/div/div[2]/div[1]/input')))
loginuser.send_keys("Password")
sleep(5)
loginuser.send_keys(Keys.RETURN)
Use this:
wait = WebDriverWait(driver, 30)
loginuser = wait.until(EC.visibility_of_element_located((By.NAME, "text")))
loginuser.send_keys("Username", Keys.RETURN)
loginPassword = wait.until(EC.visibility_of_element_located((By.NAME, "password")))
loginPassword.send_keys("password here", Keys.RETURN)
Related Topics
Minimum Euclidean Distance Between Points in Two Different Numpy Arrays, Not Within
How to Get a List of Column Names in SQLite
How to Wrap Every Method of a Class
Typeerror: Cannot Create a Consistent Method Resolution Order (Mro)
Matplotlib Figure Facecolor (Background Color)
How to Generate Random Numbers That Are Different
Running a Tkinter Form in a Separate Thread
What Is Sys.Maxint in Python 3
Any Reason Not to Use '+' to Concatenate Two Strings
Get All Object Attributes in Python
Find Usa Phone Numbers in Python Script
How to Open Multiple Webpages in Separate Tabs Within a Browser Using Selenium-Webdriver and Python
Find Substring in String But Only If Whole Words
Generating Sublists Using Multiplication ( * ) Unexpected Behavior
Python:When Is a Variable Passed by Reference and When by Value
How to Access a Function Inside a Function
Trying to Delay a Specific Function for Spawning Enemy After a Certain Amount of Time