How to Use Ec.Presence_Of_Element_Located((By.Id, "Mydynamicelement")) Except to Specify Class Not Id

How do you use EC.presence_of_element_located((By.ID, myDynamicElement )) except to specify class not ID

The relevant HTML would have helped us to construct a more canonical answer. However to start with your first line of code:

element = WebDriverWait(driver,100).until(EC.presence_of_element_located(
  (By.ID, "tabla_evolucion")))

is pretty much legitimate where as the second line of code:

element = WebDriverWait(driver,100).until(EC.presence_of_element_located(
  (By.class, "ng-binding ng-scope")))

Will raise an error as:

Message: invalid selector: Compound class names not permitted

as you can't pass multiple classes through By.class.

You can find a detailed discussion in Invalid selector: Compound class names not permitted using find_element_by_class_name with Webdriver and Python

---

Solution

You need to take care of a couple of things as follows:

• Without any visibility to your usecase, functionally inducing WebDriverWait in association with EC as presence_of_element_located() merely confirms the presence of the element within the DOM Tree. Presumably moving ahead either you need to get the attributes e.g. value, innerText, etc or you would interact with the element. So instead of presence_of_element_located() you need to use either visibility_of_element_located() or element_to_be_clickable()

You can find a detailed discussion in WebDriverWait not working as expected

• For an optimum result you can club up the ID and CLASS attributes and you can use either of the following Locator Strategies:

• Using CSS_SELECTOR:

  element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located(
    (By.CSS_SELECTOR, ".ng-binding.ng-scope#tabla_evolucion")))

• Using XPATH:

  element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located(
    (By.XPATH, "//*[@class='ng-binding ng-scope' and @id='tabla_evolucion']")))

How to locate the email address element with a Facebook page using Selenium

The Email Address field within the webpage contains dynamic elements.

To locate the Email Address field instead of presence_of_element_located() you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following locator strategies:

• Using XPATH and the text @:

  print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[contains(., '@')]"))).text)
  

• Using XPATH and the texts @ & com:

  print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[contains(., '@') and contains(., 'com')]"))).text)
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

How to scrape the Current Holdings table

To scrape the data from the current holdings_table as the <table> is present with the HTML DOM but not visible within the webpage, you need to induce WebDriverWait for the presence_of_element_located() for the <table> element, extract the outerHTML, read the outerHTML using read_html() and you can use the following locator strategy:

• Code Block:

  driver.execute("get", {'url': 'https://whalewisdom.com/filer/berkshire-hathaway-inc#tabholdings_tab_link'})
  # data = driver.find_element(By.CSS_SELECTOR, "table#current_holdings_table").get_attribute("outerHTML")
  data = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.CSS_SELECTOR, "table#current_holdings_table"))).get_attribute("outerHTML")
  df  = pd.read_html(data)
  print(df)
  

• Console Output:

  [    Stock  Unnamed: 1                  Sector  ...  Source Source Date  Date Reported
  0    AAPL         NaN  INFORMATION TECHNOLOGY  ...     13F  2022-03-31     2022-05-16
  1     BAC         NaN                 FINANCE  ...     13F  2022-03-31     2022-05-16
  2     AXP         NaN                 FINANCE  ...     13F  2022-03-31     2022-05-16
  3     CVX         NaN                  ENERGY  ...     13F  2022-03-31     2022-05-16
  4      KO         NaN        CONSUMER STAPLES  ...     13F  2022-03-31     2022-05-16
  5     OXY         NaN                  ENERGY  ...       4  2022-05-02     2022-05-04
  6     KHC         NaN        CONSUMER STAPLES  ...     13F  2022-03-31     2022-05-16
  7     MCO         NaN                 FINANCE  ...     13F  2022-03-31     2022-05-16
  8     USB         NaN                 FINANCE  ...     13F  2022-03-31     2022-05-16
  9    ATVI         NaN  INFORMATION TECHNOLOGY  ...     13F  2022-03-31     2022-05-16
  10    HPQ         NaN  INFORMATION TECHNOLOGY  ...     13G  2022-04-30     2022-04-30
  11     BK         NaN                 FINANCE  ...     13F  2022-03-31     2022-05-16
  12     KR         NaN        CONSUMER STAPLES  ...     13F  2022-03-31     2022-05-16
  13    DVA         NaN             HEALTH CARE  ...     13D  2022-08-01     2022-08-01
  14      C         NaN                 FINANCE  ...     13F  2022-03-31     2022-05-16
  15   VRSN         NaN          COMMUNICATIONS  ...     13F  2022-03-31     2022-05-16
  16     GM         NaN  CONSUMER DISCRETIONARY  ...     13F  2022-03-31     2022-05-16
  17   PARA         NaN          COMMUNICATIONS  ...     13F  2022-03-31     2022-05-16
  18   CHTR         NaN          COMMUNICATIONS  ...     13F  2022-03-31     2022-05-16
  19  LSXMK         NaN          COMMUNICATIONS  ...     13F  2022-03-31     2022-05-16
  20      V         NaN                 FINANCE  ...     13F  2022-03-31     2022-05-16
  21   AMZN         NaN  CONSUMER DISCRETIONARY  ...     13F  2022-03-31     2022-05-16
  22    AON         NaN                 FINANCE  ...     13F  2022-03-31     2022-05-16
  23     MA         NaN                 FINANCE  ...     13F  2022-03-31     2022-05-16
  24   SNOW         NaN  INFORMATION TECHNOLOGY  ...     13F  2022-03-31     2022-05-16
  

__init__() takes 2 positional arguments but 3 were given trying to wait for an element using presence_of_element_located()

You need to take care of a couple of things:

• presence_of_element_located() should be called within a tuple as it is not a function but a class, where the initializer expects just 1 argument beyond the implicit self
• presence_of_element_located() doesn't ensures that the element is interactable. Instead you need to use element_to_be_clickable()

---

Solution

You need to induce WebDriverWait for the element_to_be_clickable() and you can use the following Locator Strategy:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='menu-item-9145']/a"))).click()

Note: You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

---

Reference

You can find a couple of relevant detailed discussions in:

• init() takes 2 positional arguments but 3 were given using WebDriverWait and expected_conditions as element_to_be_clickable with Selenium Python

How to find the product name and price using the python scraper?

It appears that website will automatically redirect visitor according to the country determined from visitor's IP, and it will also change the classes for item name and price accordingly. Here is a more robust solution, which will account for such changes, and will also wait for the element to load in page:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

chrome_options = Options()
chrome_options.add_argument("--no-sandbox")

webdriver_service = Service("chromedriver/chromedriver") ## path to where you saved chromedriver binary
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)

url='https://www.neimanmarcus.com/en-jp/p/givenchy-g-chain-ring-prod250190244?childItemId=NMY5X1R_&navpath=cat000000_cat4870731_cat50910737_cat2650734&page=0&position=11'

browser.get(url) 

try:
    WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.ID, "closeButton"))).click()
except Exception as e:
    print('no pop-up, moving on')

title = WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[@data-test='pdp-title']")))
price = WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@data-test='pdp-pricing']")))
print(title.text.strip(), price.text.strip())

Result:

G-Chain Ring JPY 46709

Which expected_conditions to use?

As you are not expanding the items and sublist being invisible you are invoking click() on multiple buttons, WebDriverWait for visibility_of_element_located() and element_to_be_clickable() will always fail.

---

Solution

In such cases, your best bet would be to induce WebDriverWait for the presence_of_element_located() as follows:

element = WebDriverWait(driver, 20).until(expected_conditions.presence_of_element_located((By.XPATH, "element_xpath']")))
driver.execute_script(""arguments[0].click();", element)

In a single line:

driver.execute_script(""arguments[0].click();", WebDriverWait(driver, 20).until(expected_conditions.presence_of_element_located((By.XPATH, "element_xpath']"))))

Another JavaScript based alternative would be:

driver.execute_script("document.getElementsByClassName('theClassName')[0].click()")

---

Related Topics