How to Click on Specific Element in Canvas by Its Coordinates (Using Webdriver)

How to click on specific element in canvas by its coordinates (using WebDriver)?

Movement

move_to(element) moves to the center of the element specified and move_by is a relative move. So by the end of these two operations, you have moved to the coordinates (x of element center + x, y of element center + y).

You should use move_to(element, x, y). This will move to the x, y coordinates relative to the origin of the element.

Relevant documentation.

Firefox

Are you using a version of Selenium and Firefox for which Selenium supports native events? The combination of Selenium 2.37 with Firefox 24 does. I've had test suite fails just because native events were not available.

How to click on specific element in canvas by its coordinates (using WebDriver)?

Movement

move_to(element) moves to the center of the element specified and move_by is a relative move. So by the end of these two operations, you have moved to the coordinates (x of element center + x, y of element center + y).

You should use move_to(element, x, y). This will move to the x, y coordinates relative to the origin of the element.

Relevant documentation.

Firefox

Are you using a version of Selenium and Firefox for which Selenium supports native events? The combination of Selenium 2.37 with Firefox 24 does. I've had test suite fails just because native events were not available.

How to click the center of the canvas in Java Selenium Driver?

The relevant DOM Tree would have helped us to construct a cannonical answer. However, while dealing with <canvas> elements you need to consider a few things:

• The upper-left corner of the <canvas> has the coordinates (0,0), while
• Using the W3C Action class commands, offsets are from the center of element.

Usually, the <canvas> element will be having the following attributes:

• width
• height
• style containing the attribues width, height, etc

As an example:

<canvas id="canvas" width="227" height="294" style="position: absolute; display: block; background-color: rgb(255, 255, 255); width: 227.53px; height: 294px;"></canvas>

Now, to click at the centre of the <canvas> element within the webpage, you really, don't need to calculate:

• we.getSize().width/2;
• we.getSize().height/2;

As:

The current implementation of Action class commands, offsets are from the center of element.

So, you can use the following solution:

WebElement canvas = new WebDriverWait(driver, 10).until(ExpectedConditions.elementToBeClickable(By.id("canvas")));
//clicking on the centre
new Actions(driver).moveToElement(canvas).click().build().perform();

Incase, you want to click at an offset, you need to use moveByOffset() as follows:

new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset((683/5)*3,(341/5)*3).click().build().perform();

---

Reference

You can find a relevant detailed discussion in:

• How to find the coordinates of the buttons on a canvas, and click on them after using Java and Selenium?

How to find the coordinates of the buttons on a canvas, and click on them after using Java and Selenium?

The <canvas> element is within an <iframe>. So to invoke click() on the elements within the <canvas> you have to:

• Induce WebDriverWait for the desired frame to be available and switch to it.
• Induce WebDriverWait for the desired element to be clickable.

• You can use the following solution:

  • Code Block:

    driver.get("https://www.online-calculator.com/full-screen-calculator/")
    new WebDriverWait(driver, 10).until(ExpectedConditions.frameToBeAvailableAndSwitchToIt(By.id("fullframe")));
    WebElement canvas = new WebDriverWait(driver, 10).until(ExpectedConditions.elementToBeClickable(By.id("canvas")));
    

---

HTML5 Canvas

The element is only a container for graphics and is a rectangular area on an HTML page. By default, a canvas has no border and no content. However, an id attribute (to be referred to in a script), a width and height attribute are specified to define the size of the canvas. To add a border, the style attribute is used. An example:

<canvas id="myCanvas" width="200" height="100" style="border:1px solid #000000;"></canvas>

The HTML canvas is a two-dimensional grid. The upper-left corner of the canvas has the coordinates (0,0).

In the article Automated Testing of HTML5 Canvas Applications with Selenium WebDriver @Aaron Mulder mentions, interacting with the elements within <canvas> is possible using event support of the Actions Class API:

• moveToElement(WebElement target, int xOffset, int yOffset): Moves the mouse to an offset from the top-left corner of the element.

  new Actions(driver).moveToElement(canvas, xWithinCanvas, yWithinCanvas).click().perform();
  

But is not 100% reliable as, every mouse down, mouse up, or mouse click happens at the center of the element. So the code above produces a mouse move event to the provided coordinates (x,y), then a mouse move event to the center of the <canvas>, then a mouse down, mouse up, and click at the center of the <canvas>. That should have been fine for a <button> but is not worth for a <canvas>, where you want to be able to click at a specific location.

The workaround, is to dispatch synthesized mouse events using JavaScript as follows:

// pageX and pageY are offsets which you need to know through mouse coordinates.
((JavascriptExecutor)driver).executeScript("var evt = $.Event('click', { pageX: " + x +
   ", pageY: " + (y + 55) + " } );" +
   "$('#myCanvas').trigger(evt);");

However, to click on the elements within the <canvas> you can be at ease using firefox as the mouse move event works well in Firefox and you can avoid using the mouse coordinates as the event processing as follows:

new Actions(driver).moveToElement(
   canvas, xWithinCanvas, yWithinCanvas).perform();
((JavascriptExecutor)driver).executeScript("$('#canvas').click();");

---

This usecase

To automate a substruction operation e.g. 3-1= using Selenium you can use the following solution:

• Code Block:

  driver.get("https://www.online-calculator.com/full-screen-calculator/");
  new WebDriverWait(driver, 10).until(ExpectedConditions.frameToBeAvailableAndSwitchToIt(By.id("fullframe")));
  WebElement canvas = new WebDriverWait(driver, 10).until(ExpectedConditions.elementToBeClickable(By.id("canvas")));
  //clicking on 3
  new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset(0,(255/6)*3).click().build().perform();
  //clicking on the substract sign (-)
  new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset((174/5)*2,(255/6)*3).click().build().perform();
  //clicking on 1
  new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset(-(174/5)*4,(255/6)*3).click().build().perform();
  //clicking on equals to sign (=)
  new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset((174/5)*4,(255/6)*4).click().build().perform();
  

• Execution Video:

---

Reference

You can find a couple of relevant detailed discussion in:

• How to perform Mouse Wheel scrolling over HTML5 Canvas in Selenium?

---

Related Topics