Working with a Canvas in Selenium for Java

Learn how to work with a Canvas using Selenium for Java

Canvases are really difficult to automate. 

Let’s have a look at this one; it asks if we’re hungry.  Well, of course we are. So, we click on this Let’s Eat button. And voila! We have a juicy burger. 

The problem is, this canvas element exists in the DOM, but there’s not much about it. There’s nothing here representing the canvas’s button, so how will we click it? 

Also, there’s nothing here describing the hamburger image. So how will we verify that it’s displayed? 

Tricky indeed, but not impossible. Let’s cook up a solution to this. 

How to Click an Element on a Canvas with Selenium

Let’s start by finding that canvas element and we can do so by saying driver.findElement and we’ll do it By.id. And the ID was “burger_canvas”. Perfect. 

Now we want to click the button on the canvas. We can do so by using the Actions class in Selenium. With the Actions, we can just pass it the driver. And then we can add any actions that we want to take.

We want to moveToElement and we’ll move to the canvas. And then we want to say, click on the canvas. Finally, we want to say perform. So, all these actions are chained up and then they’re performed. 

Now this will click the canvas, but it clicks the center of the canvas. This click method always clicks the center of the element by default. And this isn’t what we want. 

Now, this moveToElement is actually an overloaded method that allows us to pass offset values for the X and Y coordinates. And this will move the cursor that many pixels from the center point. 

So, let’s go back and look at the canvas again so that we can see how much offset we need.

Because we don’t have access to the button, we’ll need to just click the area of the canvas where the button is. To do this we need to figure out the X Y coordinates. But this is also tricky because we might want to run our tests on various viewport sizes so we can’t just hard code in X Y values. 

Instead, we need to find a formula that will always work no matter what size the browser window is. 

From the center of the canvas, we have this entire work area as the offset. Instead of trying to count pixels, which would only work if you ran the test in the same viewport size every time, we’re going to visualize breaking this area into sections. 

For example, if we divided this X axis into 3 sections and the Y axis into 3 sections, we could then figure out which block we should target to click the button. So, it looks like if we go 2 over on the X axis, and also 2 down on the Y axis, we’re right in the ballpark. 

Let’s code this up.

Before we do any moving, let’s go ahead and calculate the position of the canvas button. 

We’ll call this “canvas_dimensions” and we can get this by saying canvas.getSize.  Let’s go ahead and define the center of the X axis. So, we’ll say “canvas_center_x” and we can get this by doing the canvas_dimensions.getWidth() , which would be the X axis. And we can divide this by 2. 

All right. And we’ll do the same thing for the center of the Y axis. We’ll get the dimensions and getHeight this time. And also divide that by 2. 

Now we can calculate the grid offset to target the area where the button is.

Let’s go ahead and define the coordinate for the button. So, this “button_x” will be the X coordinate and we have the center so we can get the center for X. And remember then for the offset, we’ve divided that offset by 3. Right? Let’s go ahead and put that in parentheses because then we moved 2 spaces over. So, let’s multiply it by 2. 

This will divide that section by 3, but then move over two thirds of that. Okay, great. 

Let’s do the same for the “button_y” and this time we’ll say the Y one also. We divided that into 3 sections, and we moved over 2 sections in that case. All right. 

Now, one thing to note is that if the area you needed to click was to the left of the center point, you need to divide by a negative number for the X axis. And same for if the area is above the center point, you divide by a negative number for the Y axis.

Okay, now we have our dimensions; we can use the overloaded moveToElement method to provide our button coordinates. 

So, we say, go to the center of the canvas, and then we want you to move to the “button_x” position and “button_y” position — .moveToElement(canvas, button_x, button_y). This gives it coordinates to move to. 

Okay, that should do the trick. Let’s run this in debug mode so that we can see if it worked. But first let’s add one more statement after here, just so that we can put a breakpoint. 

So, we’ll say, “is the burger there?”. All right, I’m going to put my break point here and let’s go ahead and debug this. 

Look at that, there’s the burger. So that means it did indeed find the button and was able to click it. Perfect. 

But before we celebrate, let’s make sure that this works if the window size changes.

So, I’m just going to let that continue running and this time I want to go ahead and change the dimensions of the page.

We’re going to say driver.manage().window.().setSize(). And here we can set the dimension to be, let’s say 375 by 812 — setSize(new Dimension(375,812) — which is the size of an iPhone. 

All right. And we have our breakpoint. Let’s run this again. 

Yes. This still works. So, we’re good to go. 

Visual Testing of Canvas Using Applitools Eyes SDK

But how do we verify that the burger is actually shown? Again, we don’t have access to anything within the canvas in our DOM. So, we need to use a tool that can verify the surface. 

Applitools is a visual testing tool that’s perfect for this job. We can add some visual testing with just a few lines of code here. 

I’m going to say let’s go ahead and verify that the canvas now displays “burger”. And we can do this with Applitools Eyes, which is the name of the API. So, I’m just instantiating eyes 

And I’m going to say open your eyes — eyes.open. This takes our WebDriver, it takes the name of the application, and it takes the name of the test. So, we’ll see “burger on canvas”. Cute. Okay.

The next part is the magic. We can say check the element — eyes.checkElement(canvas) — and since we know that it’s the canvas element, and we already have access to that, we can just go ahead and pass that in. 

And that’s it. We can say close your eyes now — eyes.close. 

This will do the visual. 

So let me go ahead and remove this. That sets the window. It can be any size. 

Okay, I’m going to run this test now so that we can verify the burger as well. Look at that. Everything is working so well. Okay, our test has passed.

Let’s go and look at this in the Applitools dashboard. We see that it did, in fact, take a picture of our hamburger. 

Every time we run this test, now it’s going to take a new screenshot and compare it to this one to make sure that the burger is always displayed. 

Great job. That’s how we interact with and validate canvas elements.

Resources

• Git Repo – CanvasTests.java
• Applitools Kitchen – Testing a Canvas
• Applietools Eyes SDK for Selenium – npm eyes-selenium