Walkthrough : Running a visual UI test and reviewing test results

This article will give you hands-on experience of using Eyes by walking you through an example of writing and running a visual UI test using Selenium and the Eyes SDK and reviewing test results using the Eyes Test Manager. The main steps of the workflow that the article covers are:

The Hello World web page

The test we're going to run executes a visual UI test on an example web page created by Applitools If you surf to https://applitools.com/helloworld2 you will see the following screen:

Enter your name in the text field, and then click the button that says "Click me".

The screen will now look like this:

Let's automate it!

The test code we're going to write will do exactly what you just did, but will also take screenshots after each keyboard or mouse event so that we can visually validate them to ensure that the result is consistent every time we run the test.

Here is an outline of what the test code does:

  1. Open the web page
  2. Take an initial screenshot
  3. Enter some text
  4. Take a screenshot after the text is entered
  5. Click the button
  6. Take a screenshot after the click

Our program will run this test twice, once with a viewport size of 1024 x 768 and a second time with a viewport size of 500 x 900.

This example is an extended version of the example used in the overview article Writing tests with the Eyes SDK . Since the code is very similar, we won't explain the code here.

Setting up the test execution environment

To run these test you will need:

  • An Aplitools Eyes API key
  • A Chrome Driver
  • The Eyes SDK
  • The Example code

Obtain your API key

In order to execute visual UI tests, you must obtain an API key that identifies your account and provides you with test execution permissions. If you don’t have an Applitools account yet or if you don’t know how to obtain your API key, follow the instructions in the article How to obtain your API key .

Getting the Chrome Driver

In order to run the test you will need a chrome browser and the chrome webdriver. To obtain these, follow the instruction here https://sites.google.com/a/chromium.org/chromedriver/getting-started . Make sure you install the webdriver in a folder that appears in your PATH environment.

Getting the SDK

You will need the Eyes SDK for the programing language of your choice:

In Java, if you have a test environment that uses Maven, You can add the following dependency in your Maven .pom file to load the Eyes Java Selenium Library.


<dependencies>
   <dependency>
      <groupId>com.applitools</groupId>
      <artifactId>eyes-selenium-java3</artifactId>
      <version>RELEASE</version>
  </dependency>
</dependencies>
				

Alternately:

  1. Download the latest version of the Eyes Java Selenium SDK from here and extract it to a folder of your choice.
  2. Add the extracted files to your path.
npm init
npm install eyes.selenium --save-dev
npm install selenium-webdriver
npm install chromedriver
		
In Visual Studio, in the NuGet console Browse for Eyes.Selenium and install it
					
		
Run the following command in a shell
  composer require applitools/eyes.sdk.php:dev-master

If you don't have any installed packages create a composer.json file with the following content
{
"require": {
"applitools/eyes.sdk.php": "dev-master"
}
}
And then run:
composer install
			
		
Run the following command in a shell
	pip install eyes-selenium
			
		
Run the following command in a shell
	gem install eyes_selenium				
			

Getting the code

Copy the code below, in your preferred language into a file of the appropriate type.


                            

Customizing the code

If you have a dedicated site then make the change described below:


                            

Running a test for the first time

Now let's run the test for the first time. After you start running the test in your IDE or console, a browser will open, and you will see it changing its size twice - once for each viewport size. After the test completes you should see on your run time console output similar to this:


New Baseline Created: 3 steps
results at https://eyes.applitools.com/app/batches/00000251885689413314/00000251885689412658?accountId=mvdQ30vIlkiU8F-usoJS4w~~
New Baseline Created: 3 steps
results at https://eyes.applitools.com/app/batches/00000251885689375737/00000251885689375389?accountId=mvdQ30vIlkiU8F-usoJS4w~~

If you see "All steps passed" or "Test Failed" instead of "New Baseline Created", then this means that a baseline already exists for this test and application name. This may be because you, or someone else from your team, ran this example previously. To fix this, change the test name to something unique by editing the following line which appears near the top of the code.


                            

Run the test again and when you see the "New Baseline Created" on the console, copy the first URL from the console and paste it into a browser. The Test Manager will start up with the test already loaded (you might need to log in first). Alternatively, if the test manager is not yet open in a browser, type the URL https://applitools.com/users/login into your browser, log in, and the Test Manager will start. If the Test Manager is already open before running the test click the 'update batch button' to see the new test results.

In the Batch entries list on the left, you should see two batch entries - one for each test in this test run Click on the first entry, click the "test details view" icon , and you will see a row with the test name:

Click on the test name, and the Test Manager will display the steps of the test.

Note that the batch has a status of , the test has a status of , and all three steps have a new step icon and "Accepted" icon in the top left corner. This is what we expect. The test and steps are marked as new since there was no existing baseline for this test, and Eyes has automatically accepted all the steps and saved the baseline.

Reviewing the results of a second test run

Now run the test again. This time you should see something like this on the console:

All steps passed:     3 steps
results at https://eyes.applitools.com/app/batches/00000251885688837901/00000251885688837518?accountId=mvdQ30vIlkiU8F-usoJS4w~~
All steps passed:     3 steps
results at https://eyes.applitools.com/app/batches/00000251885688825008/00000251885688824836?accountId=mvdQ30vIlkiU8F-usoJS4w~~

Press the 'update batch' button in the Test Manager, and the steps of the test should look like the screenshot below.

Note that the batch and test status have a value of "Passed" and that all the steps have the matched icon in the top left corner of the step image, indicating that checkpoint image matched the baseline image. This time, however, there is no default "Accepted" icon . Apart from the first run, this icon will only be displayed for a step after you explicitly accept that step.

Creating a batch of two test results

Till now, the two test runs appeared in the Test Manager as two separate entities in the batch list. The runs are actually the same test (the same sequence of events) that was run twice with a different viewport size each time, so it makes sense to batch them together. To do this, change your code as instructed below:


                            

Now if we run the test again and look at the Test Manager we will see a single entry "Hello World 2 Batch", and there are two test results (two rows) on the main panel. If you look at the "Viewport" column, you can see the viewport that was used in each run.

Adding some differences to a test

Now let's change the test so that there will be some differences for Eyes to discover. We'll do this in a few ways in order to demonstrate the three basic types of mismatches that Eyes can detect.

  1. We'll change the graphic look of the application by adding "?diff2" to the website URL - this is just a trick in the web application that forces it to produce some changes for this demo.
  2. We'll remove the first checkpoint from the code.
  3. We'll add an additional checkpoint after the last checkpoint of the test.

In the code sample, there is a function called test01Changed() that implements these differences. Change the value of the flag in your code as described in the comment below, so that the changed version of the function will be executed instead of the original function.


                            

Running the tests with the differences

Now run the test again. This time we will see in the console the following lines that Eyes indicate that Eyes has detected four failed steps:

 
Test Failed: 4 steps, matches=0 missing=1 mismatches=3 
results at https://eyes.applitools.com/app/batches/00000251889993686590/00000251889993686356?accountId=mvdQ30vIlkiU8F-usoJS4w~~
Test Failed: 4 steps, matches=0 missing=1 mismatches=3 
results at https://eyes.applitools.com/app/batches/00000251889993671516/00000251889993671329?accountId=mvdQ30vIlkiU8F-usoJS4w~~
			

In the Eyes Test Manager , after pressing the 'update batch button', , we will see the single batch entry with the status. Click on it, and you will see the two test results in the main panel, both with the status. Click on the first test so that it expands to show all of its steps. The screenshot below shows what you should see.

The first step has a "missing step" icon in the top-left corner since we skipped the first checkpoint. The second and third steps have a "mismatch" icon caused by differences in the output which we forced by adding the "?diff2" to the URL. The fourth step has a "new step" icon because of the extra checkpoint that we added at the end of the test.

Resolving the differences in test results

Since Eyes has found differences, the next step is to "resolve" them. As a minimum, resolving the steps means going through each step that is in an "unresolved" (orange) state and either accepting it or rejecting it. Accepting a step means that the step is considered as having passed the check. In the case of a mismatch or new checkpoint , the accept also means that the baseline should be updated from the checkpoint image. In the case where a checkpoint is marked as missing , accepting the step means that baseline image should be removed from the sequence of images. Rejecting a step means that the step has failed the check; the baseline image will either remain as is. See the article Reviewing test results and updating the baseline for a full description of what Eyes does when you accept and reject steps.

For the purposes of our demonstration let's accept all the steps in the first test (with the viewport of 1024x768) and reject all the steps in the second test (with the viewport of 500x900).

When you place your cursor over the step thumbnail, a toolbar will be displayed as below.

Press the button to accept the step and the button to reject the step.

Some things to note:

  1. As long as any step in a test is in an "unresolved" (orange) state, the test and batch state will be Unresolved.
  2. Once all the steps of a test have been accepted or rejected the test status will have a status of "Passed" with a check mark if all the steps passed or "Failed" with a check mark if one or more steps are marked as failed. The batch status is Failed since one of the tests failed.
  3. Both the test passed and the test failed status have a "diffs" status to indicate that differences were found in at least one step.
  4. There are various indication on the screen that signal that you have made changes to the baseline and that these changes need to be saved. These are marked by the green arrows in the screen image below:

Note the green arrows that show some of the places where the need to save is signaled.

Press the Save baseline button and note the changes in the saved status.

Running the test against the updated baseline

If you run the test again, then the second test will fail as it did previously. This is what we expect since we rejected all the steps, so the baseline was not updated and therefor Eyes found the same mismatches.

Examining the results of the first test, we can see that there are now only three steps instead of four. Eyes has not displayed a step that signaled a missing checkpoint, since we accepted the missing checkpoint. The last step is no longer marked as "new" since we accepted it. However, the second and third test are still showing one difference.

In the next section, we will look at some additional tools that Eyes provides to investigate and resolve these differences.

Using the Test Editor to investigate and resolve differences

Click on the thumbnail of the second step of the first test (1024 x 768) and the Test Manager will open the "Test Editor". We will only describe here briefly a few of the basic functions available in this screen. For a full description see Test Editor overview .

The Test Editor displays, and provides operations on, a single step of a test at a time. It also provides a way to move the display to other steps in the test (described further on below).

Viewing the checkpoint or baseline image

The Test Editor can display either the checkpoint image of the step or the baseline image or both images side by side. Use the image toggle control to choose between these views by clicking on the triangle on the right of the control and selecting the view you want. You can also press the T button to toggle between the checkpoint and baseline images.

Identifying the differences

There are several ways to make the differences in a step standout:

  1. If the “diffs” layer is enabled, then the impacted pixels are colored in pink.
  2. If the pinks areas are not visible then you can enable then by enabling "Diffs" in the select visible layers control

  3. If you press the “Highlight differences” control , then the Test Manager will draw animated circles in the area with the diffs of the checkpoint image.
  4. You can use the zoom in on next or previous differences controls to zoom the display to the next or previous mismatch area. You can do the same thing by using the shortcut keys < and > (without pressing shift so the actually keys are , and . but the arrow keys are easier to remember).
  5. Some types of differences can easily be noticed if you toggle quickly and repeatedly between the checkpoint and baseline images (very convenient to do with the T key).

Looking at the diffs found in steps two and three of the first test result you will see that they are caused by a text area that displays the time of day, which is different on every run, so Eyes detected a mismatch.

Eyes provides several ways to handle this situation. We will describe two of these in the next section.

Ignoring dynamic data in application pages

The test manager provides a tool you can use to mark a region on the checkpoint and baseline image and to indicate how that area should be processed when comparing the two images. When you apply the region, you can immediately see the way that it impacts the differences on the results. When you save the baseline, these regions are saved in the context of the baseline and Eyes will apply the special processing to those regions in future test runs.

We will demonstrate two options that will help us solve the mismatches found because of the dynamic time field in the second and third step.

In the Test Editor, press the Add region button . A crosshair will appear on the screen, put your mouse at the top left of the text field with the time, click and drag the mouse to the bottom right of the text field and then release the mouse. Click the rectangular region that you just added, and you should see the following:

If the text to the right of the rectangle does not say "Ignore", then click on the triangle and select the ignore option.

You have now told Eyes that once this area should be ignored when the images are compared. You can s https://applitools.com/helloworld2 ee that the pixels that were previously highlighted in pink, are no longer highlighted. The request to ignore these areas will apply to all future tests after you click the save button. (Typically you press Save only when you finish all of you reviewing, otherwise you will impact tests that run in the meantime. Changes you made are persisted when you make them, not when you press save).

Press the reject button (we don't want to change the baseline image), this does the usual reject action and also moves the Test Editor to the next (third) step.

In the third step, add another region in the same place as in the second step, but this time instead of defining an "Ignore" region define a "Layout" region. This tells Eyes that it should check that the text field is there and in the correct position and layout, but that it ignores the actual text.

Reject this step as well - Hint: you can use the shortcut R key to do this (the A key is used to accept the step).

Finally, click the save button to save all the changes to the baseline and to return to the Test Manager.

Now run the test again, and all the steps in the first test should now have a matched status. The second test will have the status as in the previous runs.

Note that the batch status is Unresolved since we have not yet resolved the second test - if you like, go ahead and resolve it the same way we resolved the first test.

Moving to other steps in the test

We've already described how when you click accept or reject in the test editor you will automatically be taken to the next step in the test. You can also move from test to test using the time line displayed at the bottom of the test editor screen. On the time line, each step is represented by a small rectangle whose color indicates the state of the step ( unresolved , accepted or rejected ). You can click on any rectangle to view that step. You can also use the time line controls to the left of the time line , to jump to the previous step, next step, previous unresolved step or next unresolved steps.

The play button plays simulates the test run by displaying all of the steps one after the other along with the user mouse and keyboard interactions that were made during the test.

Keyboard shortcuts

We've mentioned several keyboard shortcuts that can help you work efficiently in Test Manager. To see all the available shortcuts press ? in one of the Test Manager screens.

Summary

Congratulations! You now know how to:

  • Run tests written with the Eyes SDK.
  • Batch multiple tests together.
  • Identify the three different types of mismatches.
  • Use the Test Editor to investigate and resolve the mismatches in a step., and to move from one step to another
  • Use “ignore” and “Layout” regions to prevent false diffs due to dynamic data