The Test results page

After you run Eyes visual UI tests, you use the test results page to see the outcome of the test runs, and if necessary, to update the test baselines.

Using the controls and tools on this page you can you can analyze the results of the test run by filtering, sorting and grouping the information in various ways. You can look at, and compare the baseline and checkpoint images. You can accept a checkpoint image so that it will become the new baseline image. Alternatively, you can reject a step, indicating that the step failed, and you can assign the test to a team member for further action.

The main tasks you can achieve on this page are:
  • View the overall results of the batch (did all the test pass ?). See Understanding the batch status.
  • Accept or reject all the differences in a test or group of steps.
  • Annotate tests with issues and remarks.
  • Update the baseline.
  • Delete test results - see How to delete batches.
In addition, in the views that display a step thumbnail, you can open the test editor or step editor tools. These allow you to make side by side comparisons of the checkpoint and baseline images, and also provides the following functions:

Layout

You open this page using the page navigatorThe Page navigator is a control at the top of the Test manager window, that is used to switch between the main pages of the Test manager. It also displays the name of the current page. Test result menu entry.

The test results page is divided vertically into two panels. On the left, you can see the batch list - a list of all the batchesTwo or more tests can be grouped together into a "batch" using the SDK set batch method/property. In the Eyes Test Manager Test result page, you can see a list of all the batches that have run. run by your team. When you click on any entry in the batch list, it becomes the current batch, and the main panel shows information on the test results of that batch. The information you can see depends on the view you select using the view selector - each view focuses on a different aspect of the test results (Batch level, tests, steps and issues & results). See The test results views below for more about what you can see and do in each view.

The batch list panel

The batches in the list are ordered time-wise, with the most recently run test first. Since the batch list can be very long, it is updated only when you scroll further down the list.

Above the batch list is a toolbar that allows you to:

Each entry in the batch list displays the batch name, the date and time the batch started running and a batch status (Running, Passed, Failed - see Understanding the batch status). There is also an indication if you have started resolving the tests but have not yet saved the results and a checkbox that is used to select the batch for deletion.

When you click a batch in the list, it becomes the current batch and information about that batch will be displayed in the main panel.

Current batch summary panel

At the top of the main panel there is a panel that provides information about the currently selected batch:

The information displayed includes the name of the current batch, when the first test in the batch started, how long it took to run and how many tests there are in the batch. This is followed by a breakdown of the numbers of tests in different states : unresolved, aborted, failed, new, running, passed, or have issues. Only values that are non-zero are displayed. The current batch status (Passed, Failed and so on) and save state are also displayed here - See Test and Batch Status more details.

You can rename the current batch by hovering over its name, clicking on it and typing in a new name - see How to rename a batch for more details.

Saving changes to the baseline

The save changes to baseline button will "pulse" when you have made changes and have not yet saved them to the baseline.

Click on the button to save all the tests. If you don't want to update the baselines of all the tests (since this will immediately impact tests when they next run), then in test details view, select Save selected test to baseline in the sub-menu opened when you hover over this button.

Updating the page

Since tests can be running in the background, the information on this page may not be up-to-date. Both the batch toolbar and the test toolbar have a button. If these indicators rotate, then this means that more information is available on the server. Click on the button and the panel associated with that button will be updated.

Views on the test results

The current batch toolbar contains buttons that allow you to select the current view:

The other buttons in the toolbar view toolbar always contains the same buttons, but some of the operations are only enabled in some of the views, others behave slightly differently in each view. The buttons are described in the view documentation.

The test results views

What you actually see in the main panel depends on your current view selected in the view selector, there are four different views of the test results:
Batch summary view

This view provides you with a high level view of the results of the test runs, and the progress in processing these results, This view displays graphical summaries of key test attributes and states. For example, you can see the percentage of tests that have mismatches, the percentage of the steps that have been resolved, what browsers and operating systems the tests ran on and more.

 Batch details view
This view displays a list of all the tests in the batch along with key information such as the test status and execution environment. You can sort, group and filter the tests by various criteria. You can select a subset of the tests and do operations on them such as resolving all the steps, deleting the test or assigning them to a team member. You can expand a test to see thumbnail images of all the steps along with the test result for that step.
 Batch Steps view
In this view, you can see all the steps in the batch, irrespective of which test they were captured in. You can filter and group the steps using various criteria, and take actions such as resolving all the steps in one click.
  Issues & remarks view
In this view, you can see all the steps in the batch that have remarks or issues.

The step thumbnail

In the test detail and step details views, you can see thumbnail images steps. In the test details  view, these are initially hidden, and you can see all the steps of any test by clicking on the test. In the step details  view, you can filter and group the steps by various criteria.

The step thumbnail allows you to :
  1. See the baseline or checkpoint image
  2. Highlight the differences found in the image
  3. Accept or reject individual steps
  4. Invoke a filter to show all the steps with the same Browser, operating system or viewport.
  5. Open the test editor or step editor.

The test and step editors

You can open the test and step editors from any of the views that display a step thumbnail. Clicking on the thumbnail will open up the test editor. You can open the step editor using the step thumbnail menu. The main difference between the test editor and the step editor is that the test editor allows you to navigate through all the steps in a test, whereas the step editor allows you to step through all the steps in a batch, taking into account any filtering and grouping that you have applied to the test results page.

The step and test editor tools provide you with the following capabilities:
  • View larger images of the checkpoint and baseline images and zoom in on details.
  • Highlight the differences between the checkpoint and baseline images.
  • Add remarks and issues to regions in the baseline or checkpoint image.
  • Define match level regions.
  • Accept or reject the new image.
  • Save the updated baseline.