The Test Results Page

After you run Eyes visual UI tests, you use the test results page to see the outcome of the test runs, and if necessary, to update the test baselines.

Using the controls and tools on this page you can analyze the results of the test run by filtering, sorting and grouping the information in various ways. You can look at, and compare the baseline and checkpoint images. You can accept a checkpoint image so that it will become the new baseline image. Alternatively, you can reject a step, indicating that the step failed, and you can assign the test to a team member for further action.

Layout

To open this page, in the Page Navigator, select Test results.

The test results page is divided vertically into two panels. On the left, you can see the batch list - a list of all the batches run by your team. When you click on any entry in the batch list, it becomes the current batch, and the main panel shows information on the test results of that batch. The information you can see depends on the view you select using the view selector. Each view focuses on a different aspect of the test results (batch level, tests, steps, and issues & results). See The test results views below for more details about what you can see and do in each view.

The Batch List Panel

The batches in the list are presented in chronological ordered, with the most recently run test first. Since the batch list can be very long, it is updated only when you scroll further down the list.

Above the batch list is a toolbar that allows you to:

	Description
	Select /deselect all visible batches (after any filtering).
	Delete the selected batches (see How to delete batches).
	Update the batch list (see Updating the page).
	Filter the batch list (see How to filter and search batches.)

Each entry in the batch list displays the batch name, the date and time the batch started running, and a batch status (Running, Passed, or Failed - see Test and batch status). There is also an indication if you have started resolving the tests but have not yet saved the results, and a checkbox that is used to select the batch for deletion.

When you click a batch in the list, it becomes the current batch and information about that batch will be displayed in the main panel.

Current Batch Summary Panel

At the top of the main panel, there is a panel that provides information about the currently selected batch:

The information displayed includes the name of the current batch, when the first test in the batch started, how long it took to run, and how many tests there are in the batch. Then there is a breakdown of the number of tests in different states: Unresolved, Aborted, Failed, New, Running, Passed, or Issues. Only values that are non-zero are displayed. The state labels also function as quick filters that you can click to show all test results matching that state.

The current batch status (Passed, Failed, etc.) and save state are also displayed here (see Test and batch status for more details).

Renaming A Batch

You can rename the current batch by:

1. Click on the menu that appears when you hover over a batch entry in the batch list in the left pane.
2. Select the Rename option.
3. A dialog box will open, type in the new name, or change the current name. Click the Rename button to make the change permanent, or click the Cancel button to abort the rename process.

Saving Changes To the Baseline

The save changes to baseline button will "pulse" when you have made changes and have not yet saved them to the baseline.

Click on the button to save all the tests. If you only want to update the selected baseline rather than the baselines of all the tests (since this will immediately impact tests when they next run), then in thetest details view, select Save selected test to baseline in the sub-menu opened when you hover over this button.

Updating the Page

Since tests can be running in the background, the information on this page may not be up to date. Both the batch toolbar and the test toolbar have a button. If these indicators rotate, then this means that more information is available on the server. Click on the button and the panel associated with that button will be updated.

Setting Dynamic Text Patterns

By default, the text uses the dynamic text match level set at the application level.

You can modify the dynamic text pattern for a specific test:

1. In the Test results page, in the list of tests, next to a test, click > Test settings.

   

2. Click App patterns and select the required patterns.

   

3. If you need to create a new custom pattern, click Add custom text type. For details, see Creating a custom dynamic pattern.

4. Click Apply settings.

   Note that after modifying test patterns, you can reset to the application settings.

   

Views On the Test Results

The current batch toolbar contains buttons that allow you to select the current view:

The other buttons in the View selector toolbar always contain the same buttons, but some of the operations are only enabled in particular views, and others behave slightly differently in each view.

The Test Results Views

What you actually see in the main panel depends on the current view selected in the view selector. There are four different views of the test results available:

Batch summary view

This view provides you with a high-level view of the results of the test runs and the current progress in processing these results. This view displays graphical summaries of key test attributes and states. For example, you can see the percentage of tests that have mismatches, the percentage of the steps that have been resolved, the browsers and operating systems the tests ran on, and more.

Batch details view

This view displays a list of all the tests in the batch along with key information such as the test status and execution environment. You can sort, group and filter the tests by various criteria. You can select a subset of the tests and do operations on them such as resolving all the steps, deleting the test or assigning them to a team member. You can expand a test to see thumbnail images of all the steps along with the test result for that step.

Batch steps view

In this view, you can see all the steps in the batch, irrespective of which test they were captured in. You can filter and group the steps using various criteria, and take actions such as resolving all the steps in one click.

Issues and remarks view

In this view, you can see all the steps in the batch that have remarks or issues, and edit these remarks and issues.

The Step Thumbnail

In the test detail and step details views, you can see thumbnail images steps. In the test details view, these are initially hidden, and you can see all the steps of any test by clicking on the test. In the step details view, you can filter and group the steps by various criteria.

The step thumbnail allows you to:

• See the baseline or checkpoint image.
• Highlight the differences found in the image.
• Accept or reject individual steps.
• Apply a dynamic match level to a step.
• Invoke a filter to show all the steps with the same Browser, operating system or viewport.
• Open the test editor or step editor.

Filtering Test Results

You can filter test by clicking . A window will be shown that indicates the available fields to filter on. Configure each field as needed and then click Apply to save the filters.

Applying A Dynamic Match Level To A Step

To a apply a dynamic match level to a single step, click > Step settings, select the required patterns and click Apply settings.

Grouping Results

In the Test results view you can use the control to arrange the tests (in Batch details view icon ) or steps (in Batch steps view icon ) in hierarchical groups based on one or more of the following criteria:

App
Steps or baselines have the same application name.

Branch
Steps or baselines are on the same branch.

Browser
Steps or baselines ran in the same browser

Device
Steps or baselines were rendered for the same device on the Ultrafast Grid.

Environment
Steps or baselines ran that used the same baseline environment.

OS
Steps or baselines that were tested on the same operating system name.

Test name
The baselines or steps have the same test name.

Viewport
The baselines or steps ran with the same viewport size.

Assigned to
The baselines or steps are assigned to the same person.

To choose a criteria, click and select the required criteria. You can select multiple criteria, Eyes will display a hierarchical tree arranged in the order that you requested.

For example, to see what browsers and viewport size variations your baselines have been tested against you can choose the Browser and Viewport attributes and then your display will show:

Bookmarking Tests and Steps

The Eyes Test Manager lets you bookmark tests and steps by marking them with a star on the Test results page. You can then filter for and revisit the bookmarked tests and steps in either the Test results page or the Last test run page. The image below shows both starred and unstarred tests and steps. A blue star indicates that an item is bookmarked. A white star indicates that an item is not bookmarked.

Setting and Clearing Bookmarks in a Test

To bookmark a test, click the Star icon in a tests' row, it will be highlighted in blue. To clear the star click it again.

Setting and Clearing Bookmarks in a Step

To bookmark a step, click the Star icon underneath the step thumbnail, it will be highlighted in blue. To clear the star click it again

Test and Step Editors

You can open the test and step editors from any of the views that display a step thumbnail. Clicking on the thumbnail will open up the Step editor. You can open the Test editor using the step thumbnail menu. The main difference between the Test editor and the Step editor is that the Test editor allows you to navigate through all the steps in a test, whereas the step editor allows you to step through all the steps in a batch, taking into account any filtering and grouping that you have applied to the Test results page.

The Step and Test editor tools provide you with the following capabilities:

• View larger images of the checkpoint and baseline images and zoom in on details.
• Highlight the differences between the checkpoint and baseline images.
• Use the Root cause analysis feature.
• Add remarks and issues to regions in the baseline or checkpoint image.
• Define match level, ignore and floating regions.
• Accept or reject the new image and then save the baseline.