Analyzing Results
After a test run completes, you will typically want to review the results to see which checkpoints matched the baseline images and which did not, and take any necessary corrective action. The Eyes Test Manager provides tools to do the following:
- View the match results for every step.
- Decide whether a check passed or failed.
- If a mismatch was caused by an outdated baseline image, update the baseline with the image from the test.
- Remove baseline images that no longer correspond to an existing checkpoint, and add baseline images for new checkpoints.
- Report and assign bugs to team members.
- Handle special cases such as dynamic text (e.g., time of day), which changes on every run and can trigger mismatch detection.
Accessing the Eyes Test Manager
To see the results, log in to the Applitools cloud by entering the following URL in your browser:
https://applitools.com/users/login
You will be prompted to enter your username and password. After you log in, the browser will redirect you to the Eyes Test Manager.
Test Manager Screen Overview
The Eyes Test Manager screen is divided into two main areas. The left panel displays information about batches run by your team and allows you to select the current batch to review. The right panel displays the tests and steps of the selected batch.
- The batch list panel (left) displays the most recently run batches. Clicking any entry in the list makes that batch the current batch.
- The batch management panel (above the batch list) provides tools to update, filter, and delete batch list entries.
- The main panel displays the tests in the selected batch and the steps in each test.
- The toolbar above the main panel allows you to select from various views. Each view displays different information to help you focus on your current task. Other tools let you delete tests, assign them to team members, filter, group by differences, and more.
- Above the main panel, Test Manager summarizes the current batch results and the review progress.
Test Manager Screen Details
The screenshot below highlights key features in the Test Manager:
| # | Represents |
|---|---|
| 1 | Shows how many batches are in the list. Includes controls to filter, delete, and refresh the batch list. |
| 2 | The batch list, with the newest batches at the top. Clicking one loads its details into the main panel. |
| 3 | Information about the selected batch and its review status (e.g., passed, failed, unresolved tests). |
The toolbar has four sections:
| # | Represents |
|---|---|
| 4 | Operations on selected tests and steps. |
| 5 | The current view (one of five available) for displaying batch details. |
| 6 | Buttons to reload, filter, and group steps by similar differences. |
| 7 | The Auto Maintenance menu with options for test maintenance scope and grouping steps by differences. |
Main panel details:
| # | Represents |
|---|---|
| 8 | Displays batch, test, and step details based on the selected view. |
| 9 | Access to Notifications, help, and administrative settings. |
| 10 | Copy a hyperlink to the current view to share with others. |
| 11 | Save all changes to the baseline, making them permanent for future runs. |
12  | Start a chat session with Applitools support. |
Reviewing and Resolving a Test Run
The Test Manager offers multiple views to explore test results. Each view focuses on different aspects. In this section, we use the “Test details” view. Click to open it.
You’ll see a list of tests in the batch:
Clicking on a test expands it to reveal a series of thumbnails representing steps:
What is a “step”?
A step corresponds to a checkpoint in the test code. Eyes adds a step in the results for every checkpoint it detects.
If you remove a checkpoint in the code, its associated baseline image still exists. You’ll see:
- Matching or mismatched steps with baseline images.
- New checkpoints without baseline images.
- Baseline images without matching checkpoints.
Icons on each step show its match status (from left to right: missing checkpoint, match, mismatch, new checkpoint).
You’ll need to resolve two mismatch types:
- Steps where the checkpoint doesn’t match the baseline image.
- Steps with a checkpoint but no baseline or a baseline with no checkpoint.
Accepting and Rejecting Steps
Eyes provides Accept and Reject tools. These appear when you hover over a step thumbnail.
These actions affect both:
- Whether the step is marked as passed or failed.
- Whether the baseline image is replaced with the new one.
Other operations are available as well. But first, let’s look at how workflows differ between a first test run and subsequent runs.
Example: Reviewing a test run
The workflow for handling test results the first time you run a test is different from the workflow on subsequent runs. The first time a test is run, there are no baseline images, all the checkpoints are new, and they are automatically set to be the sequence of baseline images. Apart from reviewing the checkpoint images, you are not required to resolve anything. In subsequent runs, you need to analyze and resolve all the mismatches.
Example: Reviewing a Test Run
The first time a test is run, no baseline images exist. All checkpoints are new, and they are added as baseline images automatically. You’re not required to resolve anything.
In later runs, you must review and resolve mismatches.
Reviewing the First Run
After the first run, the test status is "new." The label 
indicates this:
The 
icon shows that the step is a new checkpoint. The green bar means it passed.
The icon under the plus icon shows the step was accepted as a new baseline.
Although Eyes adds all steps by default, it’s still good practice to review them.
Reviewing subsequent runs
After the first test run, Eyes will have automatically created a baseline with all the checkpoint images captured during that first run. In subsequent test runs, Eyes compares the sequence of captured images to the sequence of images in the baseline, and there are four possible outcomes, as you can see in the following screenshot of a test result:
Note the four different symbols at the top left of each step thumbnail, and the green or orange bars to the left of the thumbnail. The green bar indicates that the test has passed, and the orange bar indicates that there are differences you need to resolve (i.e. investigate the cause of the mismatch and decide what to do about it).
The four possible match results when there is an existing baseline are as follows:
| Icon | Represents |
|---|---|
 | The checkpoint image matches the baseline image. In most cases, this is the expected result, hence the default green “pass” status. However, in some rare cases, a match can indicate a bug. An example of this is if a new feature was supposed to be displayed but is missing in the UI. In this case, you should reject the step and report the issue, then the step will be marked as failed (a red bar), and the baseline image will not be changed. On the next run, after the UI is fixed, the difference will be found and you can accept the image so that it is updated in the baseline. |
 | Eyes did not find a checkpoint image that corresponds to an existing baseline image. If the checkpoint was intentionally removed from the test, then you should accept the step and Test Manager will remove the image from the baseline and pass the step. If this is not intentional, then you can report a bug (in this case, presumably, the bug is in the test and not the application). |
 | The checkpoint image does not match the corresponding baseline image. The pink areas in the thumbnail indicate regions where differences were detected. Differences may arise for different reasons, and it is up to you to analyze the differences using the tools that Test Manager provides and take appropriate actions. Typical scenarios are:
|
 | There is a new checkpoint image with no corresponding baseline image. Typically, this is because a new checkpoint has been added, in which case you will accept the Step. Otherwise, you can reject it and report a bug (probably in the test code and not in the application). Note that we previously saw this icon in the case of a new test with a green bar. In the case of the first run of a test, all the steps are automatically accepted. In the case of a subsequent test with an existing baseline, new steps need to be resolved. |
Accept and Reject Status
After clicking Accept or Reject, icons appear on the thumbnail. The bar color (green or red) shows the test’s status.
Only new tests show the Accept icon automatically—this means the step was accepted by default.
Test Status and Batch Status
In addition to step statuses, Eyes tracks the overall status of tests and batches. These update automatically based on your actions.
Updating the Baseline
Changes you make (accept/reject steps, add/remove annotations) are saved automatically. However, they apply to future runs only after you click the button.
You can save changes even if not all steps are resolved. This is helpful if you want to rerun the test before finishing the review. But be careful—this can cause inconsistencies, especially if steps have been added or removed.
Indicators for unsaved changes include:
- An asterisk (*) next to the step name.
- An asterisk (*) next to the test name.
- A count of unsaved steps in the batch summary.
- The “Unsaved” label next to the batch status.
- A pulsing button.
When you hover over with tests selected, you’ll see:
- Save all – Save all unsaved changes in the batch.
- Save selected tests – Save changes only for the checked tests (even if unchanged).
After saving, you might see a message about expired variations.
