Working with baseline variations

Baseline variation is an Eyes feature that can be used when an application is undergoing A/B testing. A/B testing is a technique used to achieve a web page or application screen design that is "optimal" from some point of view. The designers generate several variations of a page, and each time the page is viewed, a different variation is displayed and the impact on the user is measured in some way.

This technique is potentially a challenge to visual testing, which is based on comparing a checkpoint image to a baseline image, since different legitimate variations can't all match the same baseline image.

Eyes supports visual testing of applications that are undergoing A/B testing by allowing checkpoints to be matched against a baseline that includes multiple variations of the baseline image per checkpoint. Eyes will consider a checkpoint image to be matched if it matches any variation of the baseline image.

When a new variation is introduced into a test and a visual test is run, Eyes will initially detect a mismatch and will set the step status to Unresolved since the checkpoint does not match any of the existing variations defined for the baseline. The Test Manager A/B testing support allows you to specify that a checkpoint flagged as Unresolved is a new variation. After you mark the image as a variation, the status of the step will be Passed. If you save the baseline and run the test again with either the original page or the page with the new variation, the check will have a Passed status in both cases.

Overview of main features

Eyes also supports Automated test maintenance for Baseline variations when adding a new variation and removing variations. This feature is described in more details in the article Automated test maintenance of baseline variations.

Related features

Eyes also supports variations in other features. The list below describes them briefly, and you can access more detailed information in the links provided.

  1. When you add an ignore, floating, or match level region to a step variation, the new region is specific to that variation - it is not applied to the baseline as a whole. When you add a region, if you choose to apply auto maintenance, then the region may be added to the other variations of that step or other steps following the usual auto maintenance rules. If you add the region to the checkpoint window (as opposed to the baseline window) , then it will be applied to the variation that is currently marked as the active variation. When you run an Eyes test, after Eyes has chosen the closest variation, it will take into account any regions defined in the chosen variation and apply them.
  2. On the Branch baseline page you can filter the list of baselines to show only baseline that have variations.
  3. On the Compare and merge branches page you can do the merge operation you require on each variation separately.
  4. On the Test baselines page you can set the filter to only show baselines that have variations.

Detailed Operations

View step variations - the Variation gallery

In order to see all variations of a step and manage them, open the Variations gallery in the Step editor or Test editor by clicking on in the toolbar. A pane opens on the left, displaying the variation gallery which contains one control card for each variation currently defined for that step:

Each variation card includes a thumbnail of the variation, the name of the variation, and dates showing when the variation was last matched and updated. One of the cards is highlighted to indicate that it is the active card (i.e. the card that is currently displayed as the baseline image).

When you hover over any variation card apart from the active card, the following toolbar appears:

When you hover over the variation card that is currently active, the toolbar is slightly different:

Note that the remove tool is grayed out since you can't remove the active variation. The active card has an additional tool which you can use to Remove all step variations except for one.

Add a new baseline variation

You can add a new variation to a step so that in subsequent runs it will be recognized as a legitimate baseline image for that step.

To add a new variation to a step:
  • If the variations panel is not open, open the Variations gallery in the Step editor or Test editor by clicking on in the toolbar. This will open the variation gallery in a pane on the left of the window.
  • Click Create new.
  • In the dialog box that opens, enter the name of the variation. The name must be unique within that step. If the text in the box is not unique, then an error message appears and the Create button is grayed out. If you change the text so that it is unique, the error message disappears and the Create button is enabled.
  • Click Create to save the new variation. Click the Cancel button to abort creating the new variation.

The gallery will now show the checkpoint as a new variation in the variation gallery and the baseline image will display the new variation.

To make the variation available for future tests, exit the editor and click .

When you add a new variation, Automated test maintenance is triggered. See Working with baseline variations for details.

Maximum variations reached

Eyes supports a maximum of 20 variations per step. If by adding a new variation you exceed this number, Eyes displays a message telling you that you must reduce the number of variations. In order to save your new variation you must first delete at least one existing variation as described in Remove a baseline variation.

Overwrite the baseline image of an existing baseline variation

If the checkpoint with a status of Unresolved is not a visual bug or a new variation, but is an update to an existing variation, then you can override the existing baseline variation image by accepting that variation. This also accepts the step as a whole.

If the variations panel is not open, open the Variations gallery in the Step editor or Test editor by clicking on in the toolbar. This will open the variation gallery in a pane on the left of the window.
  1. Hover over the variation card you want to accept and click on .
  2. Exit the editor and click on . This will save the current checkpoint image as a new version of the accepted variation of the baseline image as well as any other changes you have made. In subsequent tests, checkpoints will be matched against this newly updated variation image.

Set the currently displayed baseline variation

When a step has variations, the baseline panel in the Step editor and Test editor always shows one of the step variations. When you first open a step, this will be the variation chosen by Eyes as the "closest" variation. You can change the variation displayed in the baseline image by clicking on the variation widget.

Rename a baseline variation

You can change the name of a variation as follows:
  • If the variations panel is not open, open the Variations gallery in the Step editor or Test editor by clicking on in the toolbar. This will open the variation gallery in a pane on the left of the window.
  • Hover over the variation card whose name you want to change and a toolbar will appear.
  • Click on the icon.
  • In the dialog box that opens, edit the current name or delete it and enter a new name.
  • The new name must be unique within that step. If this is not the case, an error message appears and the Create button is grayed out. If you change the name so that it is unique, the message disappears and the Create button is enabled.
  • Click the Rename button to save your change or the Cancel button to abort the change.

Remove a baseline variation

It is best practice to remove variations that are no longer in use since testing variations that do not occur in practice increases test time unnecessarily. In addition, you may find that having a variation gallery with unused variations makes it less convenient to view and use.

In the sections that follow, we will describe how to:

When you add a new variation, Automated test maintenance is triggered. See Working with baseline variations for details.

Understanding variation removal

The term "removing a variation from a step" means that in subsequent runs, Eyes will no longer consider this variation for a step, and the variation will not appear in the variation gallery for the step. When you remove a variation in a test run, it does remain available (albeit grayed out) in the test run it was "removed" from and can be recovered by clicking on the tool in the variation card or using the step or test undo changes menu option. After it is recovered and the baseline is saved, a variation is once again available for matching, and will appear in the variations gallery of that step in future runs of that test.

Remove a variation using the editor

To remove a variation open the Variations gallery in the Step editor or Test editor by clicking on in the toolbar., hover your mouse over the variation you want to change and a toolbar will appear:
  • Click on the icon.
  • The variation card becomes grayed out, and the tool is replaced by a tool. While the variation is in this state, you can't rename it or accept it.

Recover a removed variation

You can recover a removed variation by clicking the tool which appears when you hover over a variation card.

Remove all step variations except for one

You can also delete all variations except for one by:
  • Accepting the variation that will remain by clicking on in the variation widget.
  • Clicking on in that variation. This button only appears on the currently accepted variation.

Remove a variation using the Batch variation gallery

You can see all the variations defined in the batch in the Batch variation gallery. Open this gallery from the Test results page by clicking on the in the Test results toolbar.

You can remove individual variation using the tool in each variation card. Alternatively, you can use the standard sort, filter, and grouping tools to isolate the variations you want to remove, select them individually or using the "select all" checkbox in the toolbar, and then delete all the selected tests using the in the toolbar.

If a variation card contains the grayed out in the toolbar, it means that this variation is currently being used as the baseline image and cannot be deleted.

Expiration of unmatched variations

Eyes provides an automatic background monitoring system to log when a variation has been used for matching in a test. When a variation has not been matched by Eyes or updated by a user for a pre-configured amount of time, Eyes will flag it as Expired and bring it to your attention in the following ways:

  • In the step gallery, any step that has expired will be flagged as follows:

  • When you save a baseline by clicking on , Eyes will save the baseline and then, if there are any expired variations, display the following dialog, which gives you the choice to delete all the variations in one step, switch to the Baseline variation gallery to view the expired variations and delete them selectively, or leave the variations as they are:

Configuring the variation expiration time

Users with Admin rights can configure the amount of time after which an unmatched variation will be considered expired as follows:

  • Enter the Admin page.
  • Click the Teams tile.
  • Click the row of the team whose expiration time you want to configure.
  • Click the Settings tab.
  • In the box labeled Expire after, enter the number of days after which variations will be considered expired if it has not been matched.
  • Save the results by clicking .

For further information, see our video, Validating Baselines in Applitools Eyes and Creating Multi-Basleine Variations: