A/B Testing With Baseline Variations

What are Baseline Variations?

Baseline variations are a feature in Eyes that allows multiple reference images for a particular baseline step. This can be particularly useful in scenarios such as A/B testing. In A/B testing, various versions of a web page can be displayed randomly where the impact on the user is measured in different aspects.

Baseline variations cater to visual testing of such applications by using Visual AI to automatically match the checkpoint image against various baseline images. The checkpoint image would be considered as a match if it matches any variation of the baseline image.

Create a Baseline Variation

You can add a new variation to a step so that in subsequent runs it will be recognized as a legitimate baseline image for that step.

note

A maximum of 20 baseline variations can be created for a given test step

1. Open the Variations gallery in the Step editor or Test editor by clicking on the icon shown below in the toolbar. This will open the variation gallery in a pane on the left of the window.

   

2. Click Create new.

   

3. In the dialog box that opens, enter the name of the variation. The name must be unique within that step. If the text in the box is not unique, then an error message appears and the Create button is grayed out.

4. Click Create to save the new variation. Click the Cancel button to abort creating the new variation.

   

The gallery will now show the checkpoint as a new variation in the variation gallery and the baseline image will display the new variation.

To make the variation available for future tests, click the Save Button.

Manage Baseline Variations

View Variations

To view all existing variations, open the Variations gallery in the Step or Test editor by clicking the relevant icon in the toolbar.

The Variation gallery displays one control card for each current variation defined for that step. Each card contains a thumbnail of the variation, its distinct name, and timestamps related to when it was last matched and updated.

The card that is highlighted signifies the active card, the one currently being displayed as the baseline image. If you hover over any variation card, a toolbar appears with multiple options for editing.

Update a Variation Image

Updating a variation involves overwriting the baseline image of an existing variation. To do this, follow the steps below:

1. In the toolbar that appears when hovering, click on the accept icon. This will overwrite the existing variation with the current checkpoint image.

   

2. After accepting the new variation, you will need to save your changes.

   

tip

Remember, making changes to a variation this way updates the version of the accepted baseline image. In subsequent tests, checkpoints will be matched against this newly updated variation image.

Remove a Variation

The term "removing a variation from a step" means that in subsequent runs, Eyes will no longer consider this variation for a step, and the variation will not appear in the variation gallery for the step. When you remove a variation in a test run, it does remain available (albeit grayed out) in the test run it was "removed" from and can be recovered if needed.

To remove a variation, click on the delete icon. This will remove the variation from this test step.

Recover a Variation

A removed variation can be recovered by clicking on the undo tool in the variation card.

Merge Variations Into One

If your A/B test is over and you no longer need all of the variations, you can merge all variations into a single baseline image using the button shown below:

Auto Maintenance

Automated test maintenance is a premium feature in Eyes that works in conjunction with Baseline Variations. It intelligently identifies steps in a batch that are similar based on a set of predefined criteria and replicates any actions, such as the addition or deletion of a new variation, that you perform on one step across all similar steps. You can read more about Auto Maintenance here.

Defining Variation Groups

You can associate a checkpoint with a variation group by assigning it a unique string ID that represents the that group. This is done by appending the variationGroupId method to the eyes.check fluent API chain, as shown in the following snippet:

The 'variation1' string parameter is the variation group ID and will be associated with all of the test result steps that result from executing this checkpoint. This may be a result of running the test multiple times or the result of running the test once using the Ultrafast Grid with multiple execution environments.

Typically a unique variation group Id is used for every checkpoint, and the multiple instances of the variation IDs come about from running the test under different execution environments orscenarios.

While users can define variation groups explicitly, Eyes is also capable of automatically grouping steps into a variation group based on certain conditions. This includes:

• The steps were created on a single test execution using the Ultrafast Grid.
• The steps have not been explicitly assigned to a variation group.
• The steps are part of a test with the same test name.
• The steps have the same step name and:
  • The step name is unique within its test, or
  • The common step name is not unique within the test, but all of the steps share the same step index (the position of the step within the series of steps in the test).

Auto Maintenance Triggers

The primary triggers for Automated test maintenance include adding a new variation or removing an existing one. On performing any of these actions, Eyes searches for other steps that are part of the same variation group and automatically replicates the action across those steps.

Review and Confirm

After Eyes duplicates the user operations, you can review these automatically created baseline variations. It's recommended to verify these changes and either confirm or undo the automatically applied actions. This act of reviewing and confirming the modifications is a crucial part of automated test maintenance, ensuring that only the correct and necessary changes are carried forward.

Examples

Here are some examples where Automated test maintenance of the Baseline variations feature can simplify your workflow and make it more efficient:

• A wizard consists of a series of dialogs and is currently undergoing A/B testing to evaluate styling options for these dialogs. You are testing the wizard using Eyes and a new variant is introduced. Eyes detects mismatches for the steps that are impacted by that variant. You see that the mismatch is because of a new variant and add the checkpoint image as a new baseline variation and then Eyes searches for other checkpoints in the same variation group as that step and defines them as new baseline variations as well. You can display these automatically created baseline variations and confirm or cancel the baseline variation creation.

• You run a test using the Ultrafast Grid on, say, 10 execution environments. The test undergoes A/B testing and a new variant is introduced so that a mismatch is reported in the test results of each of the execution environments. You look at a step with a mismatch status, see that it is a variation and create a new baseline variation with the checkpoints image. Eyes automatically detects the equivalent step in the other execution environments, assigns these steps to the same variation group, and creates baseline variants for these steps as well.

• You delete an existing step variation since it is no longer required. Eyes detects that there are other steps in the same variation group and deletes the variation in those steps as well. You can review what has been deleted and undo any automatically deleted variations.

Expired Variations

The Expired Variations feature in Applitools helps monitor and manage unused variations. Applitools keeps track of all variations used for matchings in a test and logs their usage. If a variation hasn't been matched or updated by a user for a predetermined amount of time, it will be marked as "Expired."

Identification of Expired Variations

When a variation expires, Applitools brings it to your attention through several markers:

• Within the step gallery, any step bearing an expired variation is flagged.

• While saving a baseline, if there are any expired variations, a dialog box pops up, giving users the choice to delete all expired variations at once, view the expired variations and delete selectively, or maintain the variations as they are.

Configuring the Expiration Time

Users with admin rights are allowed to configure the amount of time a variation can stay unmatched before it is considered expired. The following steps are required to set the expiration time:

1. Navigate to the Admin page.
2. Click on the Teams tile.
3. Select the row of the team whose expiration time you need to set.
4. Go to the Settings tab.
5. Within the "Expire after" box, input the number of days after which variations will be considered expired if not matched.
6. Save the changes by clicking on the save button.

Related features

Eyes also supports variations in other features. The list below describes them briefly, and you can access more detailed information in the links provided.

• When you add an ignore, floating, or match level region to a step variation, the new region is specific to that variation - it is not applied to the baseline as a whole. When you add a region, if you choose to apply auto maintenance, then the region may be added to the other variations of that step or other steps following the usual auto maintenance rules. If you add the region to the checkpoint window (as opposed to the baseline window) , then it will be applied to the variation that is currently marked as the active variation. When you run an Eyes test, after Eyes has chosen the closest variation, it will take into account any regions defined in the chosen variation and apply them.
• On the Branch baseline page you can filter the list of baselines to show only baseline that have variations.
• On the Compare and merge branches page you can do the merge operation you require on each variation separately.
• On the Test baselines page you can set the filter to only show baselines that have variations.