Automated test maintenance of baseline variations

This article explains how the Eyes Automated test maintenance feature works together with the Baseline variations feature.

Baseline variations is an Eyes feature that allows you to define multiple reference images for a baseline step. This is useful when running Eyes tests on applications that are undergoing A/B testing where the checkpoint image may be intentionally different in different test runs.

Automated test maintenance is an Eyes features that automatically detects when actions taken by a Test Manager user and duplicates any action taken on similar steps. In the case of Baseline variations, steps are considered similar when they are assigned to the same baseline group. Checkpoints A point in the test code at which the screen image is grabbed and compared to a baseline image that corresponds to the check point. can be grouped by the user explicitly by assigning them a common variation group ID using the SDK, or Eyes can automatically detect checkpoints that are likely to be in the same variation group.

Automated test maintenance is triggered on Baseline variations when a user defines that a step is a new variation, or deletes an existing variation. Eyes identifies steps that are in the same variation group and duplicates the creation/deletion action for those steps as well.

Here are some examples where Automated test maintenance of the Baseline variations feature can simplify your workflow and make it more efficient:

• A wizard consists of a series of dialogs and is currently undergoing A/B testing to evaluate styling options for these dialogs. You are testing the wizard using Eyes and a new variant is introduced. Eyes detects mismatches for the steps that are impacted by that variant. You see that the mismatch is because of a new variant and add the checkpoint image as a new baseline variation and then Eyes searches for other checkpoints in the same variation group as that step and defines them as new baseline variations as well. You can display these automatically created baseline variations and confirm or cancel the baseline variation creation.

• You run a test using the Ultrafast Grid on, say, 10 execution environments An execution environment is defined as a triplet of <OS, browser, viewport size>. When a test runs, its execution environment is defined by the platform on which the test runs, or, if run on the Ultrafast Grid, by the configuration of the Ultrafast Grid. A baseline is also associated with an execution environment. By default, the baseline chosen for a test is the baseline that has the same application name, test name, and execution environment as the test. Note that if the initial page in a test is not a secure web page, the execution environment may report the wrong OS or browser version.. The test undergoes A/B testing and a new variant is introduced so that a mismatch is reported in the test results of each of the execution environments. You look at a step with a mismatch status, see that it is a variation and create a new baseline variation with the checkpoints image. Eyes automatically detects the equivalent step in the other execution environments, assigns these steps to the same variation group, and creates baseline variants for these steps as well.

• You delete an existing step variation since it is no longer required. Eyes detects that there are other steps in the same variation group and deletes the variation in those steps as well. You can review what has been deleted and undo any automatically deleted variations.

You can read about Baseline variations in the article Working with baseline variations and about Automated test maintenance in the article Automated test maintenance.

The remainder of this article consists of the following sections:

• How to define variation groups using the SDK – This section describes how to use variationGroupId to associate a variation group ID with a checkpoint so that it is part of the variation group associated with that ID.

• Cases where Eyes automatically defines variation groups – This section describes the cases where Eyes will identify related steps and assign these to a common variation group.

• Auto maintenance of variation groups – This section describes which Test Manager Baseline variations user actions trigger Automated test maintenance and provides links to the relevant Test Manager documentation.

How to define variation groups using the SDK

You can associate a checkpoint with a variation group by assigning it a unique string ID that represents the that group. This is done by appending the variationGroupId method to the eyes.check fluent API chain, as shown in the following snippet:

eyes.check("stepName",Target.window().variationGroupId("variation1");

The "variation1" string parameter is the variation group ID and will be associated with all of the test result steps that result from executing this checkpoint. This may be a result of running the test multiple times or the result of running the test once using the Ultrafast Grid with multiple execution environments.

Typically a unique variation group Id is used for every checkpoint, and the multiple instances of the variation IDs come about from running the test under different execution environments orscenarios.

Cases where Eyes automatically defines variation groups

The recommended way to achieve a smooth and accurate variation group workflow is to define variation groups explicitly, as described in Automated test maintenance of baseline variations. Eyes also automatically groups together steps into a variation group when they meet all the following conditions:

• The steps were created on a single test execution using the Ultrafast Grid.

• The steps have not been explicitly assigned to a variation group.

• The steps are part of a test with the same test name.

• The steps have the same step name and:

  • The step name is unique within its test, or

  • The common step name is not unique within the test, but all of the steps share the same step index (the position of the step within the series of steps in the test).

Auto maintenance of variation groups

Automated test maintenance is a Test Manager feature that intelligently identifies steps in a batch with similar differences and duplicates actions that you take on a step on the steps found to be similar. You can then review these changes and undo or change the automatically taken actions.

When steps in multiple tests are assigned the same version group ID, they are considered similar and trigger Automated test maintenance for the following user actions:

• Add a new variation

• Remove variations

Automated test maintenance is introduced in the article Automated test maintenance. The following sub-topics provide more specific details:

• Setting the automated test maintenance scope – Automated test maintenance always applies to steps in the current batch. Within the current batch, you can limit the scope further to given selected steps or tests, or to the "current step," which effectively disables Automated test maintenance.

• Automated test maintenance notifications and messages – How the Test Manager indicates that it is looking for, and has found Automated test maintenance tasks.

• Reviewing groups and their steps – How to review and, if necessary, change the actions taken by Automated test maintenance.