Applitools Baselines

A baseline stores reference images for Eyes visual tests. When you run a test, Eyes captures checkpoints and compares them to the baseline images to identify visual differences.

Baseline images depend on the test and its execution environment, including the operating system, browser type, and viewport size. Eyes maintains multiple baselines for each test, each tailored to a specific environment.

The Life Cycle Of A Baseline

When you run a test in a new environment, Eyes creates new baseline images, storing the captured images and test information.

For subsequent runs in the same environment, Eyes uses the existing baseline images to compare to the captured images.

If you make changes to a baseline using the Test Manager such as accepting a new or changed step, a new baseline is created. This baseline will be used as a reference in future tests.

Test and Execution Environment Attributes

Eyes uses the following test and execution environment attributes to identify a baseline:

• testName and appName - User-defined strings that describe the tested application and specific test. Together, they uniquely identify the test.
• Viewport size - For web applications, this is the dimensions of the browser pane. For mobile devices, it is the screen dimensions.
• Host environment - This includes HostOS (the operating system) and HostApp (the application platform, such as the browser for web applications).
• Version information - Both HostOS and HostApp can have version information, which affects how baselines are managed.

Where does Eyes get the test baseline attributes from?

How Eyes selects an appropriate baseline (Baseline fallback)

Eyes has two baseline management policies which differ based on the following considerations:

• Cases where Eyes supports multiple, independent baselines for different HostOS (such as Windows 10) or HostApp (such as Chrome 127.0) versions.
• Cases where Eyes uses an existing baseline even if the HostOS or HostApp version differs from the test environment.
• Cases where a new baseline can inherit checkpoints from a related baseline if no compatible one exists.

Based on these considerations, Eyes will use one of the following baseline policies:

• Standard policy - considers the HostOS version as part of the baseline signature and does not support baseline inheritance.
• Advanced policy - more flexible, allowing cross-version testing and baseline inheritance.

Standard baseline management policy

This policy requires matching HostOS major versions but ignores HostApp (browser) versions. For example, Windows 10 tests can't use a Windows 7 baseline, but Chrome 80 tests can use a Chrome 72 baseline. This approach assumes browser versions are generally backward compatible, while major HostOS versions are not.

If a new OS version, such as macOS 11 with Safari 14, is detected, a new baseline is created. If you want to maintain baseline inheritance, you should use the advanced baseline management policy, which can inherit baselines across major OS version changes.

Advanced version baseline management policy

Eyes selects a baseline based on:

1. Finding one or more matching baselines.
2. Selecting compatible baselines.
3. Choosing the most appropriate baseline.

Matching baselines have identical HostOS and HostApp names and Viewport sizes. The policy selects the baseline with the most appropriate versions for the test environment.

Finding a matching baseline

When you start a test, Eyes searches to see if a baseline exists for your test:

1. Eyes looks for matching baselines with the same Test name, App name, Viewport size, HostApp, and HostOS names.
2. If no matching baseline exists, a new one is created using the test's checkpoint images.
3. If exactly one matching baseline is found, Eyes uses it.
4. If multiple baselines are found, Eyes selects the most appropriate one.

Choosing a baseline from multiple versions

If multiple matching baselines are found, Eyes selects the most suitable one by comparing the HostApp and HostOS versions. Two criteria are used:

• The baseline version must be compatible with the test version. Major IE versions (e.g., IE 9, IE 10) are incompatible, while all Chrome versions are compatible. For further information, see When are versions compatible?.
• Versions have an inherent ordering. A test can use a baseline from an older version but not a newer one.

Given several compatible baselines, Eyes usually chooses one matching the test's execution environment or the latest previous version.

Priority	HostApp version	HostOS version
1	Same version	Same version
2	Same version	Latest previous version
3	Latest previous version	Same version
4	Latest previous version	Latest version (HostApp latest has priority)

When are versions compatible?

HostOS and HostApp compatibility depends on versioning. A version string has one or more components separated by periods (e.g., 10.1.3). The major version number is numeric; others can be alphanumeric. For details, see Version format.

Attribute	Compatibility	Example
HostOS = any value	Compatible with any version	Android 10.1 is compatible with both Android 10.0 and 9.2.
HostApp = “Safari”, “Firefox”, “Edge”, “Chrome”, “Opera"	All versions are compatible	Chrome 81 is compatible with any other Chrome version.
HostApp = “IE”	Major component matches	IE 10 is not compatible with IE 9. Any IE 10 sub-version is compatible with other IE 10 versions.n
HostApp = any other value (HostApp that is defined using the setHostApp method)	Major component matches	MyApp 1.1 is compatible with MyApp 1.2 but not with MyApp 2.1.

Running cross environment tests

Cross environment tests use baselines with different execution environments. For example, you can test your application on different browsers against a verified baseline on another browser.

Use setBaselineEnvName to set the baseline environment name. The HostOS, HostApp, and viewport size are defined using Test Manager. For further information, see Running cross-environment tests.

The baseline selection rules differ for cross-environment tests:

• If the test execution environment does not match the baseline environment name, the latest compatible OS and browser version is used.
• If the test execution environment matches the baseline environment name, the baseline is chosen based on the test environment version, as described in Choosing a baseline from multiple versions.

Version format

Eyes extracts the version string from the execution environment information. If you define the HostOS or HostApp yourself using setHostApp or setHostOS and want Eyes to use a fallback baseline, format the version string as follows:

1. The version string must appear after the name, separated by one space.
2. The version string must consist of one or more parts separated by periods. The major component) must be numeric, and the other components can be numeric or alphanumeric. For example, a-z,A-Z,0-9, or _.
3. When determining which version is older, if the version only contains numbers they are assumed to be numeric. If the version includes non-numeric characters, they are compared alphabetically. If a version is identical to a different version, but has an extra part at the end, the shorter version is considered older. For example, 1.0.0 is considered older than 1.0.0.A
4. If the version string does not match the above format, it is considered part of the name. This means that a change in version information will create a new baseline, with no fallback to a previous compatible version.

For further information, see our video, Validating Baselines in Applitools Eyes and Creating Multi-Basleine Variations:

How Eyes compares checkpoints and baseline images

As the application being tested evolves, so do its tests. New checkpoints are added to test new functionality and checkpoints are removed because the application functionality or flow has changed.

Instead of having a static association between every checkpoint and a corresponding baseline image, Eyes compares a sequence of images captured by checkpoints to a sequence of baseline images, allowing for the fact that checkpoints may have been added or removed. Then, very much like a textual "diff" program, Eyes finds images that have been added or removed as well as images that have a corresponding baseline image and either match or mismatch. Eyes then enables you to both verify existing checkpoints and ensure that the set of baseline images is up to date.

Test results and steps

When a visual UI test executes, Eyes verifies the entire sequence of checkpoints against the sequence of baseline images. The result of this verification - the test result - consists of a sequence of steps, each denoting a missing checkpoint, a new checkpoint, a checkpoint that matches its baseline counterpart, or one that differs from it.

For example, the baseline of a test may consist of the following five images:

Now, imagine that the application and test have been changed, and the test generates a different sequence of images. For example:

Comparing the two sequences, we can see that:

• Image B has been removed
• Image C has changed to C1
• Images F and G have been added
• Images A, D, and E have not changed and are in the same order

Eyes will detect these differences and generate a sequence of steps as follows:

Each step represents one of the four possible outcomes:

1. Match: Eyes has found a baseline and checkpoint image that match each other.
2. Diff: Eyes has found a baseline and checkpoint image that correspond to each other, but there are mismatches.
3. Missing: Eyes has detected that an existing baseline image has no corresponding checkpoint image.
4. New: Eyes has detected a new checkpoint image (i.e. an image in the captured sequence which has no corresponding image in the baseline).

Notes

While you can freely add and remove checkpoints, Eyes assumes that the order of captured images that are not new does not change. If you change the order of two checkpoints, then depending on the content of these checkpoints, Eyes could report two missing steps and two new steps, or it could report two steps with mismatches.

When you define a checkpoint in the code, you can optionally provide a name for that checkpoint. The name is intended to be descriptive so as to allow you to easily and conveniently navigate and search within the test results. Eyes does not take the name into account when matching checkpoint and baseline images. The name is optional, does not need to be unique, and you are free to change it without impacting the testability of the code.