Working with baselines

Eyes visual inspection is based on comparing images captured during a visual test to images previously stored in a baseline. Since a test can have a different look and feel when it runs on different execution environments (e.g. different devices, browsers, browser sizes and operating systems), Eyes supports having multiple baselines per test, where each baseline represents the look and feel of the test running on a specific environment.

This article describes how Eyes defines baselines, and how it chooses which baseline to use for a given test running on a given execution environment.

The article is divided into two sections:

  • Overview: Intended for new users, this section explains what an Eyes baseline is; which attributes identify the test and execution environment; and when baselines are created, used, and updated.
  • How Eyes selects an appropriate baseline: Intended for advanced users, this section explains how Eyes chooses which baseline to use when a test has multiple baselines for multiple environments.

Overview

What is a baseline?

A baseline stores a sequence of reference images. When you run an Eyes visual test, you drive the application through a series of application states, and for each state you execute a checkpoint which captures an image of the application in that state. Eyes compares the sequence of captured images to the sequence of reference images stored in the baseline of that test and reports any significant differences.

The images in the baseline depend on the test being run, but they may also depend on other factors, such as the operating system, the type of browser and the browser viewport size. These factors are collectively termed the execution environment.

Eyes automatically maintains multiple baselines for each test, where each baseline stores a sequence of reference images for a specific execution environment.

In summary, a baseline stores a sequence of images that define the expected outcome of capturing checkpoint images of a given test running under a given execution environment.

The life cycle of a baseline

The first time you run a test in a new execution environment, a baseline for that environment won't exist, so Eyes will declare the test to be a new test, and, by default, will create a new baseline that contains the sequence of images captured by the test's checkpoints. Along with the images, Eyes stores information such as the test name and the execution environment.

On subsequent runs of the test in that environment, Eyes will find the existing baseline for that execution environment and will compare the test run checkpoint images to the images in that baseline.

Once a baseline exists and is used as a reference in a test, if you use the Eyes Test Manager to make changes to the baseline, such as accepting a new or changed step, and then save the baseline, then a new baseline is created - and this baseline will be used as a reference in future tests.

Test and execution environment attributes

Eyes uses the following test and execution environment attributes to identify a particular baseline:

  • testName and appName: Two user-defined strings that describe the tested application and the specific test. Together, they uniquely identity the test.
  • Viewport size: For applications running on a web browser, Viewport size is the dimensions of the browser pane that displays the content (e.g. for mobile devices, the Viewport size is the dimensions of the screen).
  • Host environment: This consists of two attributes called the HostOS and the HostApp. The HostOS represents the operating system the test ran on and the HostApp represents that application platform (e.g. for web applications, this is the browser).
  • Version information: Both the HostOS and the HostApp can have version information. When multiple baselines match the test run in all other attributes, Eyes uses the version information to intelligently choose an appropriate baseline.

Where does Eyes get the test baseline attributes from?

Some of the test attributes are derived by Eyes when you start a test by calling class$eyes$open:

eyes.open(driver, appName, testName, viewportSize)
  • The test is identified by the strings appName and testName.
  • When the images are being captured on the test browser (as opposed to a test that uses the Ultrafast Grid):
    • The Viewport size attribute is defined by the viewportSize parameter.
    • The HostApp attribute is derived from the test browser name and version (i.e. from the driver parameter).
    • The HostOS attribute is derived from the operating system the browser runs on.
  • When the test uses the Ultrafast Grid, the execution environment attributes are derived from the UFG browsers used to render the checkpoint images, which are specified using the following UFG configuration methods:
    
    suiteConfig.addBrowser(width, height, BrowserType.CHROME)
    suiteConfig.addDeviceEmulation(DeviceName.Galaxy_S5, ScreenOrientation.LANDSCAPE)
    • When the target is defined using the configuration$addbrowser method, the Viewport size is based on the width and height parameters and the HostOS and HostApp are based on the BrowserType passed.
    • When the target is defined using the configuration$adddeviceemulation method, the Viewport size is based on the screen size of the device, the HostApp is Chrome and the HostOS will depend on the type of device.

The class$configuration$sethostapp and configuration$sethostos methods allow you to define the execution environment explicitly. A detailed discussion is not in the scope of this article.

How Eyes selects an appropriate baseline

Eyes chooses a baseline based on three criteria: 

  • Finding one or more matching baselines.
  • Selecting baselines from the set of matching baselines that are compatible.
  • Selecting the most appropriate baseline from the compatible baselines .

The first criteria selects all the baselines associated with the current test that match the test execution environment (i.e. have identical HostOS and HostApp names and an identical Viewport size).

The second and third criteria deal with a situation where a test has multiple matching baselines with different HostOS and HostApp versions. Eyes needs to select the baseline with HostOS and HostApp versions that are the most appropriate for the corresponding versions of the test execution environment.

These criteria are described in detail in the sections that follow.

Finding a matching baseline

When you start a test, Eyes searches to see if a baseline exists for your test:

  • Eyes searches for one or more matching baselines (i.e. baselines that have the same Test name, App name, Viewport size, and HostApp and HostOS names). The matching baselines will have different values for the HostOS version and/or HostApp version, based on the HostOS and HostApp versions used when the test was run and the baseline was created.
  • If no matching baseline exists, then this is a new test, and, by default, a new baseline will be created based on the checkpoint images captured by the test. The new baseline will be associated with the baseline attributes from the test and its execution environment.
  • If exactly one matching baseline is found, Eyes will use that baseline.
  • It more than one baseline is found, Eyes will choose the most appropriate baseline, as explained in the next section.

Choosing a baseline from multiple versions

Choosing a baseline based on HostApp and HostOS versions is a new Eyes feature currently in Beta testing. The description below is correct for the current version,but may change for the full release of this feature.

If Eyes finds multiple matching baselines, it will look for the most suitable baseline by comparing the HostApp and HostOS version information of the baseline and the execution environment. This version comparison is based on two criteria:

  • The baseline version must be compatible with the test version. For example, all the major number versions of Microsoft Internet Explorer (e.g. IE 9, IE 10, IE11) are considered to be incompatible with each other. But for a given major version, the minor versions are considered compatible. On the other hand, Eyes considers all versions of Chrome to be compatible. Compatibility is defined fully in the When are versions compatible? section below.
  • Versions have an inherent ordering. For example, Eyes will allow a test running on a newer version of a browser to run against a baseline created from an older version as long as it is compatible, but if the test is running on an older version of the browser, it won't use a baseline created from a test running on a new version.

Given a choice of several matching, compatible baselines, Eyes will, in general, choose a baseline with the same version as the execution environment. As a second option, Eyes will choose the latest version of a baseline as long as that version is previous (i.e. earlier) to the execution environment version. The following table defines the exact order of priority:

Priority HostApp version HostOS version
1 Same version Same version
2 Same version Latest previous version
3 Latest previous version Same version
4 Latest previous version Latest version (HostApp latest has priority)

If no baseline can be found with the same or earlier HostApp and HostOS versions, then the test is treated as a new test.

When are versions compatible?

The HostOS and HostApp compatibility rules are based on the way versions are defined in browsers. A version string is assumed to be constructed of one or more components separated by periods (e.g. 10.1.3). The leftmost component is called the major version number and must be numeric. The other components may be numeric or alphanumeric (i.e. a-z, A-Z, 0-9, or _). Version Format section below explains the expected format in more detail.

The following table defines compatibility for different values of HostOS and HostApp names:

Attribute Compatibility Example
HostOS = any value All versions are compatible Android 10.1 baseline is compatible with both Android 10.0 and Android 9.2 baselines.
HostApp = “Safari”, “Firefox”, “Edge”, “Chrome”, “Opera" All versions are compatible Chrome 81 baseline is compatible with any other version of Chrome.
HostApp = “IE” Versions are compatible if the major component matches IE 10 baseline is not compatible with IE 9 baseline. Any IE 10 sub version is compatible with any other IE 10 version.

HostApp = any other value

NOTE

Versions are compatible if the major component matches MyApp 1.1 is compatible with MyApp 1.2 but is not compatible with MyApp 2.2.

Running cross environment tests

A cross environment test is a test that runs against a baseline with an execution environment that is different from the test execution environment. This allows you, for example, to test your application on a number of different browsers against a single baseline that has been verified on another browser.

You run a cross browser test by calling the class$configuration$setbaselineenvname method to set the baseline environment name you want to test against. The HostOS, HostApp and Viewport size associated with this name is defined using the Eyes Test Manager. This process is described in detail in the article Running cross-environment tests.

When you run a cross environment test, the rules for selecting the appropriate version of the baseline are different from a test where the baseline is based on the test execution environment:

  1. If the test execution environment does not match the environment defined by the baseline environment name, then the latest compatible version of the OS and browser will be used. This version may be previous to, or later than the execution environment version as long as it is compatible.
  2. If the test execution environment matches the environment defined by the baseline environment name, then the baseline is chosen based on the version information of the test environment, as described in Choosing a baseline from multiple versions above.

Version Format

Eyes automatically extracts the version string from the execution environment information. If you define the HostOS or HostApp yourself (i.e. you use the methods configuration$sethostapp or configuration$sethostos) and you want Eyes to look for a fallback baseline, then you need to format the version string to meet the following guidelines:

  • The version string must appear after the name separated by exactly one space. The version number may not be followed by any non white space characters.
  • The version string must be a sequence of one or more components separated by a period (e.g. 3.0.23a.t3), where the leftmost component (the major component) must be numeric, and the other components can be numeric or alphanumeric (e.g. a-z,A-Z,0-9, or _).
  • When comparing two component strings to see which is the latest, the strings are compared component by component from the left most (major) component to the right. If both components are numeric, they are compared numerically, if they include non numeric characters, they are compared alphabetically. If the number of components is not identical (e.g. 1.0.0 and 1.0.0.A) but the component of the shorter version string matches that of the longer version string, the shorter version string is considered "older" (so 1.0.0 is considered older than 1.0.0.A).
  • If the version string does not match the above format, then it is considered part of the name. This means that a change in version information will create a new baseline, with no fallback to a previous compatible version.