Baseline management

A baseline stores reference images for Eyes visual tests. When you run a test, Eyes captures checkpoints and compares them to the baseline images to identify visual differences.

Baseline images depend on the test and its execution environment, including the operating system, browser type, and viewport size. Eyes maintains multiple baselines for each test, each tailored to a specific environment.

The life cycle of a baseline

When you run a test in a new environment, Eyes creates new baseline images, storing the captured images and test information.

For subsequent runs in the same environment, Eyes uses the existing baseline images to compare to the captured images.

If you make changes to a baseline using the Test Manager such as accepting a new or changed step, a new baseline is created. This baseline will be used as a reference in future tests.

Test and execution environment attributes

Eyes uses the following test and execution environment attributes to identify a baseline:

• testName and appName – User-defined strings that describe the tested application and specific test. Together, they uniquely identify the test.

• Viewport size – For web applications, this is the dimensions of the browser pane. For mobile devices, it is the screen dimensions.

• Host environment – This includes HostOS (the operating system) and HostApp (the application platform, such as the browser for web applications).

• Version information – Both HostOS and HostApp can have version information, which affects how baselines are managed.

Where does Eyes get the test baseline attributes from?

Some of the test attributes are derived by Eyes when you start a test by calling eyes.open:

eyes.open(driver, appName, testName, viewportSize)

• The test is identified by the strings appName and testName.

• When the images are being captured on the test browser (as opposed to a test that uses the Ultrafast Grid):

  • The viewport size attribute is defined by the viewportSize parameter.

  • The HostApp attribute is derived from the test browser name and version (i.e., from the driver parameter).

  • The HostOS attribute is derived from the operating system the browser runs on.

• When the test uses the Ultrafast Grid, the execution environment attributes are derived from the Ultrafast Grid parameters passed to the Ultrafast Grid target configuration methods:

• Java
• JavaScript
• C#
• Ruby

suiteConfig = (Configuration) new Configuration()
        // Visual Grid configurations
        .addBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.CHROME))
        .addBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.CHROME_ONE_VERSION_BACK))
        .addBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.CHROME_TWO_VERSIONS_BACK))
        .addBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.FIREFOX))
        .addBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.SAFARI))
        .addBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.IE_10))
        .addBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.IE_11))
        .addBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.EDGE_CHROMIUM))
        .addBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.EDGE_LEGACY))
        .addBrowser(new IosDeviceInfo(IosDeviceName.iPhone_X, ScreenOrientation.LANDSCAPE))
        .addBrowser(new ChromeEmulationInfo(DeviceName.Galaxy_S5, ScreenOrientation.PORTRAIT))

suiteConfig = new Configuration()
   // Visual Grid configurations
     .add_browser({width:viewPortWidth, height:viewPortHeight, name:BrowserType.CHROME})
     .add_browser({width:viewPortWidth, height:viewPortHeight, name:BrowserType.CHROME_ONE_VERSION_BACK})
     .add_browser({width:viewPortWidth, height:viewPortHeight, name:BrowserType.CHROME_TWO_VERSIONS_BACK})
     .add_browser({width:viewPortWidth, height:viewPortHeight, name:BrowserType.FIREFOX})
     .add_browser({width:viewPortWidth, height:viewPortHeight, name:BrowserType.SAFARI})
     .add_browser({width:viewPortWidth, height:viewPortHeight, name:BrowserType.IE_10})
     .add_browser({width:viewPortWidth, height:viewPortHeight, name:BrowserType.IE_11})
     .add_browser({width:viewPortWidth, height:viewPortHeight, name:BrowserType.EDGE_CHROMIUM})
     .add_browser({width:viewPortWidth, height:viewPortHeight, name:BrowserType.EDGE_LEGACY})
     .add_browser({iosDeviceInfo: {deviceName:IosDeviceName.iPhone_X, screenOrientation:ScreenOrientation.LANDSCAPE}})
     .add_browser({chromeEmulationInfo:{ deviceName : DeviceName.Galaxy_S5, screenOrientation:ScreenOrientation.PORTRAIT}})

suiteConfig = new Configuration() 
    // Visual Grid configurations
    .AddBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.CHROME)) 
    .AddBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.CHROME_ONE_VERSION_BACK))
    .AddBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.CHROME_TWO_VERSIONS_BACK))
    .AddBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.FIREFOX))
    .AddBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.SAFARI))
    .AddBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.IE_10))
    .AddBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.IE_11))
    .AddBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.EDGE_CHROMIUM))
    .AddBrowser(new DesktopBrowserInfo(viewPortWidth, viewPortHeight, BrowserType.EDGE_LEGACY))
    .AddBrowser(new IosDeviceInfo(IosDeviceName.iPhone_X, ScreenOrientation.Landscape))
    .AddBrowser(new ChromeEmulationInfo(DeviceName.Galaxy_S5, ScreenOrientation.Portrait))

suite_config = Applitools::Selenium::Configuration.new.tap do |c| 
      # Visual Grid configurations
      c.add_browser(viewport_width, viewport_height, BrowserTypes::CHROME)
      c.add_browser(viewport_width, viewport_height, BrowserTypes::CHROME_ONE_VERSION_BACK)
      c.add_browser(viewport_width, viewport_height, BrowserTypes::CHROME_TWO_VERSIONS_BACK)
      c.add_browser(viewport_width, viewport_height, BrowserTypes::FIREFOX)
      c.add_browser(viewport_width, viewport_height, BrowserTypes::SAFARI)
      c.add_browser(viewport_width, viewport_height, BrowserTypes::IE_10)
      c.add_browser(viewport_width, viewport_height, BrowserTypes::IE_11)
      c.add_browser(viewport_width, viewport_height, BrowserTypes::EDGE_CHROMIUM)
      c.add_browser(viewport_width, viewport_height, BrowserTypes::EDGE_LEGACY)
      c.add_browser(Applitools::Selenium::IosDeviceInfo.new(
        device_name: IosDeviceName::IPhone_X, screen_orientation: Orientation::LANDSCAPE)
      c.add_browser(Applitools::Selenium::ChromeEmulationInfo.new(
        device_name : DEVICES.GalaxyS5, orientation : Orientation::PORTRAIT))
    end  

In this case:

• The Viewport size is based on the viewPortWidth and viewPortHeight parameters

• The Host is based on BrowserType, DeviceName, or IosDeviceName values passed

• the HostOS is mapped by Eyes internally to the OS used when rendering for that particular browser or device.

• The setHostApp and setHostOS methods allow you to define the execution environment explicitly.

How Eyes selects an appropriate baseline (Baseline fallback)

Eyes has two baseline management policies which differ based on the following considerations:

• Cases where Eyes supports multiple, independent baselines for different HostOS (such as Windows 10) or HostApp (such as Chrome 127.0) versions.

• Cases where Eyes uses an existing baseline even if the HostOS or HostApp version differs from the test environment.

• Cases where a new baseline can inherit checkpoints from a related baseline if no compatible one exists.

Based on these considerations, Eyes will use one of the following baseline policies:

• Standard policy – considers the HostOS version as part of the baseline signature and does not support baseline inheritance.

• Advanced policy – more flexible, allowing cross-version testing and baseline inheritance.

Standard baseline management policy

This policy requires matching HostOS major versions but ignores HostApp (browser) versions. For example, Windows 10 tests can't use a Windows 7 baseline, but Chrome 80 tests can use a Chrome 72 baseline. This approach assumes browser versions are generally backward compatible, while major HostOS versions are not.

If a new OS version, such as macOS 11 with Safari 14, is detected, a new baseline is created. If you want to maintain baseline inheritance, you should use the advanced baseline management policy, which can inherit baselines across major OS version changes.

Advanced version baseline management policy

Eyes selects a baseline based on:

• Finding one or more matching baselines.

• Selecting compatible baselines.

• Choosing the most appropriate baseline.

Matching baselines have identical HostOS and HostApp names and Viewport sizes. The policy selects the baseline with the most appropriate versions for the test environment.

Finding a matching baseline

When you start a test, Eyes searches to see if a baseline exists for your test:

• Eyes looks for matching baselines with the same Test name, App name, Viewport size, HostApp, and HostOS names.

• If no matching baseline exists, a new one is created using the test's checkpoint images.

• If exactly one matching baseline is found, Eyes uses it.

• If multiple baselines are found, Eyes selects the most appropriate one.

Choosing a baseline from multiple versions

If multiple matching baselines are found, Eyes selects the most suitable one by comparing the HostApp and HostOS versions. Two criteria are used:

• The baseline version must be compatible with the test version. Major IE versions (e.g., IE 9, IE 10) are incompatible, while all Chrome versions are compatible. For further information, see When are versions compatible?.

• Versions have an inherent ordering. A test can use a baseline from an older version but not a newer one.

Given several compatible baselines, Eyes usually chooses one matching the test's execution environment or the latest previous version.

If no baseline can be found with the same or earlier HostApp and HostOS versions, the test is considered new.

When are versions compatible?

HostOS and HostApp compatibility depends on versioning. A version string has one or more components separated by periods (e.g., 10.1.3). The major version number is numeric; others can be alphanumeric. For details, see Version format.

Running cross environment tests

Cross environment tests use baselines with different execution environments. For example, you can test your application on different browsers against a verified baseline on another browser.

Use setBaselineEnvName to set the baseline environment name. The HostOS, HostApp, and viewport size are defined using Test Manager. For further information, see Running cross-environment tests.

The baseline selection rules differ for cross-environment tests:

1. If the test execution environment does not match the baseline environment name, the latest compatible OS and browser version is used.

2. If the test execution environment matches the baseline environment name, the baseline is chosen based on the test environment version, as described in Choosing a baseline from multiple versions.

Version format

Eyes extracts the version string from the execution environment information. If you define the HostOS or HostApp yourself using setHostApp or setHostOS and want Eyes to use a fallback baseline, format the version string as follows:

• The version string must appear after the name, separated by one space.

• The version string must consist of one or more parts separated by periods. The major component) must be numeric, and the other components can be numeric or alphanumeric. For example, a-z,A-Z,0-9, or _.

• When determining which version is older, if the version only contains numbers they are assumed to be numeric. If the version includes non-numeric characters, they are compared alphabetically. If a version is identical to a different version, but has an extra part at the end, the shorter version is considered older. For example, 1.0.0 is considered older than 1.0.0.A

• If the version string does not match the above format, it is considered part of the name. This means that a change in version information will create a new baseline, with no fallback to a previous compatible version.

For further information, see our video, Validating Baselines in Applitools Eyes and Creating Multi-Basleine Variations: