Setting up tests with SIDE Eyes

This article summarizes all the setup features, those that are derived from Selenium IDE properties and those you can set via a SIDE Eyes menu or command. For a more detailed description click the option name in the paragraph header.

All the settings you set are persisted, the persistence mechanism depends on the setting type. This is explained in more details below in the section Configuration Options Persistence.

Set API key

To use Applitools Eyes, you need to obtain an API Key and set up the extension to use the key. When you first use SIDE Eyes, it prompts you for the key and stores it for future use. If necessary, for example, if you use multiple accounts, you can change it in the Global settings dialog Account tab.

eyes set Eyes server URI

If your organization runs Eyes on a dedicated server or cloud and not on the public cloud, then you need to provide the URI of the server before attempting to run any tests. You set this up in the Global settings dialog Account tab.

Enable/Disable visual checkpoints

You can enable or disable visual testing in your test run. This can be useful if your test includes both visual tests as well as semantic tests using Selenium IDE assert and verify command and in some circumstances, you only want to run the latter. You can disable visual testing from both the SIDE Eyes menu and the Global setting dialog Tests tab.

Set Applitools test and application name

The Eyes application and test names, along with the execution environment An execution environment is defined as a triplet of <OS, browser, viewport size>. When a test runs, its execution environment is defined by the platform on which the test runs, or, if run on the Ultrafast Grid, by the configuration of the Ultrafast Grid. A baseline is also associated with an execution environment. By default, the baseline chosen for a test is the baseline that has the same application name, test name, and execution environment as the test. Note that if the initial page in a test is not a secure web page, the execution environment may report the wrong OS or browser version., determine which baseline Defines the sequence of images to which the sequence of images captured at checkpoints will be compared. A test can have multiple baselines, each of which is characterized by the execution environment it ran on (operating system, browser and viewport size). A baseline can have instances in multiple branches. is used to provide reference images for the visual test. SIDE Eyes uses the Selenium IDE project and test name as the application and test names respectively.

Set baseline env name

When a test runs, the execution environment it runs on is defined by three properties - the operating system it ran (OS), the application platform (typically the browser) and the viewport size. Together with the application and test names, these 5 properties identify the specific baseline Defines the sequence of images to which the sequence of images captured at checkpoints will be compared. A test can have multiple baselines, each of which is characterized by the execution environment it ran on (operating system, browser and viewport size). A baseline can have instances in multiple branches. used as a reference when matching checkpoint images to baseline images.

Eyes Test Manager allows you to assign a name to a specific execution environment, and then in the test to set the baseline environment name to this name before the first call to eyes check. Eyes then uses the baseline defined by the environment name instead of the actual test execution environment. For more information on using execution environments see Running cross-environment tests.

Set Applitools batch name

The batch Two or more test runs can be grouped together as a "batch" using the SDK set batch method/property. In the Eyes Test Manager Test result page, you can see a list of all the batches that have run. You can select the batch and can then see the tests that ran in that batch. Eyes Test Manager provides has various features that allow you to operate on a subset of the batch tests or steps as a whole. is an Eyes term that refers to a collection of test run results. When Selenium IDE runs a test suite, it runs them as a batch, and the name of the batch is the same as the name of the Selenium IDE test suite. When you run a single test the batch name is the name of the test. When you run all of the tests in the project the name of the batch is All tests.

Baseline branches

Eyes supports having multiple instances of a baseline in different branches A branch is a container of baselines that typically represent the expected results of one version of an application. A branch can be created from a parent branch, so that initially the baselines are identical to the parent, meaning that the expected restuls are the same. As the application in a version changes, the baseline in the branch is updated to represent the changed expectations. A branch can be merged into a "target branch", the resulting branch consists of baselines from both branches, and therefore represents expected results signified by the baselines from both branches.. Typically these correspond to different development versions of your application or site. You can specify the baseline branch and its parent branch for the project in the appropriate fields in the SIDE Eyes menu.

Ultrafast Grid setup

The Applitools Ultrafast Grid is a feature that allows you to render your visual test images in the cloud. You can request that every checkpoint in your test is rendered for one or more browsers and simulated devices, and on multiple viewport variations. You use the SIDE Eyes menu to set up whether Ultrafast Grid should be used, and if so, which combination of browsers, viewports, and devices are captured.

Enable Root cause analysis

Applitools Root cause analysis is a feature in the Eyes Test Manager that uses DOM information sent during a test to find the root cause of a visual mismatch in terms of HTML or CSS differences between a checkpoint and the baseline. Sending the DOM is on by default, you can turn it off if necessary in the Global settings dialog Tests tab.

Open the Test Manager after test run

Often, when a test or test suite run completes, you want to review the results in the Test Manager immediately. If you enable this option, then when the test run completes, a browser is opened displaying the Eyes Test Manager loaded with the test run that completed. You can enable or disable this feature in the Global settings dialog Tests tab.

eyes set match timeout

You can use the eyes set match timeout command if page rendering takes more time than is allowed for by default, and as a result, an image is captured before the render is complete. Increasing the timeout is sometimes necessary when the page content includes an animation when it starts, or if it is generated by a lengthy JavaScript program.

eyes set viewport size

The viewport size describes the rectangular area in the browser window that contains application content. Since the viewport size can impact the page layout, it is one of the properties that Eyes uses to determine which baseline reference images it uses for the test. In most cases, when you set the viewport size, you are doing two things - changing the browser window size, and determining which baseline will be used.

eyes set match level

The match level determines the sensitivity of the matching algorithm used by Eyes. For example, a match level of Strict checks that both text and graphics are visually identical. A match level of Layout is more lenient, checking that graphic areas and image areas are positioned similarly in the checkpoint and baseline image, but allowing the actual text or image to be different. You can change this property during the test, so that different checkpoints, can be tested with different match levels.

Set IDE extension ID

If you obtain a copy of the Selenium IDE Extension from a source other than the web store (this is necessary for example if your computer is behind a firewall that blocks access to the store) then you may need to set this ID to the Selenium IDE browser extension ID so that SIDE Eyes can interact correctly with Selenium IDE. You set this value in the Global settings dialog Advanced tab.

Configuration Options Persistence

All the options you enter into Selenium IDE are persisted, and you only need to re-enter them if they change. The type of persistence depends on the option:

• All setup done via Selenium IDE commands is stored in the project file (i.e., the .side file).
• The global settings are stored in the browser local storage, and apply to all projects.
• The project specific option in the SIDE Eyes menu are stored in local browser storage, but are project specific, so, for example, you can define the Eyes branch and Ultrafast Grid information on a per-project basis.

If you use different types of browsers, for example, you use both Chrome and Firefox, then you need to set up options in each of the browsers independently.