How to group tests into batches using the SDK
Eyes organizes the test results into a two-level hierarchy consisting of a list of batches, where each batch has a set of associated tests. Eyes provides the Batch mechanism to allow you to group your test runs in a way that is convenient to your workflow and organization. By default, every test runs in its own batch.
This article explains how you can use the Eyes SDK to group the results of multiple test runs into a single batch so that they are displayed, summarized and managed together in the Test Manager.
The simplest way to batch tests together is to run them together in a single test suite. The SDK allows you to organize the tests into multiple batches, and you can use this for example to reflect the functional structure of your application. In some circumstances, it is not possible to run all the tests in the same run (technically in the same operating system process), for example, tests may be written in different languages, there may be a need to run on multiple platforms, or the test may be distributed for performance reasons. Eyes provides a way to batch together related tests even if they run separately, this is explained in the article Batching Tests in a distributed environment..
The Batch Name
When you create a batch, you provide a string that represents the batch name. The string can have any value, for example, many Eyes customers use the Test Suite name. If you don’t explicitly create a batch, the batch name of a test is the test name. Test Manager will normally display the first 29 characters of the name so that most of the screen width is available for the current batch test information displayed in the right panel in the window. If you use names that are longer than this, then Test Manager provides you with a way to drag the batch list panel to the right to increase its width. It will show the expanded version of this panel when your cursor hovers over the batch list.
Assigning a Batch for tests running in a single program
If all your tests run in a single program and process, then the section below describes a convenient way to combine them into one batch. If your tests run as separate programs, and you want the results to be collected as a batch then this is described in the article Batching tests in a distributed environment.
- Create an instance of the class BatchInfo
- Associate the batch instance with the Eyes instance
- Thereafter, any test that is created using that Eyes instance (by calling
eyes.open()) will be associated with that batch.
- You can create and set a 2nd BatchInfo object on the same eye instance. Tests already open on that eyes instance will remain in their existing batch, and newly open()ed tests will belong to the new batch. Once a test has been associated with a batch, the association cannot be changed.
If you want to run tests on multiple eyes instances (for example in order to run them on concurrent threads), you simply call
setBatch() on each eyes instance using the same batchInfo object. For example, in the following test two tests are run, twice each on different browsers and viewport sizes and all the tests are associated with the same batch:
When the Test programmer shares a single BatchInfo object with multiple tests, every single run of the test suite will create a new batch. You will see multiple entries in the Test Manager, one for each run of the test suite. All of these will have the same batch name, but each will have an independent run time-stamp. The set of tests in each batch entry will never change.
In the screenshot below you can see in the batch list on the right two entries generated by two runs of the test whose code snippets are shown above.On the right you see the runs from the second run, you can see that there are two test names "test 1 and "test 2" and the runs were on two types of browsers and with two different viewport sizes.
Batch completion termination
As explained in the article Batch completion notificationsyou can set up Eyes using the Eyes Test Manager so that when a batch completes and it includes tests for a defined application users will get an email or a Slack notification. In order for this to happen, you need to enable this explicitly when the batch is created, for more information see Setting up batch completion notifications in the SDK.