How to group tests into batches using the SDK

Eyes organizes the test results into a two-level hierarchy consisting of a list of batches, where each batch has a set of associated tests. Eyes provides the Batch mechanism to allow you to group your test runs in a way that is convenient to your workflow and organization. By default, every test runs in its own batch.

This article explains how you can use the Eyes SDK to group the results of multiple test runs into a single batch so that they are displayed, summarized and can be managed together in the Test Manager.

The standard way to batch tests together is to run them together in a single test suite. The SDK allows you to organize the tests into multiple batches, and you can use this for example to reflect the functional structure of your application. In some circumstances, it is not possible to run all the tests in the same run (technically in the same operating system process), for example, tests may be written in different languages, there may be a need to run on multiple platforms, or the test may be distributed for performance reasons. Eyes also provides you a way to batch together related tests even if they run separately, this is explained in the article Batching Tests in a distributed environment.

.

The Batch Name

When you create a batch, you provide a string that represents the batch name. The string can have any value, for example, many Eyes customers use the Test Suite name. If you don’t explicitly create a batch, the batch name of a test is the test name. Test Manager will normally display the first 29 characters of the name so that most of the screen width is available for the current batch test information displayed in the right panel in the window. If you use names that are longer than this, then Test Manager provides you with a way to drag the batch list panel to the right to increase its width. It will show the expanded version of this panel when your cursor hovers over the batch list.

Assigning a Batch for tests running in a single program

If all your tests run in a single program and process, then the section below describes a convenient way to combine them into one batch. If your tests run as separate programs, and you want the results to be collected as a batch the this article explains ow to do this Batching Tests in a distributed environment

  1. Create an instance of the class BatchInfo
  2. Associate the batch instance with the Eyes instance
  3. Thereafter, any test that is created using that Eyes instance (by calling eyes.open()) will be associated with that batch.
  4. You can create and set a 2nd BatchInfo object on the same eye instance. Tests already open on that eyes instance will remain in their existing batch, and newly open()ed tests will belong to the new batch. Once a test has been associated with a batch, the association cannot be changed.

If you want to run tests on multiple eyes instances (for example in order to run them on concurrent threads), you simply call setBatch() on each eyes instance using the same batchInfo object. For example, in the following test two tests are run, twice each on diffedrent browsers and viewport sizes and all the tests are are associated with the same batch:

When the Test programmer shares a single BatchInfo object with multiple tests, every single run of the test suite will create a new batch. You will see multiple entries in the Test Manager, one for each run of the test suite. All of these will have the same batch name, but each will have an independent run timestamp. The set of tests in each batch entry will never change.

In the screenshot below you can see in the batch list on the right two entries generated by two runs of the test whose code snippets are shown above.On the right you see the runs from the second run, you can see that there are two test names "test 1 and "test 2" and the runs were on two types of browsers and with two different viewport sizes.