Grouping tests into batches with the SDK
In the Test Manager, test results are grouped into batches. By grouping related tests into batches you can view, process and manage the test results as a unit. For a broad view of all the features associated with batches see the article How to organize your tests with batches. This article focuses on how to use the SDK to assign tests to specific batches.
Before diving into the details of how to assign tests to batches, it is useful to understand some basic concepts.
Default and Explicit Batching
Test results are always associated with a batch, the batch may be the default batch created by the SDK or a batch that you create, name and assign to tests explicitly. The default batching scheme depends on which SDK you use:
In SDKs that do not implement runners (such as the Image SDKs), by default each test runs as a separate batch created by the SDK.
In most cases, it is advisable to use explicit batching in order to give the batch a descriptive name. Other things that you can achieve by using explicit batching are:
Batch tests from multiple runners into a single batch.
Batch a number of related tests into a single batch when using a non runner SDK.
Batch a number of related tests that are running in separate test executions.
Batch name and Batch ids
It is important to understand the difference between the batch name and the batch id.
The batch name is the user visible name displayed in the list of batches in the Test Manager. The batch name is not unique, every time you run a batch it will have the same name. It is possible to give two distinct batches (i.e. batches that include different tests) the same name, although in most cases this is not recommended practice.
The batch id is a unique identifier of a specific batch run. If you run a batch of tests a number of times they will all have the same name, but each will have a unique id. By default, the SDK assigns each batch a unique ID. The SDK provides you with a way to associate a specific batch ID with a batch, and this allows you to batch tests that run in different processes or on different hardware.
You can assign tests to a common batch in any of the following ways:
Each of these methods is described in detail in the sections that follow.
By default, all tests that are run using an SDK that supports the
VisualGridRunner are batched into a single batch whose name is the name of the first test to run. You can use the methods described below to give an appropriate name to the batch and to implement any batching scheme you want, for example, to separate unrelated tests run by a runner into different batches, or to combine related tests from multiple runners into a single batch.
The methods describe below use a shared
BatchInfo object to assign tests to a batch represented by that object. This is convenient when the tests run in the same process. Two methods are described - using a shared Configuration object and directly assigning each test to a batch.
Explicit batching using a Configuration object
Note that the call to
Eyes.setconfiguration should be made before any other test specific configuration, since it assigns all the Configuration properties - either to the values set explicitly or to the default value.
Explicit batching the Eyes object
If you previously set a Configuration object on this Eyes instance then the call to
eyes.setbatch will override it.
If the tests are running on different processes, or on different machines, then instead of sharing a
BatchInfo instance, each process creates a
BatchInfo object, and initializes it with a shared, common batch Id. The snippet below shows the basic technique. The ID can be shared in any convenient way, symbolized here by the call to the method getSharedId which you need to implement to obtain the ID using whatever method you choose.
For more information about batching tests that are run in separate processes see Batching tests in a distributed environment
Batch completion termination
As explained in the article Batch completion notifications you can set up Eyes using the Eyes Test Manager so that when a batch completes and it includes tests for a defined application users will get an email or a Slack notification. In order for this to happen, you need to enable this feature when the
BatchInfo object is created, for more information see Setting up batch completion notifications in the SDK.