Batching tests in a distributed environment
Grouping tests into batches, allows you to see and manage the results of multiple related tests together in the Test Manager. The article Grouping tests into batches with the SDK explains how to batch tests when they run in the same test execution.
This article, explains how to batch results of tests that are executed in different processes or on different machines. This technique is useful if, for example, your tests are coded in different programming languages, your tests need to run on multiple target platforms (e.g. Linux, Windows, Mac, and mobile devices), or you run tests concurrently on multiple CPUs or distributed processors. The techniques described here are also useful when you run a single process test using multiple runners.
To batch together tests that run in separate processes or separate runner, you need to do three things:
- Prevent auto-closure of batches by runners
- Associate tests with a common batch
- Close the shared batch
Each of these is described in the sections below.
Avoid batching together unrelated tests unnecessarily, since this may increase the number of tests in a batch to a large number and can cause severe performance issues and/or failures.
If you use a
visualgridrunner in your tests then Implicit batching by a runner, and by default, the runner closes the batch(es) in the call to
class$visualgridrunner$getalltestresults after the tests have completed.
If you are using multiple executions, or more than one runner in a single execution, and you want the tests to be in a common batch then you have to disable the auto closing of batches. There are two ways to do this:
By calling the method
class$visualgridrunner$setdontclosebatcheson every runner immediately after it is created, as shown below:
By defining the environment variable APPLITOOLS_DONT_CLOSE_BATCHES before the test runs.
For convenience we referred to methods from the
visualgridrunnerclass, the same methods are available when using the
When you run test in a single process, you can batch them by assigning a shared
batchinfo object to every Eyes instance using one of the methods described in How to group tests into batches
When tests are run in separate runs, sharing a
batchinfo object is not possible. Instead, each test creates its own
batchinfo object, and the tests are batched by explicitly assigning the
batchinfo object a common Batch ID.
It is best practice to use the same batch name for all the tests. However, even if you use different names, it is the common batch ID that groups the tests together. The batch name displayed in the Test Manager is the batch name of the first test that runs.
Typically sharing a batch id is done using the following steps:
- Generate a unique ID string, for example by using date and time stamp and some machine ID. Since batches are stored on the server for a long time, you need to ensure that the ID will be unique in the long term.
- Distribute the unique ID to all your test programs (e.g. by writing it to a network file or a database).
- In each test program, obtain the shared ID and assign it to its
batchinfoobject. This can be done in two ways:
- By obtaining the ID in a script and setting the environment variable
APPLITOOLS_BATCH_IDto the unique ID before starting the test process. The SDK will use this value of this environment variable if it is defined and if the batch ID is not set explicitly.
- By obtaining the batch ID in your code and setting the
batchinfoobject explicitly as illustrated below:
The call to the function
myGetUniqueBatchID() represents the code you need to add that obtains the ID from wherever it is stored.
Closing a batch means that:
In the Test Manager, the status of the batch will change from Running, Test and batch status of the tests in the batch.
If Batch completion notification has been enabled, a notification will be sent.
You can close a batch by using the
batchclose class in an SDK that supports it (e.g. Java) or you can send an HTTP request directly to the Eyes server using a program such as curl.
Closing a batch with the BatchClose class
The following snippet shows how to close a batch using the
batchclose class. You can do this in any program, not necessarily a program that runs an Eyes test.
Note that before calling the
batchclose$close method, you need to pass one or more batch ids to the method
batchclose$setbatchid. In this example we use the value of the environment variable APPLITOOLS_BATCH_ID, typically you would use the same method you used to assign the ID when defining the batch as described in the previous section Associate tests with a common batch.
Sending the server an HTTP requests directly
For SDKs that don't support the
batchclose class, you can close the batch using
curl as illustrated in the following Bash script:
curl -v -X DELETE "$APPLITOOLS_SERVER_URL/api/sessions/batches/$APPLITOOLS_BATCH_ID/close/bypointerid?apiKey=$APPLITOOLS_API_KEY"
curl -v -X DELETE "%APPLITOOLS_SERVER_URL%/api/sessions/batches/%APPLITOOLS_BATCH_ID%/close/bypointerid?apiKey=%APPLITOOLS_API_KEY%"
The script assumes that the variables
batchid have the appropriate values for your environment.