Batching tests in a distributed environment

Grouping tests into batches, allows you to see and manage the results of multiple related tests together in the Test Manager. The article Grouping tests into batches with the SDK explains how to batch tests when they run in the same test execution.

This article, explains how to batch results of tests that are executed in different processes or on different machines. This technique is useful if, for example, your tests are coded in different programming languages, your tests need to run on multiple target platforms (e.g. Linux, Windows, Mac, and mobile devices), or you run tests concurrently on multiple CPUs or distributed processors. The techniques described here are also useful when you run a single process test using multiple runners.

To batch together tests that run in separate processes or separate runner, you need to do three things:

Each of these is described in the sections below.

Avoid batching together unrelated tests unnecessarily, since this may increase the number of tests in a batch to a large number and can cause severe performance issues and/or failures.

Prevent auto-closure of batches by runners

If you use a classicrunner or visualgridrunner in your tests then Implicit batching by a runner, and by default, the runner closes the batch(es) in the call to class$visualgridrunner$getalltestresults after the tests have completed.

If you are using multiple executions, or more than one runner in a single execution, and you want the tests to be in a common batch then you have to disable the auto closing of batches. There are two ways to do this:

Associate tests with a common batch

When you run test in a single process, you can batch them by assigning a shared batchinfo object to every Eyes instance using one of the methods described in How to group tests into batches

When tests are run in separate runs, sharing a batchinfo object is not possible. Instead, each test creates its own batchinfo object, and the tests are batched by explicitly assigning the batchinfo object a common Batch ID.

It is best practice to use the same batch name for all the tests. However, even if you use different names, it is the common batch ID that groups the tests together. The batch name displayed in the Test Manager is the batch name of the first test that runs.

Typically sharing a batch id is done using the following steps:

  1. Generate a unique ID string, for example by using date and time stamp and some machine ID. Since batches are stored on the server for a long time, you need to ensure that the ID will be unique in the long term.
  2. Distribute the unique ID to all your test programs (e.g. by writing it to a network file or a database).
  3. In each test program, obtain the shared ID and assign it to its batchinfo object. This can be done in two ways:
    • By obtaining the ID in a script and setting the environment variable APPLITOOLS_BATCH_ID to the unique ID before starting the test process. The SDK will use this value of this environment variable if it is defined and if the batch ID is not set explicitly.
    • By obtaining the batch ID in your code and setting the batchinfo object explicitly as illustrated below:

The call to the function myGetUniqueBatchID() represents the code you need to add that obtains the ID from wherever it is stored.

Close the shared batch

Closing a batch means that:

You can close a batch by using the batchclose class in an SDK that supports it (e.g. Java) or you can send an HTTP request directly to the Eyes server using a program such as curl.

Closing a batch with the BatchClose class

The following snippet shows how to close a batch using the batchclose class. You can do this in any program, not necessarily a program that runs an Eyes test.


                                

Note that before calling the batchclose$close method, you need to pass one or more batch ids to the method batchclose$setbatchid. In this example we use the value of the environment variable APPLITOOLS_BATCH_ID, typically you would use the same method you used to assign the ID when defining the batch as described in the previous section Associate tests with a common batch.

Sending the server an HTTP requests directly

For SDKs that don't support the batchclose class, you can close the batch using curl as illustrated in the following Bash script:


set eyesserver="eyesapi.applitools.com"
set apikey=$APPLITOOLS_API_KEY
set batchid=$APPLITOOLS_BATCH_ID						
set url="https://$eyesserver/api/sessions/batches/$batchid/close/bypointerid/?apiKey=$apikey

curl --request POST --dump-header - $url

The script assumes that the variables server, apikey and batchid have the appropriate values for your environment.