Batching tests in a distributed environment

Using an Eyes batch allows you to see and manage the results of multiple tests as a single unit. In the simplest case, a batch can be created when tests are run together in a single program execution. How to do this is explained in the article How to organize your tests with batches.

This article, explains how to batch results of tests that are executed in different processes or on different machines. This technique is useful if, for example, your tests are coded in different programming languages, your tests need to run on multiple target platforms (e.g. Linux, Windows, Mac, and mobile devices), or you run tests concurrently on multiple CPUs or distributed processors.

To coordinate multiple tests so that they appear as a single batch you need to implement the functionality below, these are explained in detailed in the following sections:

Using the techniques described below means that the number of tests in a batch is not limited to a single test run. Reuse of an ID for unrelated tests can cause an unnecessary increase in the number of tests in a batch. This has no functional advantage since the tests are not related, but it may cause severe performance issues and/or failures and so should be avoided.

Associate tests with a common batch

When tests are run in the same process, they are batched by assigning a shared batchinfo object to the Eyes instance using the class$eyes$setbatch$morp, or by using a shared configuration object which has been configured with a batchinfo object. What the tests are actually sharing is an ID stored in the common batchinfo object that Eyes sets to a unique value when the object is created.

When tests are run in separate runs, sharing the ID in this way is not possible, so instead, each test run creates its own batchinfo object, and the tests are batched by explicitly assigning the batchinfo object a common Batch ID.

You can do this by taking the following steps:

  1. Generate a unique ID string. You can do this in any way that is convenient (e.g. using a date and time stamp and some machine ID). Since batches are stored on the server for a long time, you need to ensure that the ID will be unique in the long term.
  2. Distribute the unique ID to all your test programs (e.g. by writing it to a network file or a database).
  3. Finally, each test program needs to obtain the ID and assign it to its batchinfo object. This can be done in two ways:
    • By setting the environment variable APPLITOOLS_BATCH_ID to the unique ID before starting the test process. The SDK will look for, and use, the value of this environment variable if the batch ID is not set explicitly.
    • By obtaining the batch ID in your code and setting the batchinfo object explicitly as illustrated below:

The call to the function myGetUniqueBatchID() represents the code you need to add that obtains the ID from wherever it is stored.

It is best practice to use the same batch name for all the tests. However, even if you don't, it is the batch ID that binds the tests together. The batch name displayed in the Test Manager will reflect the batch name of the first test that is run.

Close the shared batch

When you are running tests in a single batch and the tests are run in separate processes, you need to take steps to ensure that the batch is only closed when all the test runs are complete. This consists of two steps:

  • Prevent closing the batch when each test run completes.
  • Signal the Eyes Server when all the tests have completed.

How to achieve this is described below using a Bash shell script. You can do something similar in the scripting language or CI system of your choice.

  • Before starting a process that runs tests that are part of a multi-process batch, set an environment variables APPLITOOLS_DONT_CLOSE_BATCHES to the value "true" as shown in the example below. The example also illustrates setting the the APPLITOOLS_BATCH_ID environment variable to the batch ID as described in the previous section, and if required, setting the APPLITOOLS_BATCH_NOTIFY environment variable to instruct Eyes to send a notification when the batch completes - see Batch completion notifications for details.APPLITOOLS_BATCH_NOTIFY.
    export APPLITOOLS_DONT_CLOSE_BATCHES="true"
    export APPLITOOLS_BATCH_ID=$(echo `uuidgen`)		
    export APPLITOOLS_BATCH_NOTIFY="true"
    
  • After all the tests have completed, inform Eyes by calling the server. On Linux, for example, you can do this using the curl program as follows:
    
    set server="eyesapi.applitools.com"
    set apikey=$APPLITOOLS_API_KEY
    set batchid=$APPLITOOLS_BATCH_ID						
    set url="https://$server/api/sessions/batches/$batchid/close/bypointerid/?apiKey=$apikey
    
    curl --request DELETE --dump-header - $url
    

    where the variables server, apikey and batchid have the appropriate values for your environment.