Batching tests in a distributed environment

Using an Eyes batch allows you to see and manage the results of multiple tests as a single unit. In the simplest case, a batch can be created when tests are run together in a single program execution. How to do this is explained in the article How to organize your tests with batches.

This article, explains how to batch results of tests that are executed in different processes or on different machines. This technique is useful if, for example, your tests are coded in different programming languages, your tests need to run on multiple target platforms (e.g. Linux, Windows, Mac, and mobile devices), or you run tests concurrently on multiple CPUs or distributed processors.

Using the technique described below means that the number of tests in a batch is not limited to a single test run. Reuse of an ID for unrelated tests can cause an unnecessary increase in the number of tests in a batch. This has no functional advantage since the tests are not related, but it may cause severe performance issues and/or failures and so should be avoided.

Batching tests in multiple test runs

When tests are run in the same process, they are batched by assigning a shared batchinfo object to the Eyes instance using the class$eyes$setbatch$morp, or by using a shared configuration object which has been configured with a batchinfo object. What the tests are actually sharing is an ID stored in the common batchinfo object that Eyes sets to a unique value when the object is created.

When tests are run in separate runs, sharing the ID in this way is not possible, so instead, each test run creates its own batchinfo object, and the tests are batched by explicitly assigning the batchinfo object a common Batch ID.

You can do this by taking the following steps:

  1. Generate a unique ID string. You can do this in any way that is convenient (e.g. using a date and time stamp and some machine ID). Since batches are stored on the server for a long time, you need to ensure that the ID will be unique in the long term.
  2. Distribute the unique ID to all your test programs (e.g. by writing it to a network file or a database).
  3. Finally, each test program needs to obtain the ID and assign it to its batchinfo object. This can be done in two ways:
    • By setting the environment variable APPLITOOLS_BATCH_ID to the unique ID before starting the test process. The SDK will look for and use the value of this environment variable if the batch ID is not set explicitly.
    • By obtaining the batch ID in your code and setting the batchinfo object explicitly as illustrated below:

The call to the function myGetUniqueBatchID() represents the code you need to add that obtains the ID from wherever it is stored.

It is best practice to use the same batch name for all the tests. However, even if you don't, it is the batch ID that binds the tests together. The batch name displayed in the Test Manager will reflect the batch name of the first test that is run.

Batch completion notifications

The Eyes Batch completion notifications feature allows you to request that notifications are sent to one or more people when the execution of a batch of tests has completed. The notifications are sent using Slack and/or email, so anyone who is interested, and authorized to receive updates on the build status and quality of a batch of tests has access to this information even if they don't have access to Eyes. For a full description of this feature see Batch completion notifications.

When you are running tests in a single batch and the tests are run in separate processes, you need to inform Eyes so that batch completion notifications are not sent until all the processes have completed. To do this, there are two steps:

  • Before starting a process that runs tests that are part of a multi-process batch, as well as setting the batch ID, set two additional environment variables as follows:
    
    export APPLITOOLS_BATCH_ID=$(echo `uuidgen`)		
    export APPLITOOLS_DONT_CLOSE_BATCHES="true"
    export APPLITOOLS_BATCH_NOTIFY="true"
    
  • After all the tests have completed, inform Eyes by calling the server. On Linux, for example, you can do this using the curl program as follows:
    
    set server="eyesapi.applitools.com"
    set apikey=$APPLITOOLS_API_KEY
    set batchid=$APPLITOOLS_BATCH_ID						
    set url="https://$server/api/sessions/batches/$batchid/close/bypointerid/?apiKey=$apikey
    
    curl --request DELETE --dump-header - $url
    

    where the variables server, apikey and batchid have the appropriate values for your environment.

The scrips above are for the Bash shell. You can do something similar in the scripting language or CI system of your choice.