Batching tests in a distributed environment

Grouping tests into batches, allows you to see and manage the results of multiple related tests together in the Test Manager. The article Grouping tests into batches with the SDK explains how to batch tests when they run in the same test execution.

This article, explains how to batch results of tests that are executed in different processes or on different machines. This technique is useful if, for example, your tests are coded in different programming languages, your tests need to run on multiple target platforms (e.g. Linux, Windows, Mac, and mobile devices), or you run tests concurrently on multiple CPUs or distributed processors. The techniques described here are also useful when you run a single process test using multiple runners.

To batch together tests that run in separate processes or separate runner, you need to do three things:

Avoid batching together unrelated tests unnecessarily, since this may increase the number of tests in a batch to a large number and can cause severe performance issues and/or failures.

Prevent auto-closure of batches by runners

If you use a ClassicRunner or VisualGridRunner in your tests then Implicit batching by a runner, and by default, the runner closes the batch(es) in the call to getAllTestResults after the tests have completed.

If you are using multiple executions, or more than one runner in a single execution, and you want the tests to be in a common batch then you have to disable the auto closing of batches. There are two ways to do this:

  • By calling the method setDontCloseBatches on every runner immediately after it is created, as shown below:

    
                                    
    • 
                                    
  • 
                                        
    By defining the environment variable APPLITOOLS_DONT_CLOSE_BATCHES before the test runs. 
    
                                        
    The same methods are available in the ClassicRunner class and the VisualGridRunner class.	
    
                                    
    • 
                                

                                
Associate tests with a common batch

                                
When you run test in a single process, you can batch them by assigning a shared BatchInfo object to every Eyes instance using one of the methods described in How to group tests into batches. 

                                
When tests are run in separate runs, you cannot share a BatchInfo object. Instead, each test creates its own BatchInfo object, and the tests are batched by explicitly assigning the BatchInfo object a common Batch ID.

                                
We recommend that you use the same batch name for all the tests. However, even if you use different names, it is the common batch ID that groups the tests together. The batch name displayed in the Test Manager is the batch name of the first test that runs.

                                
Typically sharing a batch id is done using the following steps:

                                
    
                                    
  1. 
                                        
    Generate a unique ID string, for example by using date and time stamp and some machine ID. Since batches are stored on the server for a long time, you need to ensure that the ID will be unique in the long term.
    
                                    
    2. 
                                    
  2. 
                                        
    Distribute the unique ID to all your test programs  (e.g. by writing it to a network file or a database).
    
                                    
    3. 
                                    
  3. 
                                        
    In each test program, obtain the shared ID and assign it to its BatchInfo object. This can be done in two ways:
    
                                    
    4. 
                                    
      
                                        
    • 
                                            
      By obtaining the ID in a script and setting the environment variable APPLITOOLS_BATCH_ID to the unique ID before starting the test process. The SDK will use this value of this environment variable if it is defined and if the batch ID is not set explicitly.
      
                                        
      • 
                                        
    • 
                                            
      By obtaining the batch ID in your code and setting the BatchInfo object explicitly as illustrated below:

      
                                        
      • 
                                    
    
                                

                                
The call to the function myGetUniqueBatchID() represents the code you need to add that obtains the ID from wherever it is stored.

                                
Close the shared batch

                                
Closing a batch means that:

                                
                                
You can close a batch by using the BatchInfo class in an SDK that supports it (e.g. Java) or you can send an HTTP request directly to the Eyes server using a program such as curl. 

                                
Closing a batch with the BatchClose class

                                
The following snippet shows how to close a batch using the BatchClose class. You can do this in any program, not necessarily a program that runs an Eyes test.

                                
Note that before calling the BatchClose method, you need to pass one or more batch ids to the method setBatchId. In this example we use the value of the environment variable APPLITOOLS_BATCH_ID, typically you would use the same method you used to assign the ID when defining the batch as described in the previous section Associate tests with a common batch.

                                
Sending the server an HTTP requests directly

                                
For SDKs that don't support the BatchInfo class, you can close the batch using curl as illustrated in the following Bash script:

                                
For Linux:

                                
Copy
                                    
export APPLITOOLS_SERVER_URL=https://eyesapi.applitools.com
curl -v -X DELETE "$APPLITOOLS_SERVER_URL/api/sessions/batches/$APPLITOOLS_BATCH_ID/close/bypointerid?apiKey=$APPLITOOLS_API_KEY"

                                    

                                

                                
For Windows:

                                
Copy
                                    
set APPLITOOLS_SERVER_URL=https://eyesapi.applitools.com                
curl -v -X DELETE "%APPLITOOLS_SERVER_URL%/api/sessions/batches/%APPLITOOLS_BATCH_ID%/close/bypointerid?apiKey=%APPLITOOLS_API_KEY%"

                                    

                                

                                
The script assumes that the variables APPLITOOLS_SERVER_URL, APPLITOOLS_API_KEY and APPLITOOLS_BATCH_ID have the appropriate values for your environment.

                                
If you use the public cloud, the value of APPLITOOLS_SERVER_URL is 'https://eyesapi.applitools.com'. If you run the Eyes server on a private cloud or on-premise system, then change the value assigned to APPLITOOLS_SERVER_URL to the URL of your Eyes server.