Batching Tests in a distributed environment

In the article How to group tests into batches using the SDK we described how to batch multiple tests that run in the same execution run so that their results are batched in the Test Manager. In this article, we will explain how to batch the results of tests when they are executed separately. This technique is useful in various circumstances, for example, if your tests are coded in different programming languages or if they need to run on multiple target platforms (e.g., Windows, Mac, and various mobile device) or need to be run concurrently on multiple CPUs or distributed processors.

The main difference between this method and the one described in Working with test batches is that instead of sharing a BatchInfo object, each test run creates its own BatchInfo object, and the tests are batched by explicitly giving them a common Batch ID.

  1. The first step it to generate a unique ID string; you can do this any way that is convenient, one way is to use a date and time stamp and some machine ID.
  2. Remember that since batches are stored on the server for a long time, you need to ensure that the ID will be unique on the long term.

  3. Once you have an ID, you need a way to distribute it to all your test programs, for example, by writing it to a network file or a database.
  4. Finally, each test program needs to obtain the ID  (represented in the code below by the call to myGetUniqueBatchID()) and set the BatchInfo batch ID explicitly, as follows:

                            

You should give the same batch name for all the tests, however even if you don't, it is the batch ID that binds them together. The information displayed in the Test Manager for each step will reflect the batch name given when the test ran.

WARNING: Using this method means that the number of tests in a batch is not limited to a single test run. Reuse of an ID for unrelated tests can cause an unnecessary increase in the number of tests in a batch. This has no functional advantage since the tests are not related – but it may cause severe performance issues and/or failures and so should be avoided.