Skip to main content

Parallel Test Suites

When running a large volume of tests, taking advantage of parallelization is crucial to avoid long running test suites that delay release cycles.

With Applitools, there are two different ways to run parallel tests:

  • On the client side by splitting up your test suite into concurrent runs
  • On the server side with UFG concurrent renders

When you run tests with the Applitools Ultrafast Grid, you only need client side parallelization if your test suite is very large and you want to further speed up the execution time. There is no need for running parallel tests say for each browser/environment you want to test since the Ultrafast Grid handles the Cross Browser Testing for you. This means you only need to run each test locally once rather than once per each browser/environment.

tip

Check out this article that discusses how the Ultrafast Grid works to learn more about this architecture!

With the Ultrafast Grid, parallelization on the server side is built-in. Simply by changing a configuration option in an Applitools SDK, you can scale up or down the amount of tests that run concurrently on the UFG Servers.

In this article, we'll show you how to run a simple test suite through a CI pipeline using the Ultrafast Grid. Then, we'll show you how to run parallel tests on the client side as well and how to ensure all results are captured from your parallel runs.

Running Tests Through CI With Server Side Concurrency​

Running tests in a CI with Applitools works just like any other test suite. All you need to do is ensure you've set up your APPLITOOLS_API_KEY environment variable correctly and then you can trigger the tests:


Then, we can update the concurrency that our tests run with by using the Applitools SDK configuration:


Using the methods above, the Ultrafast Grid will render 10 browser environments concurrently throughout the course of our test run. If more or less concurrency is needed, simply scale the value up or down and the rest is handled.

Running Parallel Builds in CI With Client Side Concurrency​

When running tests with client side concurrency, the process is similar but requires a few more steps.

Step 1: Making a plan​

It is up to you to decide how to split up your tests and how much parallelization is needed:

  • Are you going to have a single CI job that runs individual tests in parallel?
  • Are you going to have multiple CI jobs that run tests serially?
  • Are you going to have multiple CI jobs that also run individual tests in parallel?

Implementing each of the choices above depends on the tool/framework you're using as well as your team's needs. Regardless of how many jobs you're running or individual parallel tests, there are a couple things you need to handle when running with Applitools:

  1. Configuring the Batch logic
  2. Notifying the Applitools server when all tests have completed

Step 2: Configuring the Batch logic​

In order to ensure all of your parallel tests are batched together under one test run, you can configure a global Batch ID variable to be used by all jobs. You do this by exporting an environment variable called APPLITOOLS_BATCH_ID.

Be sure to choose a value for this variable that is unique to each time you run your test suite in the CI. A good candidate for this is setting the APPLITOOLS_BATCH_ID to the commit SHA that triggered the CI build.


In the above config, at the top I am setting the APPLITOOLS_BATCH_ID environment variable equal to the Commit SHA for the most recent commit. Then, I've created two jobs to run our tests in parallel.

I've also updated our SDK configuration to include a couple extra flags:


The dontCloseBatches flag will ensure that the Batch isn't automatically closed. When a batch is closed, no further tests can be added to it. When running our tests in parallel, we must ensure that we don't close the batch prematurely and only close the batch at the end after all tests have completed (see Step 3 below).

We've also set a BatchName variable to something descriptive for our test suite.

Step 3: Closing the Batch​

Since we've added the dontCloseBatches flag to our configuration, we need to ensure that we close the batch after all of our tests have completed. This can be achieved by making a curl command request at the end of the CI build:


Notice that we added another job called close_batch that waits for all of our test jobs to finish before executing our curl command.

Conclusion​

Running large volumes of tests efficiently often necessitates the use of parallelization. Applitools offers this capability both on the client-side, by splitting your test suite into concurrent runs, and on the server side with the UFG. This article showed how to implement parallel test suites through a CI pipeline using Applitools' Ultrafast Grid, detailed how to set up the appropriate environment variables, adjust the concurrency and manage batching of parallel tests. Utilizing parallelization in a CI envrionment can significantly speed up your testing and release process.