Parallel Test Suites
When running a large volume of tests, taking advantage of parallelization is crucial to avoid long running test suites that delay release cycles.
With Applitools, there are two different ways to run parallel tests:
- On the client side by splitting up your test suite into concurrent runs
- On the server side with UFG concurrent renders
When you run tests with the Applitools Ultrafast Grid, you only need client side parallelization if your test suite is very large and you want to further speed up the execution time. There is no need for running parallel tests say for each browser/environment you want to test since the Ultrafast Grid handles the Cross Browser Testing for you. This means you only need to run each test locally once rather than once per each browser/environment.
Check out this article that discusses how the Ultrafast Grid works to learn more about this architecture!
With the Ultrafast Grid, parallelization on the server side is built-in. Simply by changing a configuration option in an Applitools SDK, you can scale up or down the amount of tests that run concurrently on the UFG Servers.
In this article, we'll show you how to run a simple test suite through a CI pipeline using the Ultrafast Grid. Then, we'll show you how to run parallel tests on the client side as well and how to ensure all results are captured from your parallel runs.
Running Tests Through CI With Server Side Concurrency
Running tests in a CI with Applitools works just like any other test suite. All you need to do is ensure you've set up your APPLITOOLS_API_KEY environment variable correctly and then you can trigger the tests:
- Github Actions
Then, we can update the concurrency that our tests run with by using the Applitools SDK configuration:
Using the methods above, the Ultrafast Grid will render 10 browser environments concurrently throughout the course of our test run. If more or less concurrency is needed, simply scale the value up or down and the rest is handled.
Running Parallel Builds in CI With Client Side Concurrency
When running tests with client side concurrency, the process is similar but requires a few more steps.
Step 1: Making a plan
It is up to you to decide how to split up your tests and how much parallelization is needed:
- Are you going to have a single CI job that runs individual tests in parallel?
- Are you going to have multiple CI jobs that run tests serially?
- Are you going to have multiple CI jobs that also run individual tests in parallel?
Implementing each of the choices above depends on the tool/framework you're using as well as your team's needs. Regardless of how many jobs you're running or individual parallel tests, there are a couple things you need to handle when running with Applitools:
- Configuring the Batch logic
- Notifying the Applitools server when all tests have completed
Step 2: Configuring the Batch logic
In order to ensure all of your parallel tests are batched together under one test run, you can configure a global Batch ID variable to be used by all jobs. You do this by exporting an environment variable called
Be sure to choose a value for this variable that is unique to each time you run your test suite in the CI. A good candidate for this is setting the
APPLITOOLS_BATCH_ID to the commit SHA that triggered the CI build.
- Github Actions
In the above config, at the top I am setting the
APPLITOOLS_BATCH_ID environment variable equal to the Commit SHA for the most recent commit. Then, I've created two jobs to run our tests in parallel.
I've also updated our SDK configuration to include a couple extra flags:
dontCloseBatches flag will ensure that the Batch isn't automatically closed. When a batch is closed, no further tests can be added to it. When running our tests in parallel, we must ensure that we don't close the batch prematurely and only close the batch at the end after all tests have completed (see Step 3 below).
We've also set a
BatchName variable to something descriptive for our test suite.
Step 3: Closing the Batch
Since we've added the
dontCloseBatches flag to our configuration, we need to ensure that we close the batch after all of our tests have completed. This can be achieved by making a curl command request at the end of the CI build:
- Github Actions
Notice that we added another job called
close_batch that waits for all of our test jobs to finish before executing our curl command.
Running large volumes of tests efficiently often necessitates the use of parallelization. Applitools offers this capability both on the client-side, by splitting your test suite into concurrent runs, and on the server side with the UFG. This article showed how to implement parallel test suites through a CI pipeline using Applitools' Ultrafast Grid, detailed how to set up the appropriate environment variables, adjust the concurrency and manage batching of parallel tests. Utilizing parallelization in a CI envrionment can significantly speed up your testing and release process.