Batching Test runs

In the Applitools Test Manger, test results are organized by batches. Batching tests together can make analysis and management of your test results easier and more convenient. Eyes also has features that operate on the tests in a batch as a whole, such as filtering, deletion, and grouping by similarity.

In the screen shot below you can see a list of batches in the left panel. Scrolling through the list you can see all the batches run by your team. When you click on a batch it becomes the selected batch and the right panel displays the tests that are associated with that batch.

Configuring BatchInfo

Batches are configured using a BatchInfo class that comes bundled with the Applitools SDKs. All SDKs have default behavior for the BatchInfo object out of the box but can be configured as needed using the properties listed below.

BatchInfo Properties

Property	Description
batchName	A descriptive name for the Batch. This name will show up in the Batch List on the Applitools Test Manager. Purely cosmetic with no agency over the grouping logic.
batchId	A unique identifier for the batch. This ID is used as the primary indicator of how tests should be grouped.
batchSequenceName	A name used to track batch performance over time. See Batch Statistics/Insights section below
setNotifyOnCompletion	Toggle on/off batch completion notifications via Email, Slack or Microsoft Teams. See Batch Completion Notifications section below
Custom Properties	An optional list of custom key/value pair properties that can be applied to the batch. See Custom Properties section below.

Reviewing Batch Results

When you execute a test suite, an entry in the batch list on the Applitools Test Manager is added:

If you don't configure a batchName, the name of the batch will default to the name of the first test that is executed in the test suite.

Filtering

You can filter your batch list by many different variables including:

• Which user ran the test suite
• The name of the batch
• The status of the test results within the batch
• Custom property values configured on the BatchInfo object

and much more using the filter icon shown below:

Deleting a Batch

You can remove a batch from the list by selecting the checkbox next to the batch followed by the delete button:

Batch Details

You can view extra details of a Batch which include the batchId, the time of execution, who it was run by, custom properties and more by selecting the icon shown below followed by the Batch Details button:

note

The icon will only show up when you hover over an entry in the batch list.

In the popup menu, you can delete the batch or rename the batch as well.

Closing the Batch

Closing the batch is an important step in the lifecylce of the batch execution. If a batch is not closed, any subsequent test runs that execute with a batchId that matches an existing, open batch will be added to that previous batch rather than creating a new entry in the batch list.

Additionally, Batch Completion Notifications won't be sent until the batch has been closed.

Depending on the SDK that you're using, the Batch Closing logic can be handled differently.

tip

In order to catch a situation in which a test has crashed or is hung, unterminated batches that have been inactive for more than two hours or have been running for more than four days are automatically terminated by Eyes, at which point notifications are sent as configured.

Automatic Closing

Eyes SDKs will typically handle the closing of batches automatically when there is a call to the getAllTestResults API or equivalent for your selected SDK:

note

Sometimes you may need to more control over the batch close process. See Manually close the shared batch section below.

Custom Properties

Custom Properties allow you to associate key/value paris with a BatchInfo object. On the Test Manager, you can then:

• Search for all batches that are tagged with a set of given key/value pairs
• See the properties that where assigned to a particular batch

The property names and values are global for all batches run by a particular team in an organization.

Creating a custom property

Eyes always retains, as a minimum, the 50 most recently assigned batch property keys and the 50 values most recently associated with that key. If more than 50 properties are defined then older properties that have not been assigned in any batch in the last 30 days are removed.

Filtering by custom property

Click on the filter icon above the batch list:

At the bottom of the dialog that pops up, all custom properties that have been set will be shown as filter options. From there, you can select which property and values you're interested in:

Viewing custom properties

Open the batch details dialog using the Batch Details option shown below:

In the dialog that pops up, any custom properties and values that have been set on this batch will be shown in this dialog.

Batch Statistics/Insights

Using the SDK batchSequenceName configuration option, Eyes groups all batches that are associated with the same Batch sequence name into a chronological sequence of batch executions. Eyes collects, analyzes and presents batch and test execution status information and statistics for each batch sequence that was executed within the Scope and Date Range values that you set on the Batch statistics tab.

The batchSequenceName can be configured as shown below:

A batch sequence can be associated with any number of test runs and batches. It is up to you as the user to choose how to organize the sequences.

The Insights Batch Statistics tab, found on the Insights Page on the Applitools Test Manager, displays a list of batch sequences. The following figure shows an example of how the Batch Statistics tab appears when it displays a list of Batch sequences:

In the image above, we see each batch sequence name we've created as well as performance metrics showing how well our sequence has been over the selected date range.

If we click on one of the sequences in the table, we get a more in depth look at the data behind the performance numbers:

Tracked Metrics

Test result status

The Test Results Status is a Pie chart that displays the pass rate for all the tests that were executed over a set period of time. The Pass rate represents the ratio of the number of tests that passed as compared to the number of tests that did not pass, expressed as a percentage.

Diff status

A Pie chart that displays the percentage of Diffs that are currently accepted, rejected and unresolved for the selected batch sequence.

Baseline status

Indicates the status of steps and changes on the baseline. Shows the number of Saved steps, Unsaved steps and Unsaved changes.

• Saved steps refers to steps that at some point were saved to the baseline.
• Unsaved steps refers to steps that were never saved to the baseline.
• Unsaved changes refers to steps that contain changes that are yet to be saved to the baseline.

Tests

Indicates the total number of tests in the batch and the number of those tests that are Completed, Running and Aborted.

Checkpoints

Indicates the total number of checkpoints executed in the batch and the number of those checkpoints that are Matching, Mismatching or New.

• Matching refers to checkpoints for which Eyes did not detect any differences.
• Mismatching refers to checkpoints for which Eyes identified differences.

Browsers

A bar chart that shows a bar for each browser on which batches were executed as part of the batch sequence.

Devices

A bar chart that shows a bar for each emulated device for which tests were rendered as part of the batch sequence.

Operating systems

A bar chart that shows a bar for each operating system on which batches were executed as part of the batch sequence.

Viewports

A bar chart that shows a separate bar for each Viewport size for which tests were rendered as part of the batch sequence.

Test assignment

A bar chart that shows a bar for each team member and indicates the number of tests that were assigned toe each team member. Tests that were not assigned to any team member are represented by a bar that is labeled as Unassigned.

Issues and remarks

A bar chart that shows two bars, one for Issues and one for Remarks, that indicate the number of Issue and Remark regions that are applied in steps that were checked as part of the batch sequence. The Issues and Remarks bars are color-coded to show the number of Issues or Remarks that are New or that were already Existing. If Issue or Remark regions were applied to a batch sampled at a point in time within the set Date Range then those Issues and Remarks are listed as New. If the regions were applied at a point in time before the set Date Range then they are listed as Existing.

Batch Completion Notifications

Batch Completion Notifications can be configured and sent over Email, Slack and Microsoft Teams. These notifications will include the test result status as well as a link to the test results in the Applitools Test Manager. Check out the articles below for instructions on how to configure Notifications for the platform you're interested in:

• Email
• Slack
• Microsoft Teams

Configuring the SDK to send notifications

tip

Completion notifications are only sent when the Batch is closed. See the above Closing The Batch section for details.

Batching Parallel Executions (CI/CD)

In some cases, your test suite executes via many jobs that run in parallel. This could be distributed jobs via a CI/CD build or running multiple environments/browsers at the same time. In cases like this, it's important that all of the test runs get batched together so that you can keep the batch list orderly.

When running with parallel jobs, the test suite is distributed across many different instances and therefore, the Applitools SDK doesn't always have the full picture of all jobs that are executed. Therefore, it's up to the developer to perform the following steps when writing the batch logic:

• Prevent auto-closure of batches
• Manually close the shared batch
• Manually close the shared batch

Prevent auto-closure of batches

As mentioned in the Closing the batch section above, the Applitools SDK will automatically close the batch by default at the end of the test suite or what it perceives as the end of the test suite. This is problematic in cases like this since one execution instance will close the batch before the others have finished which will result in multiple batches added to the batch list on the Applitools Test Manager.

The first step is to prevent this automatic closing by setting the dontCloseBatches flag to false. You can set this flag using the APPLITOOLS_DONT_CLOSE_BATCHES envrionment variable:

export APPLITOOLS_DONT_CLOSE_BATCHES=false

Alternatively, this flag can be set in the SDK configuration:

Associate tests with a common batch

The next step is to ensure that each test execution job uses the same batchId. This value should be unique such that it won't be repeated for every test run but also should be shared across all parallel executions. A good example of this is setting the batchID to the commit SHA that triggers a CI/CD build.

You can set the batchId using the APPLITOOLS_BATCH_ID environment variable:

export APPLITOOLS_BATCH_ID=<YOUR_ID>

Alternatively, the batchId can be set in the SDK configuration:

Manually close the shared batch

The last step is to manually close the shared batch after all the parellel executions have finished.

Close Batches via HTTP Requests

If needed, you can close the batch using curl as illustrated in the following Bash script:

For Linux

export APPLITOOLS_SERVER_URL=https://eyesapi.applitools.com
curl -v -X DELETE "$APPLITOOLS_SERVER_URL/api/sessions/batches/$APPLITOOLS_BATCH_ID/close/bypointerid?apiKey=$APPLITOOLS_API_KEY"

For Windows

set APPLITOOLS_SERVER_URL=https://eyesapi.applitools.com                
curl -v -X DELETE "%APPLITOOLS_SERVER_URL%/api/sessions/batches/%APPLITOOLS_BATCH_ID%/close/bypointerid?apiKey=%APPLITOOLS_API_KEY%"

The script assumes that the variables APPLITOOLS_SERVER_URL, APPLITOOLS_API_KEY and APPLITOOLS_BATCH_ID have the appropriate values for your environment.

If you use the public cloud, the value of APPLITOOLS_SERVER_URL is https://eyesapi.applitools.com. If you run the Eyes server on a private cloud or on-premise system, then change the value assigned to APPLITOOLS_SERVER_URL to the URL of your Eyes server.