Test Maintenance With Applitools

As applications grow, test maintenance becomes a majority of the time spent during the testing lifecycle. Tests fail over time and need to be updated due to a variety of reasons. Let's take a simple test case of navigating to a login page, submitting a profile, clicking the submit button, and checking that we get a "Welcome" text on the next page. This test will fail if:

• any of the form fields change
• any of the form field locators change
• the button ID changes
• the "Welcome" locators changes
• the "welcome" text changes

All of these failures would require you to rewrite your test - a massive waste of time. In our other guide, Replacing Traditional Assertions With Visual AI, we saw how Visual AI assertions can prevent a large amount of selector maintenance. In the following guide, we'll show how to further reduce flaky tests and selector maintenance using Applitools Self-Healing. We'll also show you how Applitools automatically maintains baselines with a single click.

Maintaining Selectors

Selectors are integral components of test code used for navigating and interacting with the application. When these selectors change, tests can fail, requiring the engineer to review the failure, amend the selector, and then re-run the tests. This can be time-consuming and also delay critical releases/updates to your application.

Applitools Execution Cloud has a built-in Self-Healing mechanism that mitigates this issue. When a selector fails, instead of your test failing, the Self-Healing mechanism will automatically detect the failure and self-heal the incorrect selector, allowing the interaction with the application to continue and preventing the test from failing.

For example, here is the output of a test I ran where the selector was not found:

Error: no such element: Unable to locate element: {"method":"css selector","selector":"#username"}

Errors like this can be especially frustrating when trying to release a new version of our application since there's nothing wrong with the application itself.

Without changing any selectors, I then ran this test using the Applitools Execution Cloud with built in Self-Healing:

All Tests Passed

result summary {
    passed=5
    unresolved=0
    failed=0
    exceptions=0
    mismatches=0
    missing=0
    matches=5
}

Process finished with exit code 0

As we can see, the test now passed even though the selector I'm using is still incorrect.

When reviewing your test results on the Applitools Test Manager, you will be notified that Self-Healing was used, which selector failed and how it was repaired:

You can then update your test code to use a correct selector going forward.

Getting Started with the Execution Cloud

See how to set up your tests to use the Applitools Execution Cloud with built-in Self Healing in a few simple steps. Check it out here.

Maintaining Baselines

In Applitools, tests are maintained by updating a baseline, the reference image Applitools Eyes compares screenshots from the test. Think of the baseline as the previously accepted state of your application. The overall workflow of using Applitools for test maintenance involves running your tests, reviewing the test results and, if differences are found, either approving or rejecting the checkpoint image as the new baseline. For example, you might run a test and notice differences on the Applitools Test Manager.

Using the thumbs up/thumbs down buttons in the top right, you'd then either approve these differnces and mark this image as the new baseline image going forward or you'd reject them and keep the baseline image the same.

Baseline maintenance can also include tasks such as adding regions, toggling advanced features like Ignore Displacements, adding baseline variations for A/B tests and more.

Automating Test Maintenance

When you have a few test results, maintaining baselines is a manageable task. However, when you scale up your tests and run hundreds or even thousands of them, maintaining baselines becomes significantly more complex and time consuming. For instance, if a shared component changes within the application under test, this can result in many tests flagging differences found. We'd then need to review each test result individually and take actions on it as outlined above. With hundreds or thousands of tests, we can see how it's not feasible to click through each one of them everytime we make a change to our application.

Applitools provides methods for large-scale baseline maintenance: Grouping by similar differences and Auto Maintenance.

Grouping by similar differences analyzes all test results and, if the differences between the baseline and checkpoint images are the same, groups them together. This enables us to take bulk actions such as approving/rejecting the differences for all test results within the group. Let's say for example we changed the color of the text for the navigation bar in our application as shown below:

This change occurs across all of the pages in our application on various viewports that we're testing against and therefore has affected many of our test results:

If we select the button highlighted above, our tests will then be grouped by similar differences:

Now, instead of having to review 16 similar test results, we only need to review 3 since our tests were grouped into 3 groups based on the differences found.

Grouping Criteria

In this case, you may be wondering why all of our test results were not grouped together. Each browser renders content slightly differently in terms of spacing, font size etc. and this causes slight variance in the differences that are reported on each browser.

Only differences that are exactly the same will be grouped together by the grouping algorithm. Therefore, you can be confident when taking bulk actions that you're not missing any visual bugs.

Auto Maintenance works similarly, but it automatically scans your test results whenever you take a certain action (approving/rejecting differences or adding a region). It then finds other tests where the same action could be performed, saving you from having to do it repeatedly.

For example, if we wanted to add an Ignore Colors Region around our changing navbar, we could perform that action once and then select this button shown below:

Clicking the "Apply to other steps in scope" button highlighted above will trigger Auto Maintenance to scan your test results and automatically apply the Ignore Colors Region to other tests that also have that navbar. As we can see below, the same type of region was added to our other test automatically:

Conclusion

From maintaining Visual AI assertions and key selectors to dealing with large-scale baseline maintenance, Applitools offers a reliable solution that simplifies and streamlines test maintenance. So whether you are a QA engineer dealing with a handful of tests or juggling hundreds, you can focus more on quality assurance and less on repetitive maintenance tasks.