Visual UI Testing in conjunction with functional testing is a new approach to improving application quality without additional effort, and like all new technologies, people have questions. The most common is that visual testing seems to have “false negatives” or is over-inclusive. This post eliminates that concern and outlines the 4 most important capabilities a smart visual testing solution should offer you and your team.
Visual UI testing looks at applications from the user’s perspective, by comparing screenshots of a current release to previous releases, and is a way to immediately spot visual regressions, thus helping you quickly enhance the quality of your applications with minimal effort. In its most basic form, visual testing is done by pixel to pixel comparison of the baseline image with the new image.
More advanced variations of visual testing compare layouts and relationships between objects on the page and do this across a matrix of browser versions, device types, screen resolutions, and various operating systems. This type of visual testing requires sophisticated layout mapping and normalization, which is based on highly advanced image analysis techniques. It is therefore not feasible to implement this type of visual testing over existing open source tools.
One of the first things a tenured QA professional will say about such a tool is that it sounds great, but wouldn’t it be over-inclusive? Meaning wouldn’t it catch all differences in the application, even if they are expected or insignificant (in terms of relevancy to what the human user actually sees)?
The answer is yes and no. Some tools will indeed be over inclusive, however, tools like Applitools Eyes have already considered this problem, and addressed it. With their considerations and your smart test design, you end up with a solution that is helping and optimizing your processes, without the setbacks.
Here are the 4 things to look for in an automated visual testing solution, which will help you not only avoid “false negatives” in your visual tests (i.e. when a test fails when it shouldn’t) but also to fully optimize your testing processes:
1) Going Beyond Pixels: Mimicking Cognitive Human Vision
Unlike open source visual testing libraries, which only do a pixel-by-pixel comparison of screenshot images, advanced solutions look beyond that. They look also at document layouts and graphical structure and will identify text, graphics, etc. – and realize that these objects often carry more weight than their contents.
For example, such a tool can identify that part of the page is an image, and another is a text paragraph. This means that if you take a screenshot at two separate moments where the image and the text change, but their position is the same, the solution will “know” that these changes are not errors and the test will not fail.
2) Change Propagation in New App Version: Eliminating the “Groundhog Day” Effect
A good visual testing solution will know the relationship between one page and all its proceeding ones. Manually implemented visual testing solutions are ad-hoc and treat all pages as their own layer with no connection to the next. In this approach, the only relationship of each page image is its counterpart screenshot from later versions.
However, if you use smart visual testing solution with automatic maintenance technology, it automatically understands the relationships between pages in the same app version. This means that when you accept a change in one page – the visual testing solution can automatically identify this change in subsequent pages as an *approved* change, so tests do not fail unnecessarily and you won’t have to re-accept the same change over and over for each page in the new version. This is a huge timesaver for anyone who ever felt testing processes were a bit like Bill Murray in”Groundhog Day”.
3) Cloud-based Collaboration Tool: Knowing The Latest Change
There is an issue with any type of functional testing: sometimes the testing team does not know which change is the latest. They are unable to spot a change that should be approved, consider it an error, and waste time bouncing it back to dev. Good communication that makes the QA team aware of all new elements is critical when working with functional and visual testing.
Implementing automated visual testing can help teams prevent this needless ping-ping and save precious “passing the torch” time. More advanced solutions, such as Applitools Eyes, can be used as a cross-department collaboration tool: they offer a cloud-based visual reporting that can be accessed by anyone in the organization, visual logs allow non-techies to see the new versions and visual bugs. Most of all, it allows QA to quickly pinpoint issues, and the developers can quickly take a look at the visual log and decide if it’s a visual bug or an approved change.
The strenuous process, where developers need to brief testers on every new feature and change in each version – can be built-in with cloud-based automated visual testing, eliminating the need and time spent on cross-department version meetings.
4) Capture Order is Important: Validating Front-end Functionality
Just like in pure functional testing, where you are trying to truly implement user flows, as the user would experience them – in visual testing you should view pages in the order that users do as well. The reason you want to do this is that the flow itself impacts the chances of “false negatives” since new pages in the flow can also be considered “changes” that are errors unless expected. Without having them in sequence, it is hard to know if a page is new or just out of order.
However, if you design it with the proper user flow, this ends up being an opportunity. It allows the QA team to both automatically add the new page to future tests, and to validate its quality in a glance without writing a special script for it. More than that, sophisticated visual testing tools can perform alignment of the pages: if some pages are missing and/or new pages are added, or if pages were replaced, these tools would know where to place these pages or show missing steps in the flow!
If in the past you gave visual testing a go but were bombarded by “false negatives”, due to the identification of irrelevant changes in your application, and therefore automatically assumed it’s going to overwhelm your team with too much information – Think Again!
Solutions like Applitools Eyes have paid specific attention to these issues and solved them, with advanced algorithms and processes that make visual testing one of the best things you can do to instantly increase overall coverage, improve test reliability, and significantly decreased the overall time spent on code maintenance, not to mention move from a reactive testing team to a proactive one.
Ready to dive deeper? Read more about Applitools’ visual UI testing and Application Visual Management (AVM) solutions and check out the resources section on the Applitools website. To get started with Applitools, request a demo or sign up for a free Applitools account.
Chris Riley is a developer turned DevOps enthusiast. He helps organizations realize the benefits of modern tooling.
