You have a choice. You can be a puppet master and test applications by making them do cool things from within. Crossing your fingers you cover all use cases. Or you can be a Deity who comes to earth and sees the world from the user’s eyes, but maintains a million foot view of the application as well. Both a puppet master and a Deity are fun roles, so which approach should you choose?
We assume you know what functional testing is: the ability to test applications from the user’s perspective. The most common way to do this is to leverage Selenium and some web driver to emulate user actions – much like a puppet master controlling his puppet. Most of the time you are testing the common use cases to make sure that a new release did not break something. But often avoid new or fringe user scenarios because they are too complex to write.
The benefit of testing this way is that it is repeatable and you can very easily test complete user flows. The downside, however, is you are testing from within and focused on reactive testing vs. proactive. When you write the tests you are re-creating actions you think the user will take. There is a high chance you will miss something, and human error is a common element in the creation of any Selenium script. The puppet master is not always perfect.
And substantial effort put in maintaining scripts. The benefit is it standardizes tests and builds consistency in application quality. But it is also problematic because the script creation and maintenance is time intensive. And as scripts age, they often result in exceptions during a test run, or wasted time testing something that no longer exists.
There is another type of testing which focuses on what the user sees on the screen. It is an “outside looking in” approach, the Mythical Deity. It compares all pages of the application to previous versions, or layout changes that are expected or not.
This approach leverages visual regression technology that compares one screenshot image to another, exactly what the user sees in the browser. It is capable of identifying even a 1×1 pixel difference from one state to another. These are the basics.
However, advanced implementations of visual testing also include tools to identify expected changes and propagate them automatically throughout the entire test, to avoid “false negative” (i.e. a test fails when it shouldn’t) or to avoid looking at every instance of a visual change when it goes across many pages.
The benefit of the visual testing approach is that it is really fast to implement. It automatically addresses all pages on a page-by-page use case. And it is like being the Deity who comes to earth just to see how everyone else sees the world. An additional benefit of visual testing is that it can help you gain functional coverage at a fraction of the effort.
These are the differences between pure functional testing and visual testing. And besides it being similar to one of my favorite games “Who would you rather be” the choice of which approach to take should be fairly obvious.
Both!
Of course, both, when you have the opportunity to leverage both the puppet master and Deity techniques at the same time, why wouldn’t you? Just like the development side of the house, where the best tool is used for the job, combining visual and functional testing is the ideal way to push the team further. And the best part is that because you are using both – they end up benefiting each other.
Functional testing scripts can focus on user flows, while the visual UI testing focuses on look and feel, and regression testing, which means you can trim your functional testing scripts to the essential user flows and leave new functionality and detailed visual elements to the visual testing. You then can transition your testing from regression only to testing new functionality and existing functionality in a deeper way.
The visual UI testing will integrate into your existing test scripts by simply adding three lines of code that are consistent for every page you test. During the script run, it captures that page and sends it to a portal to manage and review testing results. The portal will allow for playback, screen by screen comparison, and note any unexpected differences from one version of the application to another.
The best part is that because you are looking at the application from the outside in, if and when you forget to functionally test a critical part of the application, it is not so painful. For example, if there is a new menu bar where you need to look at how it fits on all pages across the entire application. This is a simple test, but hard to remember, and annoying to implement in your spider web of tests. However, if you do forget to test it, it will be a huge waste of time to fix or worse yet if ignored a huge impact to users.
With visual UI testing, it won’t be as costly, because the test will automatically capture the menu bar without writing specific Selenium commands to do it. This is because the screenshot catches everything on the page. So you do not need to change the capture for any changes in the application. In this example, you will not need to do a repeat test run, because we all know how much of a delay and anger this can cause.
There is no way to test comprehensively without seeing the application through the users’ eyes. And there is no way to test complex user flows without scripting. Combining functional testing and visual testing is not only a way to stretch your imagination of being a puppet master or Deity, it also is the only way to progress QA teams and improve software quality without additional effort.
To read more about Applitools’ visual UI testing and Application Visual Management (AVM) solutions, check out the resources section on the Applitools website. To get started with Applitools, request a demo or sign up for a free Applitools account.
Chris Riley is a developer turned DevOps enthusiast. He helps organizations realize the benefits of modern tooling.
