Record Playback and Visual Testing – Part 1

Test automation folklore is full of horror stories of failed attempts to apply Record & Playback tools to perform GUI-based Functional Test Automation. In this post we will explore the main causes for these failures. In part 2 of this post we will show how Record & Playback tools can be effectively used for automating Visual Testing. 

Many commercial and open source test automation tools provide record and playback (RPB) features, that allow testers to easily create tests by recording manual interactions with the UI of the Application Under Test (AUT). Recorded tests can then be played back to automatically test the AUT, possibly in different execution environments than the one used to create the test. For example, Selenium IDE, a popular open-source RPB test tool for web-applications, allows testers to record tests on a Firefox browser, and play them back on a variety of other desktop and mobile browsers.

The appeal of RPB test automation tools is clear: they allow testers to quickly and easily create automated tests while requiring little to no programming effort and tool specific skills. Since the introduction of the first RPB test automation tools in the early 90’s, the technologies underlying them have evolved substantially. Robust, multi-layered UI object identification replaced absolute cursor coordinates, implicit delays are automatically added when missing UI objects are accessed while an application window loads, the sensitivity of the recorded tester actions can be throttled, UI object maps allow UI objects identification details to be shared between tests to simplify maintenance, etc.

However, despite all these advancements, test automation folklore is full of horror stories of failed attempts to apply RPB tools to perform GUI-based Functional Test Automation. Let’s take a deeper look at some of the main causes for these failures:

• No test reuse: with RPB tools, if you want to add a test, you simply record it, resulting with test steps that invariably overlap step sequences in other tests, but have to be maintained separately. When the behavior of the AUT changes, the affected scenarios must be re-recorded and corrected individually in each of the tests.
• No data parameterization and control flow structures: when validating the functionality of an AUT, it is often required to repeatedly perform the same series of actions, each time using different data as input. A pure RPB tool requires a tester to record each and every one of these repeated interactions, hard-coding the input data inside the test, instead of looping through the repeated scenario using different input and expected output data (defined externally) in each iteration. Moreover, in order to add, modify or remove data entries, actions must be re-recorded, and when the AUT changes, each and every recorded “iteration” has to be updated separately.
• Validation: simulating user interactions is just a small part of creating and maintaining automated tests. The bulk of effort involved is validating the correctness of the AUT. A manual tester validates the functionality of the AUT by looking at the screen or by performing procedures that are external to the UI of the AUT. Of course, this sort of interaction cannot be recorded by the tool and must be manually defined by the tester. Different RPB tools provide different facilities to define validation commands and assertions, ranging from simple WYSIWYG abstractions and up to full-fledged programming. To accomplish this task effectively, tool specific skills and even programming skills are required, which contradicts the initial motivation to use an RPB tool. Combined with the little or no test reuse described above, there is little chance for large-scale RPB-based test automation to succeed.

Due to these limitations, RPB tools are hardly used in-practice for automating GUI-based functional testing. Most commercial and open-source test automation tools have evolved beyond RPB, to facilitate large-scale functional test automation, but at the expense of raising the bar of entry in terms of tool specific skills and programming skills. In part 2 of this blog we will show how a pure RPB approach can practically and effectively be used for automating Visual Testing.

Yeah! Take me to part 2 of this post!

To read more about Applitools’ visual UI testing and Application Visual Management (AVM) solutions, check out the resources section on the Applitools website. To get started with Applitools, request a demo or sign up for a free Applitools account.