Solution
The screenshot comparison tool, developed internally by WalkMe, had its limitations that could not be reconciled with the needs of the Test Automation & QA teams: pixel-by-pixel tests were brittle, over-inclusive and resulted in too many “false negatives”, and manual QA simply couldn’t cover all product versions across all platforms. A faster, smarter solution was needed.
After learning about its capabilities, WalkMe decided to implement Applitools Eyes, which differs from other screenshot comparison tools due to its unique algorithm that mimics human vision: this means that tests are robust, do not generate “false negatives” and are easy to create, maintain, and update.
Ease of Implementation – WalkMe’s experienced test automation team quickly created a wrapping to the Applitools Java API, and integration to Selenium test framework and the TeamCity Build server was fast. The complete deployment of Applitools into the existing end-to-end, fully functional tests, integrated with all WalkMe test automation tools, took just two days.
Increased Coverage, More Bugs Caught Pre-release – With Applitools Eyes, WalkMe can now catch visual regressions, which they had no other means of detecting without it. In addition, with Applitools, WalkMe were able to add additional scenarios that were not possible before, thus increasing test coverage even further. Prior to Applitools, manual testers were checking 3 browsers per cycle; with Applitools Eyes now all app versions are regularly tested across 10 browsers.
Huge Time Saver – Whereas before, a manual tester would’ve needed a week to go through all the visual UI tests, Applitools covers those and more in a matter of 5-6 hours. This means, manual testers can now focus on more advanced testing, such as acceptance testing and load testing.
Cross Department Collaboration Tool – Another time consuming element was bouncing issues back and forth between QA, Test Automation and the Devs. This is a huge time waster for most companies. New bugs caught would require a long “open bug” cycle: running old version, taking screenshot, then running the new version with new screenshots, then sending the logs to the developers. This process would take about 30 minutes per issue – just on the Test Automation team’s end.
WalkMe solved this problem with Applitools Eyes, cutting the relentless cross-department ping-pong. Applitools cloud-based solution stores all tests in the cloud, allowing team members immediate access to all tests from everywhere. When the Test Automation team spots an issue (i.e. failed test) they send a link – with the visual log – to the devs. Applitools Eyes playback option allows all team members to see the visual comparison between the two versions, including the chain of events that lead to the bug. This helps the team to quickly pin-point the issue and decide if the test failed due to a bug, or if the change is to be accepted.