WalkMe

Case Study — Digital Transformation

Leading guidance & engagement platform WalkMe saves days of manual testing and transforms cross-team collaboration with Applitools Eyes Automated Visual Testing

It is hard to cover all the bases with manual testing; and we needed something that will guarantee a flawless user experience everywhere – this is why we chose Applitools Eyes. It's the perfect solution for our visual validation requirements; it helped us improve our product, ensure quick delivery, increased coverage and saved our teams hours on cross-department briefing.

Nir Pinchas — WalkMeHead of Test Automation

Download case study

Highlight

This is how WalkMe’s Test Automation Department drastically reduced time spent on manual testing and improved test coverage, preventing more visual regressions & functional bugs. In the process, they transformed cross-department collaboration, thus saving valuable development hours – all with Applitools Eyes: an easy-to-integrate automated visual UI testing solution.

About WalkMe & its Test Automation Infrastructure

WalkMe has developed innovative and complex guidance and engagement application suite, which accelerates software training, reduces customer care costs and increases revenue. With complex business applications – ranging from interactive user guidance to online surveys, user feedback forms, product guides, and back-end/admin dashboards – WalkMe widgets were designed to supports apps and websites across the entire device/OS ecosystem, in order to ensure a flawless user experience, aimed at increasing conversion rates. As such, WalkMe needed to make sure their customers themselves received a flawlessly-working and adaptable product that will guarantee a great user experience.

WalkMe uses Java-based Selenium as its test automation framework, TeamCity CI build server, and test code is written with Junit.

WalkMe widget and menu are complex products, with thousands of tests checking thousands of possible scenarios, including graphic elements tested via screenshots. WalkMe has two test cycles: short build test run conducted nightly, and a full cycle running on the weekends. Applitools tests run as an integral part of those QA cycles.

Early on, WalkMe automation team developed a proprietary screenshot comparison tool, in order to answer their need for visual validation of graphical elements that cannot be tested manually (too time consuming), nor with standard tests.

Soon after, they saw that instead of elevating the pressure on the Test Automation and QA teams, their in-house tool actually created additional maintenance, and was unreliable due to a high number of “false negatives” (i.e. over inclusiveness, when a test fails when it shouldn’t, due to insignificant changes). They needed a tool that could replace human vision, allow for better coverage and have reliable results.

Challenges

  • Complex graphic elements that cannot be tested with standard tests
  • Manual testing cannot cover all product versions across all platforms in a timely manner
  • Multi-browser/resolution/OS ecosystem – maximum coverage is a must
  • In-house visual UI testing tool was too sensitive, and was hard to monitor and maintain
  • In-house screenshot comparison tool was brittle and generated a lot of “false negatives”
  • Needed a system that would integrate quickly with Selenium, TeamCity and other automation tools

You’re looking at 30 minutes per bug without Applitools, compared to 5 minutes with – this accumulated to saving an average of 4 working days per month. In my book, this means another 2 months of available man-hours we did not have before.

Solution

The screenshot comparison tool, developed internally by WalkMe, had its limitations that could not be reconciled with the needs of the Test Automation & QA teams: pixel-by-pixel tests were brittle, over-inclusive and resulted in too many “false negatives”, and manual QA simply couldn’t cover all product versions across all platforms. A faster, smarter solution was needed.

After learning about its capabilities, WalkMe decided to implement Applitools Eyes, which differs from other screenshot comparison tools due to its unique algorithm that mimics human vision: this means that tests are robust, do not generate “false negatives” and are easy to create, maintain, and update.

Ease of Implementation – WalkMe’s experienced test automation team quickly created a wrapping to the Applitools Java API, and integration to Selenium test framework and the TeamCity Build server was fast. The complete deployment of Applitools into the existing end-to-end, fully functional tests, integrated with all WalkMe test automation tools, took just two days.

Increased Coverage, More Bugs Caught Pre-release – With Applitools Eyes, WalkMe can now catch visual regressions, which they had no other means of detecting without it. In addition, with Applitools, WalkMe were able to add additional scenarios that were not possible before, thus increasing test coverage even further. Prior to Applitools, manual testers were checking 3 browsers per cycle; with Applitools Eyes now all app versions are regularly tested across 10 browsers.

Huge Time Saver – Whereas before, a manual tester would’ve needed a week to go through all the visual UI tests, Applitools covers those and more in a matter of 5-6 hours. This means, manual testers can now focus on more advanced testing, such as acceptance testing and load testing.

Cross Department Collaboration Tool – Another time consuming element was bouncing issues back and forth between QA, Test Automation and the Devs. This is a huge time waster for most companies. New bugs caught would require a long “open bug” cycle: running old version, taking screenshot, then running the new version with new screenshots, then sending the logs to the developers. This process would take about 30 minutes per issue – just on the Test Automation team’s end.

WalkMe solved this problem with Applitools Eyes, cutting the relentless cross-department ping-pong. Applitools cloud-based solution stores all tests in the cloud, allowing team members immediate access to all tests from everywhere. When the Test Automation team spots an issue (i.e. failed test) they send a link – with the visual log – to the devs. Applitools Eyes playback option allows all team members to see the visual comparison between the two versions, including the chain of events that lead to the bug. This helps the team to quickly pin-point the issue and decide if the test failed due to a bug, or if the change is to be accepted.

Simply put: Applitools Eyes cost-effectively deals with visual regression & visual validation issues. It saves us a lot of time and money and its product quality assurance is impeccable.

Results

  • Substantially better, more robust automated visual UI testing solution, with over 15,000 visual checkpoints per week
  • Over 20% improvement in overall number of regressions & UI bugs caught pre-release
  • Saved weeks of manual testing: all visual UI tests take 5-6 hours per cycle, instead of weeks needed with manual efforts
  • 2 months-worth of man-hours saved yearly: saved 4 working days per month on average on regressions cycles, with Applitools’ cloud-based cross-department collaboration tool
  • Greater test coverage: from 3 browsers checked to 10
  • Peace of mind, knowing the all of the app’s visual elements are validated and regression-free

Summary

With Applitools Eyes automated visual UI testing solution, WalkMe substantially increased coverage, improved the amount of regression bugs caught prior to release, and saved valuable time for all teams involved.

The smart visual validation enabled them to test graphical elements that could not be tested in any other means, hours of manual testing hours were optimized, and the cloud-based dashboard with visual logs allowed for quick cross-department iteration for failed tests.

For a company who relied on providing top-notch user experience, ensuring perfect product that functions and looks as intended everywhere is the basis of their service.

Ready for the next generation of testing?

Get started today Schedule a demo