Case study

Wix.com: a leading cloud-based web development platform with over 62 million registered users worldwide

Applitools Eyes Successfully Used by Wix.com to Prevent Regressions & Reduce Manual Testing

Highlight

Applitools Eyes, an easy-to-integrate automated visual testing solution, was used by Wix.com‘s Test Automation Team for inspecting aspects in their HTML 5 Website Editor and generated sites. Benefiting from Applitools Eyes algorithm that mimics the human eye, the Wix team was able to reduce the number of regressions & UI bugs as well as cut down manual testing efforts.

"Applitools Eyes made testing our Editor product much better" – Gady Keren, Automation Guild Manager at Wix.
About Wix.com & its Test Automation Infrastructure

Wix Automation team is using several test solutions, written in Java, JS, angularJS, using Selenium test framework, and triggered by Team City CI server, for testing products that are built using a variety of coding languages.

The Wix platform's core product – its code-free HTML5 Website Editor – is a complex and feature-rich solution, enabling users to create a beautiful and dynamic web presence themselves. It includes thousands of commands and features, all of which are being tested daily by the company's QA and Automation departments.

When the company's Automation team realized the necessity of adding frontend visual testing, providing a screenshot comparison of the app's viewer side, to the overall set of tests that covers Wix's web products, a visual testing solution was developed in house. The goal was to enable QA to compare "before and after" screenshots to easily detect regressions and visual defects in the Editor.

However this solution didn't answer the company's needs. Tests often created "false positives": any pixel change – even those undetectable by human eyes – caused tests to fail. As a result QA had to deal with not only real bugs and regressions, but also with tests that failed for no meaningful reason.

Challenges
  • Wix.com's Editor is a complex product, requiring thousands of tests to run daily
  • Editor must be tested on variety of browsers, screen resolutions and operating systems
  • Manual testing does not catch all visual bugs and UI regressions
  • In-house visual testing tool was too sensitive, and was hard to monitor and maintain
  • Any test solution must integrate with Selenium, TeamCity and in-house automation tools
  • Testing tools must be Continuous delivery friendly
Solution

The Wix Automation team decided to implement Applitools Eyes which automatically validates all UI elements across devices, browsers and form factors. Applitools Eyes differs from other screenshot comparison tools due to its unique image matching algorithm that mimics the human eye: this means that tests are robust, do not generate "false positives" and are easy to create, maintain, and update.

Implementing Applitools Eyes in Wix's existing Selenium test automation framework and TeamCity Build Server was quick and within days the Automation team was utilizing it to its full extent. Wix's teams quickly realized the value of Applitools Eyes' robust, automated test suite: it saved valuable resources by automatically detecting UI, layout and functionality issues that needed attention, while ignoring those that are insignificant to human users.

Two features that proved most valuable to Wix are "Ignore Regions", allowing to label regions to ignore (usually where dynamic data is placed), and "Floating Regions" (used to mark areas that can move within a page). Regions placed on any UI page are automatically applied to other UI pages that include similar elements thereby substantially reducing maintenance times.

Furthermore, when Wix makes a change in the Editor that affects multiple pages in the same way (e.g. change in navigation), they only need to review and accept only a single occurrence of this change; Applitools Eyes scans all other pages and automatically approves similar changes.

Results
  • Substantially better, more robust automated visual testing solution
  • With Applitools Eyes, every release requires just 5 hours of manual QA, compared to the ±70 hours of manual testing needed before
  • Greater test automation coverage
  • Improvement of overall number of regressions & UI bugs caught pre-release
  • Peace of mind with each new release and update, knowing that all of the app's visual elements are validated and are regression-free

Following the addition of Applitools Eyes Automated Visual Testing Solution to Wix Selenium-based Test Automation infrastructure, automated visual testing is now a daily part of the Wix test suite. This substantially improved QA ability to catch regressions & UI bugs, while radically cutting manual testing efforts & development resources.

Summary

With Applitools Eyes automated visual testing solution, Wix managed to substantially reduce the amount of UI regression bugs, while drastically reducing manual testing efforts. In addition, Wix reported substantial time saving for all departments involved: developers, test engineers and DevOps.

Manual regression cycles are now leaner and more agile allowing the QA team to focus on deep functionality and progression testing instead of acceptance and visual testing.

"Applitools Eyes made testing our Editor product much better, It saves us time and enables us to catch more regression bugs than manual QA would. The implementation of Automated Visual Testing allowed us to achieve greater automated UI test coverage, and to automatically detect many UI layout and, flow issues that flew 'under the radar' before." – Gady Keren, Automation Guild Manager at Wix