Wix

Highlight

Applitools Eyes, an easy-to-integrate automated visual UI testing solution, was used by Wix.com‘s Test Automation Team for inspecting aspects in their HTML 5 Website Editor and generated sites. Benefiting from Applitools Eyes algorithm that mimics the human eye, the Wix team was able to reduce the number of regressions & UI bugs as well as cut down manual testing efforts.

About Wix.com & its Test Automation Infrastructure

Wix Automation team is using several test solutions, written in Java, JS, angularJS, using Selenium test framework, and triggered by Team City CI server, for testing products that are built using a variety of coding languages.

The Wix platform’s core product – its code-free HTML5 Website Editor – is a complex and feature-rich solution, enabling users to create a beautiful and dynamic web presence themselves. It includes thousands of commands and features, all of which are being tested daily by the company’s QA and Automation departments.

When the company’s Automation team realized the necessity of adding frontend visual UI testing, providing a screenshot comparison of the app’s viewer side, to the overall set of tests that covers Wix’s web products, a visual UI testing solution was developed in house. The goal was to enable QA to compare “before and after” screenshots to easily detect regressions and visual defects in the Editor.

However this solution didn’t answer the company’s needs. Tests often created “false positives”: any pixel change – even those undetectable by human eyes – caused tests to fail. As a result QA had to deal with not only real bugs and regressions, but also with tests that failed for no meaningful reason.

Challenges

• Wix.com’s Editor is a complex product, requiring thousands of tests to run daily
• Editor must be tested on variety of browsers, screen resolutions and operating systems
• Manual testing does not catch all visual bugs and UI regressions
• In-house visual UI testing tool was too sensitive, and was hard to monitor and maintain
• Any test solution must integrate with Selenium, TeamCity and in-house automation tools
• Testing tools must be Continuous delivery friendly

Solution

The Wix Automation team decided to implement Applitools Eyes which automatically validates all UI elements across devices, browsers and form factors. Applitools Eyes differs from other screenshot comparison tools due to its unique image matching algorithm that mimics the human eye: this means that tests are robust, do not generate “false positives” and are easy to create, maintain, and update.

Implementing Applitools Eyes in Wix’s existing Selenium test automation framework and TeamCity Build Server was quick and within days the Automation team was utilizing it to its full extent. Wix’s teams quickly realized the value of Applitools Eyes’ robust, automated test suite: it saved valuable resources by automatically detecting UI, layout and functionality issues that needed attention, while ignoring those that are insignificant to human users.

Two features that proved most valuable to Wix are “Ignore Regions”, allowing to label regions to ignore (usually where dynamic data is placed), and “Floating Regions” (used to mark areas that can move within a page). Regions placed on any UI page are automatically applied to other UI pages that include similar elements thereby substantially reducing maintenance times.

Furthermore, when Wix makes a change in the Editor that affects multiple pages in the same way (e.g. change in navigation), they only need to review and accept only a single occurrence of this change; Applitools Eyes scans all other pages and automatically approves similar changes.