Our quality increases exponentially with Applitools. We run it with every build - which takes around five minutes. Without Applitools - the process would take 4 hours per build for less coverage than what we do now in 5 minutes - simply not something we could afford
Walt Harris — MedalliaHead of Quality
Medallia, an AI-driven customer engagement platform was looking for a UI testing solution that would reduce manual testing time and increase automated UI coverage.
Usage of Applitools grew from 2 to 20 engineers within Medallia as they discovered the ease of test automation that his team had not previously experienced.
As Medallia continues to look for ways to improve efficiency, they have started to validate their entire component library with the codeless Storybook integration saving time writing and maintaining brittle test code.
Medallia offers it’s AI-driven ONE Platform, a transformational technology for customer engagement and the recognised global leader in the Customer Journey Orchestration and Analytics market. The ONE Engagement Hub helps global brands build customer engagement in the era of digital transformation.
The ONE Engagement Hub is a quick-to-deploy SaaS solution designed to discover customer insight in real-time, across every interaction throughout the entire customer journey. ONE then uses that insight to help brands deliver individualised, relevant, and consistent conversations across every touchpoint, providing a completely seamless experience throughout each unique customer journey.
Medallia provides its users with great insights into customer behavior. To continue to improve user experience with their highly-visual web application, Medallia develops continuously. How does Medallia keep this visual user experience working well? A key component is Applitools.
Walt Harris, head of quality at Medallia, notes that Medallia had no visual testing solution in place when they first tried Applitools. He recognized that it might be nice to use a visual validation solution. Applitools seemed to be a leap above anything they might have considered otherwise. Walt and his engineering team all had experience with pixel diff tools and had long ago given up on them.
What seemed “cool” was Visual AI, the internal Applitools technology that can consider higher-order elements above pixels. By inspecting visual elements, Applitools Visual AI identifies user-noticeable differences while ignoring pixel differences. This makes Applitools much more accurate across subtle browser and operating system changes that pixel diff tools report.
Frankly, Walt was curious to see how Applitools would contribute to his existing test infrastructure.
Walt’s team wrote plenty of Selenium tests to validate both UI behavior and end-to-end functionality. They had no automation for the visual components of these tests. The best they could do was write tests that applied a test condition and used Selenium to inspect the output HTML. By checking various IDs, names, CSS selectors, or XPATH identifiers, Medallia could inspect the code used to generate their UI.
By validating HTML, they could capture every element on the screen. However, small changes to UI code would require significant rewrites to get tests to pass again. Frustratingly, many of these changes to internal element locators had no visual impact.
As they started using Applitools, Walt and his team discovered an ease of test development they had not previously experienced. All that code used to validate content on screen got replaced by a simple call to Applitools, which captured the screen and compared it with the last good screen capture. Any differences got highlighted. Walt discovered that Applitools reduced his coding load. It also reduced his test times. Instead of holding a page state to capture each relevant element one at a time, Applitools ran a single capture inline and the comparison offline.
Once the initial users started talking about Applitools, more engineers wanted to try it. While two users worked on the pilot, there are now 20 engineers who use Applitools to validate their code.
The Applitools customer support team provided excellent assistance when needed. Walt said,
“It’s been great working with support and the customer success team over the years. As an engineer, I can’t tell you what a difference it makes to work directly with other engineers after getting a quick response.”
Applitools now provides significant productivity gains to Medallia. According to Walt, the key UI tests run regularly after every build continues to grow as their codebase grows. Before they switched to running the tests with Applitools, the tests would take 4 hours to complete. That run time led Medallia to question whether to change their strategy of testing after each build.
“With Applitools, the same tests that used to take 4 hours now take 5 minutes,” Harris said. Medallia can validate after every build because their test time with Applitools can give them a real-time acceptance after the build completes.
Another benefit has come from moving visual validation from final acceptance to code check-in. When visual tests were run at final acceptance, developers often had moved onto other projects. They had to context switch back to old work, make repairs, and then switch back to their current project. This switching made them less efficient. By running visual validation at code check-in, all functional and visual discrepancies get discovered in context. Bugs get discovered and repairs happen without developer context switching.
For Medallia, the benefit is faster code turns. They can create builds much faster because they have code validation with less code and greater coverage. Medallia can use its engineers to create new features in their ONE customer engagement platform, rather than have them engaged in testing and late-stage bug fixing. Medallia can use their time more effectively – enabling them to create more features and deliver more value to their users.
As Medallia looks to improve overall efficiency, they build and maintain their component library in Storybook. The component library, along with a series of mock-ups, helps Medallia’s engineers document and prove user interface functionality in isolation and provides a platform for collaborative feedback with stakeholders.
To validate the library as they continue to develop it and add to it, Medallia has used Applitools. Kai Johnson, who implemented Storybook for Medallia’s component driven development process, said that Applitools was the key to automated testing of these UI components.
“With Applitools on Storybook, our team gets no-code and low-code automated UI testing so our engineers can focus on developing a state-of-the-art user interface rather than get bogged down in writing tons of brittle test code.” Johnson said.
And, Kai noted, the key testing requires visual validation. With their previous testing tools, visual validation would take tens of minutes, if not hours, to run. Medallia’s Storybook Applitools tests are integrated into their build process alongside other unit tests and are done in under five minutes
Medallia has 20 developers who use Applitools regularly. Because each developer can see the visual impact of his or her coding immediately at check-in, developers have made huge efficiency gains. Engineers discover and fix visual errors in the context of their development cycle.
Because they catch visual issues at check-in, Medallia now uncovers virtually no issues in end-to-end pre-release tests. “We’re thinking about taking Applitools out of those end-to-end tests because we haven’t caught a visual issue in those tests in a long time,” Harris said.
Of course, Harris said, he realized that he would not do that. “Applitools give our entire team huge peace-of-mind,” Harris said. “We know about the visual impact of our code. Using Applitools makes us more nimble and lets us develop more quickly.”