We have found the right toolset. With Applitools we are moving in the right direction, and we can get there faster as more and more teams adopt Applitools.
Siddharth Kaushal — FrontdoorDirector QA
Frontdoor, the parent company of home warranty giant American Home Shield, wanted to make it easier for the customer to connect and engage with the company. This meant many more OS/browsers had to be supported, which overwhelmed the existing team that relied on manual testing.
Using Applitools Ultrafast Test Cloud, release velocity has more than doubled, while maintaining total coverage for all browsers and native apps.
Thanks to the easy learning curve, Frontdoor was able to quickly scale their test automation team and incorporate automated Visual Testing into their testing process without having to worry about false positives. This rapidly improved test coverage for all customers and gave the testing team more time to focus on other key goals.
Frontdoor is the #1 company for home warranties, providing essential services to customers to help them maintain their home appliances and systems for nearly 50 years. Frontdoor intends to disrupt the US home services business by leveraging technology to go digital-first and deliver innovation faster for their customers by developing a modern, customer-driven platform on a microservices architecture.
American Home Shield, Frontdoor’s largest brand, has a large and varied customer base, and they wanted to make it easier for the customer to connect with them more frequently. “We wanted to make the business more interactive,” explained Siddharth Kaushal, Director QA. To accomplish this the Frontdoor team set about creating modern microservices to replace parts of the existing monolithic architecture, opening the door for innovative user engagement across numerous additional operating system and browser combinations.
However, to get the user experience right, this meant that suddenly there was an enormous increase in the number of tests required to ensure full coverage. “Even with multiple excellent manual testers or functional experts, this became really difficult,” noted Siddharth. At that point, the team realized they’d have to embrace UI automation, but they also knew they needed Visual Testing. “Based on our experience as QA Engineers, we knew that unless we had Visual Testing too we’d be spending unnecessary extra time on automation, and we might still require a lot of time-consuming manual testing on top of that.”, noted Nikhil Chandwani, Automation QA Engineer.
One of the complexities for the Frontdoor team was that while supporting the latest platforms was critical, many of their customers as well as the contractors supplying the repairs continued to use older browsers or devices. The team never wanted to make a contractor conduct a potentially time-consuming or expensive update just to continue doing business with the company. As a result, getting the experience right meant covering everything that was new without breaking compatibility with the old.
With the rising number of OS/browser combinations that needed coverage, there was a strong business case for automating Visual Testing. Moving from a manual focus to an automation focus meant a new team would have to be hired and trained and being able to get up and running quickly was essential.
Siddharth, Nikhil and other members of the Frontdoor team had used Visual Testing tools before – both commercial libraries and open-source alternatives. These manual tools utilized a pixel-by-pixel snapshot comparison strategy that was ultimately far too sensitive for their needs, with frequent issues around test flakiness and false positives. Test maintenance after making even minor changes to the baseline became a frustrating challenge.
Once Siddharth tried Applitools, he knew he’d found something that could help them get automation going fast.
“The Applitools learning curve and its features are simple enough for anyone to learn in just a day. It’s not limited to QA testing – developers can use it too for shift-left testing,” Siddharth said, noting that it was so easy to learn that “even the product and marketing teams could use it to verify any changes they make as they go.”
Of course, a quick learning curve by itself isn’t enough – the Visual Testing platform needs to meet the business case, in this case by ensuring timely coverage of all OS/browser combinations.
The Frontdoor team has at least two dozen combinations that they focus on, prioritized by what their users report using most. But they can’t leave any stone unturned. “For us, even .02% usage matters,” Siddharth emphasized. “If we sign a new vendor using an old browser and they can’t access our application, it’s a miss for us.”
With Applitools and Visual AI, Siddharth and his team save a lot of time compared to using manual testing tools, and this gives them the ability to truly provide complete coverage without cutting back on other important goals the team is working on. Velocity has more than doubled, with the team improving from testing two sprints behind development (n-2) to one sprint (n-1) or less. As testing shifts further left and closer to development Siddharth expects the velocity to improve even further towards in-sprint testing.
A key advantage of the Applitools platform is that in addition to Visual AI helping the team identify visual bugs more easily and with less test maintenance, the Applitools Ultrafast Test Cloud helps to dramatically speed up cross-browser and cross-device testing. It allows Visual AI to run at scale, with fast feedback received quickly (often in seconds compared to hours without it) across different browsers and devices, so the team can shift left and catch bugs closer to when the change was introduced. Given the breadth of offerings American Home Shield provides, the Ultrafast Test Cloud plays a critical role in making sure testing is both complete and extremely fast.
Siddharth was particularly excited to use Applitools to test a native mobile app. “Applitools gives us a quicker way of looking at all possible screens without worrying about how the object is placed in the DOM,” he said. Unlike with some other tools, “we don’t have to change our tests with every device. We are really looking to Applitools to accelerate our native mobile testing.”
Applitools provides an audit trail of all changes that the Frontdoor team has found useful as well. It effectively archives all digital assets ever used and tested on the site for future reference. Now, when someone asks if a particular OS/browser combination is covered or what a page looked like previously, there is something his team can quickly refer to and demonstrate the coverage or change. “I’ve had several requests from senior management just recently” with this kind of question, Siddharth noted.
This also helps the testing team keep track of what’s been done and what tests have passed or failed across the site, even outside of the software development cycle. Nikhil appreciated that Applitools made it possible for the QA team to shift right and validate changes in production – such as content edits by the sales and marketing teams. Visual AI ensures that minor content updates aren’t flagged as failed tests, but that if important formatting breaks (like a hyperlink not appearing in bold due to a typo) the test will fail, and the team will know. “Now we don’t have to worry about all these pages anymore,” he said, “and this saves a tremendous amount of effort.”
The Frontdoor team continues to innovate and improve their automation coverage and velocity. They plan to get even faster by moving to in-sprint testing and incorporating Applitools directly into the content pipeline for instant validation, enabling the team to rapidly improve the customer experience and deliver innovation to their users faster.
Enhancing accessibility testing and supporting localization efforts to enable expansion into new geographics are also exciting future improvements. Another key goal is to use Applitools to collaborate more closely with the marketing team on high-impact efforts like A/B testing of the UI to increase conversion rates, and confidently ensuring all content changes create an excellent user experience on any device.