With Applitools, we can now run visual tests multiple times a week, or even a day if we need to. We have a faster turnaround cycle, and now we can catch things earlier and closer to when they were developed. I feel like even our designers and product owners could use the tool to keep an eye on things.
Jeremy Jarvis — SEPSenior Software Engineer
SEP, a software product development firm whose software is vital to keeping airplanes flying and medical devices safe, was working on a critical app for healthcare professionals. Manual visual validation meant a lot of delays and inconsistent subjective judgments about what was “good enough.”
Using Applitools Ultrafast Test Cloud, visual tests could be run over 5 times faster, and the learning curve was blazing fast.
Taking advantage of Applitools lets SEP catch bugs earlier in the process and gives the development team more time to work on innovative features and design.
SEP is a software product design and development services firm specializing in high cost of failure engagements. They provide powerful teams of thoughtful developers and designers to bring ideas to life. Founded in 1988, SEP is one of Indiana’s largest software development firms with 150 employees. Their clients span from Fortune 100 to scale-up companies. The company is 100 percent employee-owned through an ESOP and is consistently recognized for its great culture (Top Workplaces, Best Place to Work in Indiana, Techpoint Mira).
SEP was working with a leading pharmaceutical company to develop an app for healthcare professionals that would make it easier to visualize patient information. It needed to be responsive and clear across multiple breakpoints and browsers.
Manual validation, requiring manual steps around creating and executing scripts undertaken by manual testers, was complex and time-consuming. Each browser they were testing for took more than a day, and they had four browsers to support. That meant it took a test engineer about a full week just to do visual design verification.
Testing manually also required a lot of subjective judgment, which led to inconsistency. “A subjective judgment of ‘is this good enough’ had to happen multiple times,” recalled Aaron Alexander, Delivery Lead at SEP, who was leading the team doing the testing. With different engineers looking at different browsers, or even with the same engineer (whose sensitivity might change over time), this meant inconsistent results.
Due to the regulations governing the healthcare environment the app was operating in, it was also important to have a clear and accurate log of every change made, including visual ones. This was challenging to do using manual methods.
The Applitools Ultrafast Test Cloud Platform allows users to rapidly run cross browser automated tests, and crucially, generates a thorough audit trail of all visual test results. It is designed to securely deliver speed and compliance over the cloud.
When describing the value of Applitools, Jeremy Jarvis, Senior Software Engineer at SEP and Test Lead on the project, highlighted the time saved. With the Ultrafast Grid, tests could be run for all four browsers in just 30-40 minutes. Including some manual work for edge cases, the total time to test all four browsers dropped from five days to less than one.
That’s a massive increase of over 5x in productivity.
“Quality is another way I’d measure value,” added Jeremy. “With Applitools, we can now run visual tests multiple times a week, or even a day if we need to. We have a faster turnaround cycle, and now we can catch things earlier and closer to when they were developed. I feel like even our designers and product owners could use the tool to keep an eye on things.”
Getting set up was quick, and it was helpful that the Ultrafast Test Cloud requires no inbound connections, reverse proxies or IPsec VPNs. Using a single, reliable and automated tool for visual analysis also removed the consistency issues resulting from the subjective judgments of manual testers.
The SEP team working on the project shifted in the middle of development, as one of the test engineers using the Applitools Platform was pulled away. Grant Kimsey, Senior Software Engineer at SEP, replaced the original engineer and was amazed at how easy it was to get up to speed.
“The learning curve was surprisingly shallow,” Grant noted. “I just needed about 30 minutes with the previous person. The interface did more of the educating than anybody onboarding me. It was very intuitive.” Reflecting on his experience using the tool over time, Grant said that the simplicity of the purple highlighting that shows changes and the ability to ignore regions and adjust content modes made it easy to keep going once he got started. “It was all so well labeled and easy to use, I loved it,” he added.
Jeremy, who performed all the visual testing, appreciated how easy it was to incorporate and reuse the automated visual tests. “I like how I can build the test suite just once and then run the test as often as I want,” he said. “It was also great that relatively little code was needed to add Applitools to our existing tests.”
By using the Applitools Platform to catch bugs earlier in the process, the SEP team was able to iterate faster while improving quality.
“Once Applitools was in place, we were able to see things closer to development and catch them earlier. Before we had to wait until the end to manually test,” said Jeremy.
This enabled the team to shorten the feedback loop and deliver results on time.
“Applitools Eyes allowed us to compare the differences between browsers earlier than we would have manually,” added Grant, “which meant we could make fixes earlier rather than extending our timeline at the end.”
SEP continues to grow and innovate. One key goal is to find ways to increasingly embrace agile technology like Applitools that enables more complete testing, both early on and throughout the development process. This will allow them to keep shortening the review cycle, while staying within the guidelines and regulations required.