If you’ve been following the quality engineering community over the past couple years, you’re probably familiar with Test Automation University (a.k.a TAU). New technologies require new skills — it’s a constant for all of us in today’s world. This is why TAU now boasts over 40 free on-line courses for emerging quality engineering techniques including an Introduction to Cypress by Gil Tayar or Scaling Tests with Docker by Carlos Kidman. And most relevant to this blog are courses on Visual AI including Automated Visual Testing by Angie Jones and Modern Functional Test Automation by Raja Rao DV.
Our main goal in creating a free, open Test Automation University was to help the global test engineering community upskill routinely and have fun doing it. We also recognized the need to help that community understand how and where Applitools Visual AI would fit in. Too often, emerging technologies ask you to rip and replace everything. Not only is that hard to do, but it’s also not realistic. Your team has invested a lot of time and money in their quality management process to date. You’re not going to just throw that all away every one or two years for the latest shiny new tech or trend. Before you make a change, you need to be confident that it will evolve your team from where you are today, yet integrate easily with what is already there.
That’s why we ran the Visual AI Rockstar Hackathon recently. We’re Applitools! Of course we’re going to talk about the wonders of Visual AI. It’s much better, and far more credible, if 288 of your peers in quality engineering do a side-by-side comparison of Visual AI and Selenium, Cypress, or WebdriverIO, then tell you about the wonder of Visual AI themselves. Over the next seven weeks, we will drill into what we learned in a series of blog posts starting this week with test creation time.
Or– if you just can’t wait that long — go ahead and grab the full report here. No better time than the present to learn and upskill for the future.
The obvious question — why is it so much faster to author new tests using Visual AI? It’s because Visual AI uses just a single line of code to take a screenshot of the entire page. You’re still automating the browser to click and navigate, but replacing a huge number of code-based assertions with just a simple line of Visual AI code to test for UI elements, form fill functionality, dynamic content, even tables and graphs. With this modern technique, you’re now authoring tests much faster than before and can use the time you save to both test more and test faster.
To help us understand Visual AI’s impact on test automation in more depth, we looked at three groups of quality engineers including:
By comparing and contrasting these different groups in the report, we learn more about the impact of Visual AI on test creation time.
The main point behind this data is subtle, but important. The average test-writer took 7 hours to write conventional code-based assertions that only covered, on average, 65% of potential bugs. How likely will that coverage result in rework?
World-class tests cover 90% or more of potential failure modes. For code-based assertions, the Grand Prize winning submissions achieved, on average 94% coverage. In order to do a “Grand Prize Winning” submission, quality engineers had to increase their test creation time commitment from 7 hours to 10.2 hours. That’s another 3.2 hours of effort to achieve acceptable test coverage!
Contrast code-based coverage with Visual AI results. Using Visual AI, all the testers achieved 95% or more coverage in, on average, 72 minutes of coding. It took the Grand Prize winners only an additional 6 minutes to achieve 100% coverage using Visual AI. This trend continues when you compare the data across all 3 groups like we did here:
This table shows that using Visual AI speeds up all test code development.
Conventional wisdom says that you should spend time on being sufficiently productive. So, should you be writing code-based assertions to validate your functional tests? In a real world setting, our data shows that, in contrast with Visual AI, code based assertions slow down releases to ensure that quality engineers provide sufficient coverage. As a practical response, your test team suffers because quality-time tradeoff either makes them a bottleneck or a source of quality concerns.
By including Visual AI in their approach, quality engineers obtain the same amount of coverage 7.8x times faster. Even better, they can use this found time to better manage quality.
The data we collected from Hackathon participants makes one clear point: no matter your level of coding expertise, using visual AI makes you faster.
Each of the testing use cases in the hackathon – from handling dynamic data to testing table sorts – requires you utilize your best coding skills. We found that even the most diligent engineers encountered challenges as they developed test code for the use cases.
For example, while your peers can easily apply test conditions to filter a search or sort a table, many of them labor to grab data from web element locators, calculate expected responses, and report results accurately when tests fail.
We encourage you to read The Impact of Visual AI on Test Automation. We also encourage you to sign up for a free Applitools account and try out these tests for yourself. But, if you do nothing else, just check out the five cases we researched and ask yourself how frequently you encounter these cases in your own test development. We think you will conclude – just as our test data shows – that Visual AI can help you do your job more easily, more comprehensively, and faster.
Cover Photo by Shahadat Rahman on Unsplash
