How do you run cross browser testing today?
Applitools released a new product and it looks to be rather innovative. Focused on visual testing, Applitools first caught my eye with a 2015 demo which showed how their algorithms could detect the difference between a layout modification and a content modification on a webpage, quite the step up from pixel-perfect image diffing that other early visual UI testing tools were doing.
Their new product, Visual Grid, drastically improves the speed of running a suite of cross browser visual UI tests. Visual Grid improves speed and reduces cost for all teams using visual web testing, but I’m personally excited about the opportunity for promoting visual testing habits directly to frontend developers. Frontend developers can now get instant feedback for their changes across an array of desktop and mobile browsers. Applitools bets that if the total time of visual UI test suites can be reduced to seconds or minutes, they can be fit into the instant feedback cycle employed by modern developers while actively coding. There’s a huge difference between waiting an hour to find out if your commits broke something and waiting a minute. Retaining a developer’s flow-state boosts productivity, and reduces frustration.
I tested a beta version of Applitools Visual Grid and it delivered on its promise. I wrote up a script to visit every single page and article of our Appium Pro newsletter on two different browsers, with three different screen resolutions. The entire process took less than three minutes!
62 pages * 2 browsers * 3 screen sizes
= 372 images captured and verified in less than 3 minutes
Crossing the 1-3 minute mark for an entire cross-browser test suite took considerable technical effort on the part of Applitools, and Visual Grid takes a novel approach that, as far as I’m aware, nobody else is offering. The crux of their innovation is that a single browser is automated to crawl a site locally on the developer’s machine while a fleet of machines in their cloud take screenshots and compare them to past images.
The developer opens up Chrome locally and visits example.com, the Visual Grid client then sends all of the HTML elements, images, text, etc currently displayed in Chrome to the Visual Grid Cloud. The Applitools browsers in the cloud load the same HTML assets and take a screenshot, effectively testing the layout and rendering in every browser and screen size selected. The traditional way of gathering these comparison screenshots is to use a Selenium cloud provider, spinning up an entire automation session on each browser for testing, navigating to example.com, loading the site, and taking a screenshot.
Avoid The Headaches of Cross-Browser Testing
The actual user actions being automated on each webpage are only performed locally on a single browser, while screenshots are rendered and taken in the cloud. You shouldn’t have to pay for Applitools’ remote browsers to click on checkboxes and fill in forms, with Visual Grid you only use their cloud resources for rendering the screens generated by the automation script running locally.
When running automated tests on your site, are you concerned with testing whether Firefox correctly scrolls a combo box, or Safari can put placeholder text in a textbox? Of course not, what we’re all testing for is that the site looks right in every browser, and all our controls are visible. Visual Grid cuts out the overhead of testing browser actions, allowing us to run visual regression tests extremely quickly.
For those familiar with the current paradigm on using Applitools along with a cloud of automated browsers, here are three cases which demonstrate the difference when using Visual Grid:
Case Study 1: Random Number Generator
You are testing a single web page which displays a random number on every page load. No two users will receive the same number. Your automation script opens the page locally using Chromedriver and tells Applitools Visual Grid to check the UI on Chrome and Firefox, with 800×600 resolution and also 720×1280 resolution. The Visual Grid takes screenshots on the four browser/resolution combinations. Do all four images contain different random numbers or the same random number?
All four screenshots will show the same random number which was displayed on the developer’s local Chrome instance. The Visual Grid browsers did not visit the web page in question. Instead, a full copy of the HTML elements, styling, images, etc (the HTML Snapshot), is sent to each of the Visual Grid browsers to render.
Case Study 2: IP Address
You are testing a single web page which displays the IP address of the browser. Your automation script opens the page locally using Chromedriver and tells Applitools Visual Grid to check the UI on Chrome and Firefox, with 800×600 resolution and also 720×1280 resolution. The Visual Grid takes screenshots on the four browser/resolution combinations. Do all four images contain different IP addresses or the same IP addresses?
As with Case Study 1, all four screenshots will show the IP address of the developer’s local computer. It’s interesting to note that the site’s server will only log a single HTTP request emanating from the local Chrome instance, the four Visual Grid browsers never send requests to the site.
Inserting Visual Grid Into The Testing Pyramid
I see Applitools Visual Grid as occupying a new layer in the so-called “Testing Pyramid”.
As we move up the pyramid, tests take longer to run and are less reliable but exercise the entire software stack more rigorously. Visual UI tests using Applitools Visual Grid are much faster and less flakey than full browser automation tests, and this translates to cost savings for users. I love how Visual Grid raises the bar for tests which can provide feedback to frontend developers while they are actively coding.
For more detail, including a great explanation of the Testing Pyramid and more detail on how Visual Grid is implemented, see this Webinar by Applitools Architect Gil Tayar. I support Gil’s opinion that in order for a software organization to produce a high-quality product and be able to iterate quickly, developers need to write and run their own tests. Applitools Visual Grid enables frontend developers to work more closely with their visual UI testing.
Want To Know More About Applitools?
Other reading for you: