Writing a Visual AI Test Suite

In this guide, we're going to take a look at an E-Commerce site and create a Visual AI test suite that validates both the visual and functional aspects of the application.

We'll start out by going over each page, creating a plan and then executing that plan on our path to build our test suite. At the end, we'll have a fully functional test suite that we can use to release new versions of our application with confidence.

Testing plan

Let's go through all of the pages on our site, plan out our strategy and then write the test cases.

Home page

Above is the home page of our application. The home page consists of a nav bar at the top and then a product grid in the main section of the page. Based on the products we're selling, the product grid could change at any time.

Strategy

If we decided to use traditional pixel-to-pixel visual comparisons, this page would be nearly impossible to test unless we kept our prices, product names, etc. the exact same which typically isn't the case for most E-Commerce sites. With Applitools Visual AI, we can instead use Layout Match Level regions to handle the dynamic content on this page.

If you've read the guide on Match Levels you know that the Layout match level will only fail the test if the relative positioning of the elements or overall structure of the page has changed. If the content of the elements have changed, such as the price or the name, the test will still pass.

Additionally, if the content has failed to load, the test will also fail when using the Layout Match Level. If the content disappeared and you were using a traditional ignore region from a pixel-to-pixel tool, your tests would still pass allowing a bug to pass through to production.

Our strategy for this page will be:

1. Capture a full page Visual AI checkpoint with Applitools using the Strict Match Level to validate the navbar, page title and page footer.
2. Wrap a Layout region around the products in the grid to ensure our test doesn't fail from dynamic content but will still capture images or missing data that doesn't load properly.

Code

Cypress

Category Pages

The next page we need to test is a category page.

Strategy

The category pages are the exact same as the Home Page except they only show a subset of the products. Therefore, our strategy can be exactly the same as the Home Page outlined above!

Code

Cypress

Product Pages

The next page type that we need to capture are the individual product pages. These pages consist of a photo, title, description and sometimes a countdown timer indicating a sale or special deal.

Additionally, there are options for selecting a variant of the product (i.e. selecting a color) as well as adding a certain number of these products to the cart.

Strategy Part 1: Visually testing the visual aspects of the Product page

There are a few different ways we can approach testing all of our product pages:

Option 1: a generic test template

If our products are going to be in flux frequently, then we should design a generic test that assumes all of the content on the product pages are dynamic. We're not interested in the actual price data, descriptions that we've written or the photo itself. Instead, we're only interested that all of these features are present on the screen.

In this case, we can take a similar approach to the Home screen and Category pages and use Layout regions around the dynamic content including: The photo, category label, product name, countdown timer, product description, price and tags. Then, we can use this same test template for each of our product pages. As our product list grows and shrinks, we can easily scale up/down our test suite.

Option 2: Specific tests per Product

If our products don't change that often and we want to validate that the photo, price, description, category, tags, etc. are all the expected values for each product, then we need to change our approach. Instead of using Layout regions for these pieces of content, we'll let the Strict Match Level handle them and notify us if they've changed.

This way, if our description is changed, we'll be notified and can review it. Same idea for the photo and price. This will give us the confidence that key information is set to the values that we expect before releasing to production.

Code

In my case, I went ahead with option 2 as I want to ensure my pricing data, photos and descriptions are shown as expected for each product. Below is an example of one of these tests:

Cypress

Strategy Part 2: Using Visual AI for functional tests

There are a couple of things that we also want to functionally validate on the Product page:

1. Selecting a color from the variant dropdown
2. Choosing a quantity of items and adding them to the cart

Both of these cases can be funtionally tested using Visual AI checkpoints.

Case 1: Selecting a color from the variant dropdown

For this case, we want to ensure that the dropdown changes from "Select Color" to the option that we selected. To do this, we'll need to write the automation code to select a value from the dropdown and then validate the dropdown using the Strict match level.

Case 2: Choosing a quantity of items and adding them to the cart

For this case, we need to confirm that the up arrow and down arrow both work for setting the quantity in the cart as well as manually typing a value into the input field updates the cart correctly. The test flow will be as follows:

1. Click the up arrow to change the quantity to two
2. Take a Visual AI checkpoint with the Strict match level checking the quantity number
3. Click the down arrow to change the quantity to one
4. Take another Visual AI checkpoint to confirm this change
5. Type in a quantity (let's say 5) to the input field
6. Take a 3rd Visual AI checkpoint to confirm this change
7. Click the Buy Now button to add 5 of these products to our Cart
8. Take a final Visual AI checkpoint and confirm the cart icon in the top right shows 5 items in the cart.

The above flow will functionally validate that the input actions we're taking on the page are reflected correctly in the quantity of items as well as adding them to the cart. However, since we're using Visual AI we're also getting full Visual Validation that our actions have not unexpectedly changed anything else on the page.

Why use Visual AI assertions?

If we wrote traditional functional assertions using an assert library, we could check that the quantity has been updated in both the quantity field as well as the cart icon.

However, these assertions would not confirm that the cart icon shows up as expected or that the rest of the page hasn't been messed up by our actions. Visual AI assertions allow you validate both the functional and visual aspects of your application at the same time.

Code

Cypress

Cart Page

The final page that we need to validate is the cart page. The cart page shows all of the products in our cart and allows us to edit the quantity or remove them altogether. Just like the Product page above, we're going to perform functional tests as well to confirm these actions work as expected.

Strategy

Our testing strategy will be:

1. Navigate to a product and add 5 of them to the cart
2. Navigate to the cart page and take a Visual AI checkpoint to confirm we have successfully added 5 items to the cart using the Strict match level to check the quantity field
3. Click the up arrow on the quantity selector to change it to 6
4. Take a Visual AI checkpoint using the Strict match level to confirm the quantity is now 6 as well as the cart icon in the top right shows 6
5. Click the down arrow on the quantity selector to change it to 5
6. Take a Visual AI checkpoint using the Strict match level to confirm the quantity is now 5 as well as the cart icon in the top right shows 5
7. Manually edit the quantity by typing 3 into the input field
8. Take a Visual AI checkpoint using the Strict match level to confirm the quantity is now 3 as well as the cart icon in the top right shows 3
9. Click the remove button to clear this product from our cart
10. Take a Visual AI checkpoint using the Strict match level to confirm that the empty cart message is shown and that the cart icon has been updated to reflect the empty cart

Once again, we're using Visual AI checkpoints to test the functional in addition to the visual aspects of our page ensuring that we get maximum coverage. This will prevent both functional and visual bugs from entering into our checkout flow. If our checkout flow doesn't work as expected, it will directly effect our business so we can't afford to cut corners on coverage!

Code

Cypress

Test Results

We've now written all of our test cases for our application. You can find the full test suite code in this public Github repository. In this last section we'll examine each of our test results

Home Page Result

As we can see above, we've executed our strategy of applying a Layout region to our product grid.

Above, we see what happens when one of our products is removed and our product grid order changes. Instead of our tests failing due to this change, the Applitools Visual AI with the Layout Match level is smart enough to understand the Layout of the Product Grid is the same and only the content has changed. Therefore, the test passes.

Category Page Result

We see similar results for the Category page.

Product Page Results

Initial Visual Checkpoint

Image 1/6

For the product page, we have a number of results that validate our visual and functional test cases outlined earlier.

Cart Page Results

Products In Cart

Image 1/5

For the cart page, we also have a number of results that validate our visual and functional test cases outlined earlier.

Conclusion

In conclusion, designing and implementing a Visual AI test suite for an E-Commerce site ensures both the visual and functional aspects of the application are up to par. By meticulously planning and executing test cases for every page of the website, and using Applitools Visual AI, we've created a comprehensive strategy for handling dynamic content and validating key information before releasing to production. Whether a layout change, price update, or any variation in the product details, our testing process keeps up with these changes, making this suite an integral part of an effective quality assurance system. This holistic approach manages to not only streamline our operations but also ensures an optimal customer experience, thereby positively affecting the bottomline.