How Eyes compares checkpoints and baseline images

As the application under test evolves, so do its tests. New checkpoints are added to test new functionality and checkpoints are removed because the application functionality or flow has changed.

Instead of having a static association between every checkpoint and a corresponding baseline image, Eyes compares a sequence of images captured by checkpoints to a sequence of baseline images, allowing for the fact that checkpoints may have been removed or added. Then, very much like a textual "diff" program, Eyes find images that have been added and removed as well as images that have a corresponding baseline image and either match or mismatch. Eyes then gives the user the tools to both verify existing checkpoints as well as to ensure that the set of baseline images is up-to-date.

Tests results and steps

When a visual UI test executes, Eyes verifies the entire sequence of checkpoints against the sequence of baseline images. The result of this verification, the test result, consists of a sequence of steps, each denoting a missing checkpoint, a new checkpoint, a checkpoint that matches its baseline counterpart or one that differs from it.

For example, the baseline of a test may consist of the following five images:

Now imagine that the application and test have been changed, and the test generates a different sequence of images, for example:

Comparing the two sequences, we can see that:

  • Image B has been removed
  • Image C has changed to C1
  • Images F and G have been added
  • Images A, D, and E have not changed and are in the same order

Eyes will detect these differences, and it will generate a sequence of steps, as follows:

Each such step represents one of the four possible outcomes:

  • Match: Eyes has found a baseline and checkpoint image that match each other.
  • Diff: Eyes has found a baseline and checkpoint image that correspond to each other, but there are mismatches.
  • Missing: Eyes has detected that an existing baseline image has no corresponding checkpoint image.
  • New: Eyes has detected a new checkpoint image, i.e., an image in the captured sequence which has no corresponding

Notes

While you can freely add and remove checkpoints, Eyes assumes that the order of captured images that are not new does not change. If you change the order of two checkpoints, then depending on the content of these checkpoints, Eyes could report two missing steps and two new steps, or it could report two steps with mismatches.

When you define a checkpoint in the code, you can optionally provide a name for that checkpoint. The name is intended to be descriptive, so as to allow users to easily and conveniently navigate and search within the test results. Eyes does not take the name into account when matching checkpoint and baseline images. The name is optional, does not need to be unique, and you can freely change it without impacting the testability of the code.