Organizing tests with Selenium IDE

Selenium IDE has three entities that you can use to organize your tests. A project is a collection of tests. Each test is a script that executes command actions that drive a website by sending it mouse and keyboard events, and other commands that access the website and examine elements, to assert or verify their state. With the addition of the SIDE Eyes, there are additional commands to configure the visual testing and to add visual checkpoints that capture a screen image and send it to the Eyes server for processing. A Selenium IDE test suite is a set of tests in a project that are run together as a single entity. To summarize, the project is a list of related tests, and a test suite collates subset of these tests to be run together as a unit.

The diagram below illustrates the relationship between the Selenium IDE terms project, test and test suite to Eyes terminology of application, and test, names and a batch. On the left, you can see a Selenium IDE project named 'p1' that consists of 4 tests t1, t2, t3 and t4, and two tests suits 'A' and 'B'. Test suite A runs tests t1 and t2, and test suite B runs all the tests.

When a test suite runs, the result is a batch of the same name. The batch contains the results of running all the test in the test suite.

The Eyes test result has two properties called the application name and the test name, SIDE Eyes uses the Selenium IDE project name as the application name, and the Selenium IDE test name as the Eyes test name. If you request to run all the tests in the project, without explicitly creating a test suite, then SIDE Eyes names the batch All Tests.

Checkpoints

When you run a Selenium IDE script, each calls to eyes check is called a checkpoint. The reference baseline for the test has nominally the same number of images as there are checkpoints in the test, and these are ordered in the same order as the checkpoints are executed. For a test to pass, every checkpoint in the test has to match the corresponding image in the baseline.

As application functionality evolves, checkpoints image content can change, and checkpoints can be added or removed. Eyes compares the sequence of checkpoints to the sequence of images in the baseline, and the test results consist of a sequence of steps, where a step indicates one of the following:
  1. The checkpoint and baseline images match.
  2. The checkpoint and baseline images do not match (mismatch).
  3. There is a checkpoint without a corresponding baseline image, which is presumably new.
  4. There is a baseline image without a corresponding checkpoint; the checkpoint is presumed missing.

The user uses the Eyes Test Manager to view the steps and can accept the new and missing steps, as well as changed steps that are due to expected changes. He can then save the baseline so that the next time the test is run it fully matches the baseline. If a mismatch is due to a bug, or the new or missing checkpoint is not expected, then the user can reject that step. For more information see How Eyes compares checkpoints and baseline images.

Summary

To summarize, when you consider how to organize all the things you want to test using Eyes checkpoints, you might find it useful to consider the following:
  • Checks that are always tested together, or follow a flow where one check build on the outcome state of the previous check are usually best implemented in a single test. Every run of a test should have the same number of checks, executed in the same order. Eyes detects new and missing checkpoints.
  • Since the status of a test (passed, failed or unresolved) is a function of the status of all its steps, it is a convenient way to view the outcome of the test run. If the steps in a test are logically related, then it can be easier for a user to monitor the status of ten tests, with 5 steps each, then to monitor the status of 50 individual tests.
  • The converse side to putting many steps in a test is that sometimes you don't want to test everything. After a small change, you may decide to check just the impacted features with a short "sanity check". You may decide to run, a more complete test once a day, and run a full coverage test before delivery. The IDE test suite allows you to create subsets of tests that can easily be run together, as a unit. Using the Eyes Test Manager, you can view the batch which provides a summary of the all the checks in all the tests in a suite, and allows you to handle the results of each step or test individually, as well as to handle all, or a subset of the steps and tests as a unit.

Further reading

For more information in what you can do with batches see How to organize your tests with batches