Test and batch status

The test and batch status are displayed in the Eyes Test Manager. The test status changes to reflect the resolution decisions made by the Tester for each step in the test. The batch status changes to reflect the status of the tests in the batch. In some cases Eyes provides automatic resolution, the default values can be changed using the SDK

Test and batch status in Eyes Test Manager

Test status

The test status is displayed in the Test detail view and is composed of the four components described below.

• The text of the status (e.g. Running, New, Passed, Failed etc. see below)
• An icon that, if it appears, indicates that at least one step had mismatches.
• An icon that, if it appears, indicates that the Tester pressed accept or reject on every single step
• The color of the text and icons (blue: Test still running, Orange test unresolved, green test passed, red test failed)

The following table defines the possible test state values:

Test details view	Batch steps by test view	Meaning
		The test is currently running. No further information is available at this point.
		This is a new test, i.e. there was no previous baseline. By default, all steps are considered passed, the baseline is updated from the images captured in the test run, and the check mark is visible in the test status, even though the Tester did not actually accept or reject every step. The status will only be red if the test was explicitly failed by the tester.
		This is not a new test . The test passed either implicitly (no diffs in any step) or explicitly by the Tester.
		The test contains at least one unresolved step, i.e. at least one step had a status of different (), missing () or new () and the Tester has not accept or reject it explicitly.
		The Tester resolved all tests (otherwise it would be orange) and has rejected at least one step (upper icon) Tester directly set the test status to failed (the lower icon)
		The test aborted, before all the steps where completed. Initially this will be orange (i.e. need resolution). The final color and the presence of the icons depends on what the tester did as described above in the other cases.

Manual override of the test status

The test status is also button, if the test has been resolved, you can press this button, and then a dialog will open as follows:

Choose from one of the following 3 options:

	Force the test to be passed irrespective of the pass/fail status of its steps
	Force the test to be failed irrespective of the pass/fail status of its steps
	The pass/fail status should have the default value derived from the state of its steps

Batch status

The status of a batch is derived from the status of all the tests in the batch. The batch status appears in two places in the Eyes Test Manager:

• It is displayed as part of every entry in the batch list
• It is displayed for the current batch entry.

The status has two parts - the Matching Status of the run (has the test run, did it pass or fail) and the Baseline Status - has the Tester made any baseline changes and do they need saving.

The Matching Status

The following table summarizes the possible batch status values in order of precedence, and how they are derived from the status of the batch tests:

	At least one test in the batch is unresolved. i.e., has mismatches or has missing or new steps that the Tester has not explicitly Accepted or Rejected.
	All the tests have been resolved, and at least one has the test has a Failed status.
	One or more tests in the batch are still running, but there are no unresolved or failed tests in the batch.
	All tests in the batch have completed and passed - there were no mismatches, or the user accepted all mismatches.

The baseline status

If the Tester has made any changes to any test in the batch, i.e. has pressed accept or reject , reported an issue and so on, and has not yet pressed the save button then:

Controlling test status using Eyes SDK

Use the setSaveNewTests() to force new tests to take an "unresolved" status, instead of defaulting to "passed and update baseline"

Use the command to force an update of the baseline to all the checkpoints in a test, if one step with a mismatch is found. This can be used when you know that the test is being run with a new version of the application with a large amount of UI updates and you want to force rebuilding of the baseline.