Test and batch status

The Eyes Test Manager displays the status of each batch and test. The test status shows the status of each step in a test. The batch status shows the status of the entire batch.

Test status

The test status is displayed in the Test detail view and includes the flowing:

• Status (Running, New, Passed, Failed, or Aborted)

• At least one step had mismatches.

• A user has manually accepted or rejected every single step

• Color to represent the current status:

  • Blue – Test is still running

  • Orange – Test unresolved

  • Green – Test passed

  • Red – Test failed

The test status can have one of the following values:

	The test is currently running. No further information is available at this point.
	New test with no previous baseline. By default, all steps are considered passed, even though a tester did not manually accept or reject every step.
	New test that was manually failed by a user.
	New test run with setSaveNewTests(false). This method sets the default status of a new test to Unresolved instead of Passed.
	The test is not new (has a baseline) and automatically marked as passed as no steps had mismatches.
	User manually marked the test as passed.
	User manually marked a test as passed even though there were mismatches.
	The test contains at least one unresolved step, i.e. at least one step had a status of different (), missing (), or new () and a user has not manually accept or reject the test.
	User manually marked an unresolved test as failed.
	The test was aborted before all the steps were completed. A test could be aborted by a user, or by the system if the SDK was inactive for at least 60 seconds, or the test was inactive for 15 minutes.

Manual override of the test status

You can click on the status of a test to mark the entire test as passed or failed, You can pass or fail a test even if some of the steps in the test have differences that were not accepted.

Select Default for the system to automatically define a status. Eyes will mark the test as Passed is all steps in the test have a status of passed, otherwise it will assign a status of Failed.

Batch status

The status of a batch is derived from the status of all the tests in the batch. The batch status is displayed at the top of the Test Results and in the list of batches on the left.

The following table summarizes the possible batch status values:

	At least one test in the batch is unresolved. i.e., has mismatches or has missing or new steps that have not been accepted or rejected.
	All the tests have been resolved, and at least one has the test has a Failed status. A test may have a status of failed because: A tester resolved all tests, but marked at least in test as Failed. A test was Failed by automated test maintenance A test has an issue with a Failed status
	One or more tests in the batch are still running, but there are no unresolved or failed tests in the batch.
	All tests in the batch have completed and passed and there were no mismatches or the user accepted all mismatches.

Unsaved changes

If you have unsaved changes to any test in the batch, such as accept , reject , or reported an issue, there is an Unsaved icon at the top of the batch, and the Current Batch information pane indicates the number of unsaved changes. We recommend that you save changes and confirm that the status is updated before closing the Test results page.

Controlling test status using Eyes SDK

You can automatically control the status of a test with the following methods:

• setSaveNewTests(true) — A new test will automatically have a status of Passed, and baselines will be created automatically (Default behavior).

• setSaveNewTests(false) — A new test will have a status of Unresolved, and you will need to manually pass or fail the test.

• setSaveDiffs(true) — A new baseline will automatically be created if there is a diff. For example, if a website has been redesigned and every page will have changed, use this method to automatically create new baselines for all pages with differences.