Accesibility Testing - Contrast Advisor

The Applitools Contrast advisor functionality helps you comply with visual impairment requirements such as the Web Content Accessibility Guidelines (WCAG). The Contrast advisor focuses on the standards that define minimum contrast ratios for text and graphics:

• You can specify which level and version of the standard you want to use for a test.
• If the Contrast advisor is enabled, then during the course of a normal test run, Eyes automatically identifies the type of each element of a checkpoint (text or iamge), measures the contrast ratio, and reports elements with contrast ratios that are below the minimum specified for that type of element.
• When the test is complete, you can easily view all checkpoints that have elements whose measured contrast ratio is below the minimum specified by the WCAG standard you chose.
• If required, you can override the automatic detection of element type and specify the element type explicitly or define that contrast ratios should not be tested for a given element or region.

Overview

Contrast ratio criteria

Eyes can validate contrast ratios against both the WCAG AA and AAA validation levels. Specifically, the following criteria are measured:

• Minimum text contrast ratio (WCAG 1.4.3 AA)
• Enhanced text contrast ratio (WCA 1.4.6 AAA)
• Graphical objects contrast ratio (WCAG 1.4.11)

How it works

Validating contrast ratios differs from standard Eyes visual testing in that Eyes does not compare the checkpoint image to a reference baseline image that defines the expected results. Instead, Eyes identifies the various elements on the page and their type (text or image), measures the contrast ratio between the foreground and the background of these elements, and compares the contrast ratio to the minimum contrast ratio defined by WCAG for that type of element.

If the measured contrast ratio of any part of an element is less than the minimum defined by WCAG, then a contrast ratio violation is flagged for that element. Both the range of measured contrast ratio values and the expected contrast ratio are displayed in the Step editor and Test editor.

The Eyes vision algorithm that measures the contrast ratio has a number of advantages over other DOM-based techniques:

• Only visible UI elements are reported. Elements that appear in the DOM but are not visible (and therefore do not need to conform to the standard) are not reported.
• The text contrast ratio is measured when text appears in an image as well as in HTML text.
• The contrast ratio is measured for icons.
• It is possible to test the contrast ratio of non-DOM content such as mobile applications.

Validation workflow

The main steps to validate contrast ratio are outlined below:

1. Enabling contrast ratio measurement: To enable contrast ratio validation for all checkpoints in a test, when setting up the test, configure the SDK as shown in the section below.

2. After the test has run, you can see the test results in the Test manager Test results page, and you can view the details and images of a particular step in the Step editor and Test editor.

3. Filtering by contrast ratio validation status: You can use the filter on the Test results page, to apply the Accessibility criteria to choose which results you see - the options and tests and steps that passed, failed, or not validated.

4. Viewing contrast ratio validation result status: You can see the contrast ratio validation results in the Accessibility status column in the Batch details view . The value displayed in this column tells you the contrast ratio validation results of each test. This status is independent of the visual test result status column, which indicates the status of the standard visual validation.

The Accessability status on a row will have a value of Passed if all steps passed the contrast ratio measurement and Failed if any steps failed. The column will be blank if contrast ratio was not tested.

5. Viewing contrast ratio validation status of a step: Steps that failed a contrast ratio measurement are flagged with the icon in the step thumbnail displayed in the various Test results page views. Hover over the thumbnail to view the validation level used (AA or AAA).

6. Viewing detailed checkpoint contrast ratio validation results: Within the Step editor and Test editor, you can enable accessibility mode by clicking on the button in the tool bar. In this mode, there is only a checkpoint image pane.

   Areas that fail to meet the minimum contrast ratio criteria are highlighted in red. For example, in the screenshot below, there are four highlighted areas:

Hover over any point in the checkpoint image to view the classification of the region, the passed or failed status, and the actual and expected contrast ratios:

In some cases, the actual contrast ratio is displayed as a range of values (e.g. 1.37 - 2.09) . This reflects the minimum and maximum ratios found between neighboring pixels in the element. The contrast ratio validation will fail if the measured minimum value is less than the required minimum (i.e. if any portion of the element did not meet the minimum contrast ratio requirements).

7. Managing contrast classification regions with Test manager: In addition to the automatic element type classification done by Eyes, you can also explicitly define the classification of elements and regions to be, for example, Normal text or Graphics. You can also specify that the contrast ratio of a particular region or element should not be measured by Eyes.

   In these explicitly defined regions, Eyes uses the classification you provide and measures the contrast ratio accordingly, or ignores that region if that is what you specified. You can define such contrast classification regions within the test using the SDK method .accessibility(), or you can specify the regions in the Eyes Test Manager after a test run and Eyes takes them into consideration in future tests after you save to the baseline.

Viewing contrast ratio validation results

Result status

The Accessibility status column in the Batch details view contains a Passed indication if all the checkpoints in the test passed the contrast ratio validation. It contains a Failed indication if any checkpoint violated the contrast ratio limits. The column is empty if contrast ratio was not validated in the test run.

If the Accessibility status column is not visible in the test results table, click in the table header and then:

1. Find and click on Accessibility status in the Hidden columns list on the left.
2. Click on the move selected items to the right button .
3. Click on the or buttons to place the column in the required position.
4. Click Apply.

You will now see the Accessibility status column in the test table.

Status of a step

On the Test results views where you can see a step thumbnail, the step thumbnail will contain a icon in any step that includes at least one contrast ratio violation.

Hover over the thumbnail to view the validation level (AA or AAA) and WCAG version used in the test.

Filtering by status

You can use the Accessibility criteria of the filter test results tool to limit the steps displayed to those that have a status value selected from the following list:

• Passed
• Failed
• Not checked

By default, only steps that meet one of the selected filter values are displayed. If you select Show entire tests in the filter dialog, then all the steps of a test are displayed if at least one step in the test meets the filter criteria.

Viewing specific sections

Hover over any shaded region, Eyes to view information about the contrast ratio measurement for that region. In the screenshots below you can see an example where the cursor is hovered over a text element that has passed and another example where the cursor is hovered over a graphic object element that has failed:

The dialog box displays:

• Classification: The classification of the element is defined automatically by Eyes or you define it explicitly using the SDK or the Test manager. Possible classifications are:

  • Regular Text
  • Large text
  • Bold text (if defined explicitly)
  • Graphical object

• Status: Whether the element Passed or Failed the contrast ratio validation.

• Actual contrast: The contrast ratio measured by Eyes. This may be provided as a single value or as a minimum - maximum range of values. A region will be flagged as Failed if any portion of the element has a contrast ratio less than the minimum acceptable contrast ratio (i.e. if the minimum value in the range is less than the acceptable contrast ratio).

• Expected contrast: The minimum acceptable contrast ratio at the current accessibility level. By default, the current accessibility level is the level used in the test. If you change the level using the or buttons in the tool bar, then the expected contrast ratio and the passed/failed status in the dialog box will reflect the change, although the step and test accessibility status will not be affected.

Managing contrast regions

Adding a contrast region

You can add a new contrast classification region in the Step Editor and Test Editor when in accessibility mode, as follows:

1. Click on the tool

2. Clickon the region tool

3. Mark a rectangular region using the cursor

4. In the menu, choose how the region should be classified:

   • Ignore
   • Regular text
   • Large text
   • Bold text
   • Graphical object

5. If you want this region to be applied to similar checkpoints using Automated test maintenance, click on Apply to other steps in scope. Otherwise, click outside the dialog to close it and create the region.

   Eyes re-evaluates the contrast ratio of the checkpoint, taking into account the newly defined region to provide immediate feedback. However, this does not affect the accessibility status of the current step or test.

Once you have saved the baseline, on subsequent runs, Eyes first classifies regions based on all explicit classification you have provided, and then automatically classifies any other parts of the image.

Note that you can also add contrast classification regions using the SDK as described below in the Adding a contrast classification region with the SDK section. The advantage of using the Eyes Test Manager is that you can see the rectangular region that you are marking, whereas the advantage of using the SDK is that you can specify a particular element using a selector which remains well defined even if that element changes size or position.

Viewing contrast regions

You can add contrast classification regions to a checkpoint in the Step editor or Test editor as described in Adding a contrast classification region using the editors, or in the SDK as described in Adding a contrast classification region with the SDK. You can view regions of either type in the Step editor or Test editor. As you can see in the screenshot below, the regions are marked differently in order to distinguish between them:

Selecting contrast ratio information

Click the layers menu to select which elements to view on the checkpoint image.

The available options are:

• Violations: Regions that failed the contrast ratio validation.
• Remarks & issues: Regions that were annotated by a user using the remark or issue tools.
• Baseline regions: Regions that were added by a user in the Step editor or Test editor.
• Code regions: Regions that were added by a user using the SDK.
• Viewport: The bounding rectangle of the viewport.

Changing contrast regions

Click on a contrast classification region added to the baseline to adjust the bounding rectangle, change the classification, or delete the region. If you click a code region, you will see the basic information about the region (i.e. the bounding rectangle and the region classification that was set), but to change these values you need to make the necessary changes in the code where it was defined.

Deleting contrast regions

To delete a contrast classification region added to the baseline:

1. Click on the region you want to delete.
2. Click on the delete button.
3. Eyes then opens a dialog box that asks if you want to apply Automated test maintenance. Click No to delete only the current region and Delete in Scope to apply Auto maintenance.

You cannot delete regions added using the SDK using the Eyes Test Manager, you need to make the necessary changes in the code where it was defined.

Copying contrast regions

To copy a contrast classification region created using Test manager to another step:

1. Click on the region you want to copy.
2. Click on the copy button or press CTRL+C.
3. Press CTRL+V on a different step to create an identical region at the same position in that step.

You cannot copy regions added using the SDK using the Eyes Test Manager.

Managing contrast ratio validation with the SDK

Enabling contrast ratio measurement

The following snippet shows how contrast ratio validation is enabled in the Applitools SDK. The level and version you set using this method applies for all checkpoints in the test and cannot be changed for a specific checkpoint.

Adding a contrast classification region

The following snippet shows how to define contrast classification regions. When a region is defined explicitly, either in the SDK, as shown here, or by Adding a contrast classification region using the editors, Eyes will not try to classify the pixels in the region, and will evaluate the acceptability of that region based on the classification type that you specified.

Retrieve contrast ratio validation results

You can use the SDK to retrieve the overall results of the contrast ratio validation results validation. The example below shows how to retrieve the pass/fail status of each test run.