The Contrast advisor

The Applitools Contrast advisor functionality helps you comply with visual impairment requirements such as the Web Content Accessibility Guidelines (WCAG). The Contrast advisor focuses on the standards that define minimum contrast ratios for text and graphics:

  • You can specify which level and version of the standard you want to use for a test.

  • If the Contrast advisor is enabled, then during the course of a normal test run, Eyes automatically identifies the type of each element of a checkpoint (text or iamge), measures the contrast ratio, and reports elements with contrast ratios that are below the minimum specified for that type of element.

  • When the test is complete, you can easily view all checkpoints that have elements whose measured contrast ratio is below the minimum specified by the WCAG standard you chose.

  • If required, you can override the automatic detection of element type and specify the element type explicitly or define that contrast ratios should not be tested for a given element or region.

Overview

What contrast ratio criteria does Eyes validate?

Eyes can validate contrast ratios against both the WCAG AA and AAA validation levels. Specifically, the following criteria are measured:

  • Minimum text contrast ratio (WCAG 1.4.3 AA)
  • Enhanced text contrast ratio (WCA 1.4.6 AAA)
  • Graphical objects contrast ratio (WCAG 1.4.11)

How does Eyes test the contrast ratio?

Validating contrast ratios differs from standard Eyes visual testing in that Eyes does not compare the checkpoint image to a reference baseline image that defines the expected results. Instead, Eyes identifies the various elements on the page and their type (text or image), measures the contrast ratio between the foreground and the background of these elements, and compares the contrast ratio to the minimum contrast ratio defined by WCAG for that type of element. If the measured contrast ratio of any part of an element is less than the minimum defined by WCAG, then a contrast ratio violation is flagged for that element. Both the range of measured contrast ratio values and the expected contrast ratio are displayed in the Step editor and Test editor.

The Eyes vision algorithm that measures the contrast ratio has a number of advantages over other DOM-based techniques:

  • Only visible UI elements are reported. Elements that appear in the DOM but are not visible (and therefore do not need to conform to the standard) are not reported.

  • The text contrast ratio is measured when text appears in an image as well as in HTML text.

  • The contrast ratio is measured for icons.

  • It is possible to test the contrast ratio of non-DOM content such as mobile applications.

The contrast ratio validation workflow

The main steps to validate contrast ratio are outlined below:

  • Enabling contrast ratio measurement: To enable contrast ratio validation for all checkpoints in a test, when setting up the test, call the SDK method setAccessibilityValidation. Pass this method the required accessibility level (AA or AAA) and version of the guidelines (WCAG_2_0 or WCAG_2_1).

  • After the test has run, you can see the test results in the Test Manager Test results page, and you can view the details and images of a particular step in the Step editor and Test editor.

  • Filtering by contrast ratio validation status: You can use the filter on the Test results page, to apply the Accessibility criteria to choose which results you see – the options and tests and steps that passed, failed, or not validated.

  • Viewing contrast ratio validation result status: You can see the contrast ratio validation results in the Accessibility status column in the Batch details view . The value displayed in this column tells you the contrast ratio validation results of each test. This status is independent of the visual test result status column, which indicates the status of the standard visual validation.

    The Accessability status on a row will have a value of Passed if all steps passed the contrast ratio measurement and Failed if any steps failed. The column will be blank if contrast ratio was not tested.

  • Viewing contrast ratio validation status of a step: Steps that failed a contrast ratio measurement are flagged with the icon in the step thumbnail displayed in the various Test results page views. Hover over the thumbnail to view the validation level used (AA or AAA).

  • Viewing detailed checkpoint contrast ratio validation results: Within the Step editor and Test editor, you can enable accessibility mode by clicking on the button in the tool bar. In this mode, there is only a checkpoint image pane.

    Areas that fail to meet the minimum contrast ratio criteria are highlighted in red. For example, in the screenshot below, there are four highlighted areas:

    Hover over any point in the checkpoint image to view the classification of the region, the passed or failed status, and the actual and expected contrast ratios:

    In some cases, the actual contrast ratio is displayed as a range of values (e.g. 1.37 - 2.09). This reflects the minimum and maximum ratios found between neighboring pixels in the element. The contrast ratio validation will fail if the measured minimum value is less than the required minimum (i.e. if any portion of the element did not meet the minimum contrast ratio requirements).

  • Managing contrast classification regions with Test Manager: In addition to the automatic element type classification done by Eyes, you can also explicitly define the classification of elements and regions to be, for example, Normal text or Graphics. You can also specify that the contrast ratio of a particular region or element should not be measured by Eyes. In these explicitly defined regions, Eyes uses the classification you provide and measures the contrast ratio accordingly, or ignores that region if that is what you specified. You can define such contrast classification regions within the test using the SDK method accessibility, or you can specify the regions in the Eyes Test Manager after a test run and Eyes takes them into consideration in future tests after you save to the baseline.

Viewing contrast ratio validation results

Viewing contrast ratio validation result status

The Accessibility status column in the Batch details view contains a Passed indication if all the checkpoints in the test passed the contrast ratio validation. It contains a Failed indication if any checkpoint violated the contrast ratio limits. The column is empty if contrast ratio was not validated in the test run.

If the Accessibility status column is not visible in the test results table, click in the table header and then:

  • Find and click on Accessibility status in the Hidden columns list on the left.

  • Click on the move selected items to the right button .

  • Click on the or buttons to place the column in the required position.

  • Click Apply.

You will now see the Accessibility status column in the test table.

Viewing contrast ratio validation status of a step

On the Test results views where you can see a step thumbnail, the step thumbnail will contain a icon in any step that includes at least one contrast ratio violation.

Hover over the thumbnail to view the validation level (AA or AAA) and WCAG version used in the test.

Filtering by contrast ratio validation status

You can use the Accessibility criteria of the filter test results tool to limit the steps displayed to those that have a status value selected from the following list:

  • Passed

  • Failed

  • Not checked

By default, only steps that meet one of the selected filter values are displayed. If you select Show entire tests in the filter dialog, then all the steps of a test are displayed if at least one step in the test meets the filter criteria.

Viewing detailed checkpoint contrast ratio validation results

You can view detailed contrast ratio validation results for a specific checkpoint in the Step editor or the Test editor. The image below shows a screenshot of the Test editor with key contrast ratio-related functionality annotated (not all failed areas are annotated).

In the image you can see that all regions in which violations have been detected are shaded in red. The other accessibility related features are:

  • Use this button in the toolbar to enable and disable accessibility mode. In accessibility mode, only the checkpoint image is visible, and the toolbar is populated with contrast ratio validation related tools.
  • The contrast ratio violation icon is the most immediate indication that there are contrast ratio violations on the page. The icon appears in the editors and the step thumbnails.
  • Click this button to highlight the contrast ratio area with an animation.
  • Use these buttons to zoom into each failed element in turn. You can also use the shortcut keys , and . to move to the previous/next failure respectively.
  • In the accessibility results summary you can see the accessibility level and version at which the test ran, as well as the number of violations found. If you have changed the level or version from within the Test Manager (see next item) it also shows you what level and version are currently being displayed.
  • Use the level and version buttons to rerun the contrast ratio measurement on the current step at the desired accessibility level or version. Selecting the level or version in this way allows a quick assessment of the contrast ratio measurements you can expect. The state of these buttons does not impact the test status or how the test is run in the future. See Enabling contrast ratio measurement to see how to set the accessibility level for the test.
  • Use this button to add a contrast classification region. You can override the automatic element classification done by Eyes, or mark regions that should be ignored. Eyes will take these regions into account on subsequent test runs after you save the baseline . After you add a contrast classification region, Eyes immediately re-evaluates the contrast ratio of the current step using the currently selected accessibility level  . However, this will not impact the results of the test. See Adding a contrast classification region using the editors for more information.

Viewing the test contrast ratio validation status

Hover over any shaded region, Eyes to view information about the contrast ratio measurement for that region. In the screenshots below you can see an example where the cursor is hovered over a text element that has passed and another example where the cursor is hovered over a graphic object element that has failed:

The dialog box displays:

  • Classification: The classification of the element is defined automatically by Eyes or you define it explicitly using the SDK or the Test Manager. Possible classifications are:

    • Regular Text

    • Large text

    • Bold text (if defined explicitly)

    • Graphical object

  • Status: Whether the element Passed or Failed the contrast ratio validation.

  • Actual contrast: The contrast ratio measured by Eyes. This may be provided as a single value or as a minimum - maximum range of values. A region will be flagged as Failed if any portion of the element has a contrast ratio less than the minimum acceptable contrast ratio (i.e. if the minimum value in the range is less than the acceptable contrast ratio).

  • Expected contrast: The minimum acceptable contrast ratio at the current accessibility level. By default, the current accessibility level is the level used in the test. If you change the level using the  or buttons in the tool bar, then the expected contrast ratio and the passed/failed status in the dialog box will reflect the change, although the step and test accessibility status will not be affected.

Managing contrast classification regions with Test Manager

Eyes Test Manager provides operations that allow you to manage the classification of elements in a given region. Each of these is described in more detail in the following sections:

Adding a contrast classification region using the editors

You can add a new contrast classification region in the Step Editor and Test Editor when in accessibility mode, as follows:

  1. Click on the tool.

  2. Mark a rectangular region using the cursor.

  3. In the menu, choose how the region should be classified:

    • Ignore
    • Regular text
    • Large text
    • Bold text
    • Graphical object

  4. If you want this region to be applied to similar checkpoints using Automated test maintenance, click on Apply to other steps in scope. Otherwise, click outside the dialog to close it and create the region.

    Eyes re-evaluates the contrast ratio of the checkpoint, taking into account the newly defined region to provide immediate feedback. However, this does not affect the accessibility status of the current step or test.

Once you have saved the baseline, on subsequent runs, Eyes first classifies regions based on all explicit classification you have provided, and then automatically classifies any other parts of the image.

Note that you can also add contrast classification regions using the SDK as described below in the Adding a contrast classification region with the SDK section. The advantage of using the Eyes Test Manager is that you can see the rectangular region that you are marking, whereas the advantage of using the SDK is that you can specify a particular element using a selector which remains well defined even if that element changes size or position.

Viewing contrast classification regions

You can add contrast classification regions to a checkpoint in the Step editor or Test editor as described in Adding a contrast classification region using the editors, or in the SDK as described in Adding a contrast classification region with the SDK. You can view regions of either type in the Step editor or Test editor. As you can see in the screenshot below, the regions are marked differently in order to distinguish between them:

Selecting what contrast ratio information to view

Click the layers menu to select which elements to view on the checkpoint image.

The available options are:

  • Violations: Regions that failed the contrast ratio validation.

  • Remarks & issues: Regions that were annotated by a user using the or tools.

  • Baseline regions: Regions that were added by a user in the Step editor or Test editor.

  • Code regions: Regions that were added by a user using the SDK.

  • Viewport: The bounding rectangle of the viewport.

Changing contrast classification regions

Click on a contrast classification region added to the baseline to adjust the bounding rectangle, change the classification, or delete the region. If you click a code region, you will see the basic information about the region (i.e. the bounding rectangle and the region classification that was set), but to change these values you need to make the necessary changes in the code where it was defined.

Deleting a contrast classification region

To delete a contrast classification region added to the baseline:

  • Click on the region you want to delete.

  • Click on .

  • Eyes then opens a dialog box that asks if you want to apply Automated test maintenance. Click No to delete only the current region and Delete in Scope to apply Auto maintenance.

You cannot delete regions added using the SDK using the Eyes Test Manager, you need to make the necessary changes in the code where it was defined.

Copying a contrast classification region

To copy a contrast classification region created using Test Manager to another step:

  • Click on the region you want to copy.
  • Click on or press CTRL+C.
  • Press CTRL+V on a different step to create an identical region at the same position in that step.

You cannot copy regions added using the SDK using the Eyes Test Manager.

Managing contrast ratio validation with the SDK

Enabling contrast ratio measurement

The following snippet shows how contrast ratio validation is enabled, and the accessibility level and version are set using the setAccessibilityValidation method. The level and version you set using this method applies for all checkpoints in the test and cannot be changed for a specific checkpoint.


                                
Adding a contrast classification region with the SDK

                                
The following snippet shows how the eyes.check  fluent interface accessibility method  can be used in various ways to define manual contrast classification regions. When a region is defined explicitly, either in the SDK, as shown here, or by Adding a contrast classification region using the editors, Eyes will not try to classify the pixels in the region,  and will evaluate the acceptability of that region based on the classification type that you specified.

                                
Using the SDK to retrieve contrast ratio validation results

                                
You can use the SDK to retrieve the overall results of the contrast ratio validation results validation. The example below shows how to use the getAccessibilityStatus method to retrieve the pass/fail status of each test run in the current Runner session.