Visual locators

In some types of application, using DOM locators to generate keyboard or mouse events is inconvenient or even impossible. Eyes provides visual locators that allow you to use Eyes technology to analyze an application page and discover the coordinates of predefined graphic entities. You can then use these coordinates to simulate mouse and keyboard events using the standard browser device handler methods.

Currently, this feature is only supported by the eyes-appium-java4 SDK. For information about availability on other SDKs please contact customer success.

Overview of using visual locators

A visual locator has two attributes - a rectangular image and a name. Two visual locators can have the same name. This means you can implement one "logical entity" which looks different in different circumstances or states by simply creating two visual locators with the same name.

Using visual locators is a two-step workflow. The first step is to define the set of visual locator images and their names. Once Eyes knows what the locator images and names are, the second step is to use an Eyes query to search for instances of the locator images on the current application page view port.

Visual locators are organized by test application name (i.e. the application name that is passed as a string when calling eyes.open( ).

The following sections describe how to use visual locators in more detail:

Defining visual locators in Eyes

Defining visual locators is a one-time action. You do this by uploading the images using the Eyes SDK and then you use the Eyes Test Manager to identify the locator to name it. locators have been created, they are available for query in any Eyes test run. If necessary (for example, if you need to search for new visual locators), you can repeat the process described below to define new visual locators.

You define visual locators as follows:

  1. Run a program that uses the Eyes SDK to execute checkpoints on pages that include UI elements you would like to interact with.

    Some points to note:

    • The purpose of this program is to upload the images to Eyes and not to validate checkpoints. We are simply using an API that is familiar as a way to import images into the Eyes system.
    • The application name passed to eyes.open( ) in the program that uploads the images to Eyes must be identical to the application name passed to eyes.open( ) in the test that will search for the visual locators associated with those images.
    • A checkpoint can contain more than one visual locator. This is convenient when identifying the visual locators in the Eyes Test Manager.

      It might be helpful if the test name contains some standard text such as "VISUAL-LOC". This makes it easy to find and delete the test runs and baselines used to import the visual locators.

  2. Once the test has run, you can use the Eyes Test Manager Step editor or Test editor to mark rectangular regions around the UI elements you wish to define visual locators for and give them a name. This process consists of the following steps:
    • Enter the Step editor for each step that contains one or more UI elements by clicking on the thumbnail of the step in the Test results page.
    • Open the visual locator gallery by clicking on .
    • Click to add a new visual locator.
    • Use the cross-hair cursor to mark a rectangular region around the UI element. The image should have a significant amount of unique detail (i.e. edges) so that it can be distingushed from other parts of the image and other visual locators. The image should not include a background that might change. For example, if you are marking a rectangle with rounded corners, it is better to create a smaller image that does not include the area on the outer side of the rounded corners.
    • In the dialog box that opens, enter a name for the visual locator. If the name is already in use in the application, a dialog will appear and you can choose from the following options:
      1. Replace any existing images with this locator name and associated this image for that locator name.
      2. Rename the visual locator that is associated with the current image.
      3. Add a new image with this visual locator name
      4. Click to cancel creation of a new visual locator

Searching for visual locators

The API to locate a visual locator has the following signature: 

Map<String, Array<Rectangle>> Map<String, Array<Rectangle>> elementsList =
				 eyes.locate(VisualLocatorSettings locator);
The method takes as a parameter a VisualLocatorSettings type, which is a Fluent interface, meaning that it can be a chain of method calls. The following example calls the name() method three times, passing it the name of a visual locator in each call. You can call this method once for each visual locator. The call to the first() method, indicates that we only wish to get a single result for each visual locator. 
In the next example, we use the method names() to pass a list of visual locator names. 

		This is an alternative way to pass a number of visual locators and is especially useful when the list of visual locator names is dynamic (e.g. it is read from a file or depends on user input). At the end of the chain, we call the all() method instead of the first() method to indicate that we wish to get all instances of each locator and not just the first. You can call either first() or all() but not both. If neither is called, then the default is  first(). Note that using all() requires an exhaustive test of the entire image and takes more time so only use it if you know that there can be multiple instances of a locator and you want to find all of them.
The value returned by the call to eyes.locate is a Map, which maps each of a set of String values to an Array of Rectangles, where each String  is one of the locators' names passed as a parameter to the name() or names() methods and the  list of Rectangles represents all the graphic elements that Eyes found with that locator name. The Rectangle class exposes methods getLeft( ) and getTop( ), which give the position in pixels of the top-left corner of the locator graphic relative to the  top-left corner of the viewport. It exposes getWidth() and getHeight(), which return the dimensions of the locator graphic.
If the image for a particular locator name is not found in the viewport, the array returned for that name will be a zero length array. If a locator name is passed, and a locator of that name has not been defined on the Eyes server, a value of , null is returned instead of an array.
The following code is an example of how to  search for a coordinate and then use it to generate an event:
Managing visual locators
You can see and manage visual locators  in the Eyes Test Manager in the following ways:
Each of these is described in more detail below, but first we will describe functionality that is common to all types of visual locator gallery.
Visual locator operations
In all types of gallery, you will see a window with one or more visual locator cards - one per visual locator. 
Each card contains a thumbnail of the image, the locator name, and the date it was created. A toolbar  appears when you hover over a particular card, allowing you to take the following actions:
  • Click on this button to enter the Step editor loaded with the step where this visual locator image was defined.
  • 
						Click on this button to rename the visual locator. If the name is already in use in the application, a dialog will appear and you can choose from the following options: 
						
    • Replace any existing images with this locator name and associated this image for that locator name. 
    • Rename the visual locator that is associated with the current image.
    • Add a new image with this  visual locator name 
    • Click   to cancel creation of a new visual locator
  • 
					 Click on this button to delete the visual locator.
View visual locators in the Step and Test editors
The visual locator pane in the Step editor and Test editor provides access to all the visual locators defined for the application of the current step.
You can view the locator gallery pane as follows:
  1. Enter the Step editor by clicking on a thumbnail in the Test results page.
  2. Click  to open a panel to the left of the checkpoint image.
In addition to the actions you can take on each card, you can also take the following actions:
  •   Click on this button to go to the application locator gallery.
  • As you type into the filter box, Eyes will update the display, showing only visual locator cards for visual locators with names including the typed text.
  •  Click on this button to add a new visual locator as described in Defining visual locators in Eyes.
The global visual locator gallery
The global visual locator gallery allows access to all visual locators defined for your team.
You can access the global visual locator gallery as follows:
  1. Select the Apps & tests page.
  2. Select the application you require from the list of applications on the left.
  3. Click . 
  4. The display will change to show all the visual locators for that application.
  5. You can change the currently shown application using the application menu in the top-right corner.
  6. You can also reach this page from any step by clicking on  in the Step editor or Test editor.
In addition to the actions you can take on each card, you can also take the following actions:
  • As you type into the filter box, Eyes will update the display, showing only visual locator cards for visual locators with names including the typed text.
  • Click on this button to select/deselect all the visual locators visible. If you have applied a filter, then only the visual locators that are displayed are selected/deselected.
  •  Click on this button to delete all selected visual locators.