Using Eyes with GitHub

Introduction

The Eyes GitHub integration builds on the branching capabilities in Eyes, and on the GitHub pull request.

Eyes supports multiple branches, each of which can consist of multiple test baselinesDefines the sequence of images to which the sequence of images captured at checkpoints will be compared. A test can have multiple baselines, each of which is characterized by the execution environment it ran on (operating system, browser and viewport size). A baseline can have instances in multiple branches.. Eyes supports creating a new branch based on an existing branch, independent updates to the baselines in each branch, comparing branches to detect conflicts, and, finally, merging of the baselines in branches.

GitHub integrates with Eyes in the following aspects:

• Applitools visual UI tests are automatically configured to use and save baselines in branches with names derived from the Git branch names.
• All visual UI tests that are in a specific CI build are automatically batched together, with a batch name that includes the Git commit information.
• The GitHub pull request panel includes a status indicating the latest visual UI test results and merge status.
• When the user initiates a GitHub pull request merge, this also triggers an Eyes branch merge.

The remainder of this article has two parts. The first two sections describe the one-time setup required to start using the integration:

• How to configure the Eyes GitHub integration
• How to configure your CI

The remainder of the article describes how you can use this integration in your daily development workflow:

• Working with GitHub
• How to review branch differences and resolve them

This article focuses on the Eyes GitHub integration and assumes that you are already familiar with GitHub and a CI system that is integrated with GitHub.

Getting started

How to configure the Eyes GitHub integration

In any case, the first step is to navigate to your Admin/Team page:

• Use the Page navigatorThe Page navigator is a control at the top of the Test manager window that is used to switch between the main pages of the Test manager. It also displays the name of the current page. to display the Admin page. (You can only see this page in the Page navigator if you have Admin privileges.)
• Click on the Teams tile.
• Click on the row of the team you want to configure.
• Click on the Integrations tab.
• Click on the GITHUB row.

  

• Select a server to view and manage integrated repositories: If your organization uses an Eyes Cloud server and github.com, then you can skip this step. If your organization has a GitHub Enterprise server or if you have an on-premise Eyes Server, then this one-time step needs to be completed, and it is then applied to all of the teams in the account. In the box displaying the server names, click on to open the server selection menu and select your GitHub server. If your server is not yet on the list, then click on the Add button and add the hostname of your GitHub server and other details as directed by the wizard.
• The GitHub Enterprise server is an on-premise server; therefore, you need to white list the Eyes server. You should white list inbound HTTPS traffic from the Eyes server to the on-premise GitHub Enterprise server in your company's firewall. If you use the public cloud, then you need to white list eyesapi.applitools.com, and if you use a private cloud, then you need to white list https://myServerapi.applitools.com.
• Select the repositories that should include Eyes/GitHub integration: Click the Manage repositories button and in the dialog that opens, select the check boxes of the repositories that you want to activate in the Eyes/GitHub integration. The dialog includes a Filter repository name field that you can use to narrow down the set of repositories that are displayed. If you enter text in the box, only repositories that include that filter text are displayed.
• If your build runs as a single instance, you can now proceed to configure the CI as is described in the next section How to configure your CI.
• If your build runs as multiple concurrent instances, then you need to take extra steps to ensure that the test results from all of the builds appear together in a single Test Manager batch:
• Select the Show batch closing control (advanced) check box. The Test Manager adds an another column labeled Automated batch closing that provides a toggle switch for each repository in the list of repositories.
• If the toggle switch for a repository is enabled (the default), when a build finishes execution, Eyes closes the batch. If the CI runs the build in multiple concurrent instances, then the test results from the concurrent build may appear in the same batch or in a different batch. In order to ensure that for a given repository build, all of the test results from all of the builds appear in a single batch, disable the toggle for that repository.
• If you disable automated batch closing for a repository, you need to ensure that the CI notifies Eyes when all of the concurrent builds are complete so that Eyes can then close the batch. For details see Synchronizing multiple concurrent builds.

How to configure your CI

Apart from configuring the Test Manager as described above, you also need to configure your CI in the following aspects:

• Setting up the start of the build: Setup that is done by the CI as part of the build before the Eyes tests are executed. This sets environment variables that are read by the Eyes SDK to obtain the information required by Eyes to execute the test.
• Synchronizing multiple concurrent builds: If the CI runs a build using multiple concurrent instances, then it needs to notify Eyes when all of the instances have terminated.

For detailed instructions on how to configure a variety of popular CI platforms, see Configuring your CI for the Eyes GitHub integration.

Working with GitHub

Once your system is set up as described above, you can use your CI and GitHub in the usual way. A CI job is normally triggered by a push to your repository, and this, in turn, runs your visual UI tests.

The Eyes GitHub integration is based on the GitHub pull request. When there is an open pull request, Eyes adds the following functionality to your GitHub workflow:

• When your CI initiates a build that includes an Eyes test, Eyes detects that the run is associated with a GitHub pull request and searches for a baseline to use as a reference. Eyes checks for a matching baseline in the following order:
• On a branch that corresponds to the repository name and source branch defined by the pull request.
• If there is no such branch, Eyes creates a branch and populates it with the latest baseline for that test from the repository name and target branch defined by the pull request.
• If there is no such baseline, Eyes marks the test as New.
• Eyes sends GitHub status information that is displayed on the GitHub pull request panel, as shown below. Note the icons in the lines with the Applitools logo . These icons represent the Eyes test status.
If the link at the top right-hand corner of the panel says Show all checks instead of Hide all checks in the screenshot below, then no items are visible in the list. Click the link so that all the items will be visible.

• The item labeled “tests/applitools” indicates the results of the visual UI test that ran as part of this build. In this case, the icon indicates that some differences were found and need to be resolved. A icon would indicate that all of the visual UI tests passed. If you click on the Details link, then the browser switches to the Test results screen of the Eyes Test Manager, showing the results of the tests in the most recent build related to that pull request. If you click Accept on all of the steps of all of the tests, the icon is replaced with the (passed) icon in the GitHub pull panel.
• The item labeled “scm/applitools” represents the merge conflict status. In this example, the icon indicates that baseline conflicts have not yet been checked. A icon would indicate that there are merge conflicts. After you resolve any conflicts, Eyes sends an updated status to GitHub, and the icon replaces the icon next to the “scm/Applitools” item. If you click on the Details link, then your browser navigates to the Merge branches screen in the Test Manager. See How to review branch differences and resolve them for more details.
• You can merge the baseline changes of the source branch with the target branch in two ways:
1. By clicking on the Merge button on the Merge branches screen in the Test Manager (enabled only after all conflicts have been resolved). This operation is independent of any merge process in Git, and occurs immediately, even if the GitHub merge was not done.
2. By merging the GitHub pull request as usual. This merges the Git files and, once this has completed successfully, Eyes is notified that it should merge the visual UI test baselines.

How to review branch differences and resolve them

Merging an Eyes source branch with a target branch results in adding new baselines to the target branch or updating its existing baselines according to baseline changes found in the source branch. Changes can include both updated baseline images and new, changed, or removed annotations (e.g. an added ignore region).

Eyes knows how to compare two baselines and determine whether they can be merged automatically or whether user intervention is required. For example, an ignore region added to a source baseline step that did not change in the target baseline can be automatically added to the target baseline, but if another ignore region was added to the same step in the target baseline, user intervention is required to determine whether to keep only one of the added regions or both of them.

As described above, the item on the GitHub pull request page marked “scm/applitools” indicates the merge status of the Eyes baselines in the source branch.

When you click on the Details link on this item, the browser switches to the Test Manager Merge branches page preloaded with the source and target branches derived from the pull request. If there are no conflicts, then the Test Manager displays a pop-up window to confirm this.

Click Got it! to remove the pop-up.

A full description of the merging process is not within the scope of this article; we will only briefly mention the main features available on the Merge branches page.

The Merge branches page shows all of the baselines that have changed in the source branch with respect to the target branch. That is, either they don't exist in the target branch, or they exist in the target branch but have changed in the source branch. Note that baselines that have changed in the target branch and were not changed in the source branch are not listed and are ignored in the merge process.

On the far left of each row, the merge status can show a value of Conflict, indicating that the baseline requires user intervention.

You can resolve conflicts by using the buttons on the right of each row, which allow you to choose whether to replace the target baseline with the source baseline or keep the target baseline unchanged. Choosing one of these changes the merge status from Conflict to Resolved. You can also compare the two baselines and manually edit the source baseline to include the changes from both branches.

An alternative to resolving each test individually is to use the check boxes at the start of each row to select rows and then use the buttons in the toolbar at the top left of the window to choose how to resolve the selected rows. The toolbar also has buttons that allow you to delete selected baselines in the source branch or undo the resolution made for the selected baselines.

Once you have resolved all conflicts, the merge status of the scm/applitools item in the GitHub pull request page changes to green (passed), and you can proceed to merge the branches, either from within the Test Manager by clicking the Merge button, or via the GitHub pull request page, as described above.