Using Eyes with GitHub

Introduction

The Eyes GitHub integration builds on the branching capabilities in Eyes, and on the GitHub all commits.

Eyes supports multiple branches, each of which can consist of multiple test baselinesDefines the sequence of images to which the sequence of images captured at checkpoints will be compared. A test can have multiple baselines, each of which is characterized by the execution environment it ran on (operating system, browser and viewport size). A baseline can have instances in multiple branches.. Eyes supports creating a new branch based on an existing branch, independent updates to the baselines in each branch, comparing branches to detect conflicts, and, finally, merging of the baselines in branches.

GitHub integrates with Eyes in the following aspects:

  • Eyes tests are automatically configured to use and save baselines in branches with names derived from the Git branch names.
  • All Eyes tests that are in a specific CI build are automatically batched together, with a batch name that includes the Git commit information.
  • The GitHub pull request panel includes a status indicating the latest Eyes test results and merge status.
  • The Github Commits panel also includes a status indicating the visual UI test result.
  • When the user executes a commit that triggers CI, the Applitools test status is updated to reflect the result of the test.

The remainder of this article has three parts. The first two sections describe the one-time setup required to start using the integration:

The second part of the article describes how you can use this integration in your daily development workflow:

This article focuses on the Eyes GitHub integration and assumes that you are already familiar with GitHub and a CI system that is integrated with GitHub.

The last section describes how to uninstall the GitHub App.

Getting started

Integration options

The Eyes/GitHub integration is based on GitHubs recommended 3rd party GtHub App approach. The integration installation procedure depends on the type of the GitHub service and is summarized in the following table:

GitHub App based installed via the Test manager For all customers that use the GitHub cloud. Described in this article, section Enabling the Eyes GitHub integration.
GitHub proxy service and GitHub App This integration solution is for users that use a self hosted GitHub system. Described in the article Using the Applitools GitHub proxy service.

For users that are currently integrated with the Legacy GitHub OAuth based solution and want to continue using it instead of changing to the GitApp based integration please see Configuring the direct Eyes/GitHub integration.

Enabling the Eyes GitHub integration

The Eyes/GitHub integration is based on a GitHub App created by Applitools that defines the minimum access that Eyes requires to interact with GitHub on the repositories where Eyes tests are run.

To avoid problems during the integration the user that runs the integration should have the following authorizations:

  • Eyes team admin

  • GitHub Organization

The following steps describe how to start the integration process:

Install the GitHub App

This section describes how to install the Applitools GitHub App. This integration solution is intended for users that use the GitHub cloud and the Eyes Cloud. If you use a self-hosted GitHub system or have a private cloud or on-prem Eyes system, then you should use the Appltiools GitHub proxy service. Installing and configuring the proxy service is described in Using the Applitools GitHub proxy service.

  1. If the Applitools GitHub App has not yet been installed, an Install the App button is displayed.

    Click this button to start the installation process. This only needs to be done once for your organization, so if you don't see the button, the GitHub App has already been installed and you can move on to the Configuring your repositories section below.

  2. The GitHub installation wizard opens in a separate window. Note that the wizard is implemented by GitHub - if the GitHub team changes the interface, the following screenshots may not match the GitHub screens you see. Select the GitHub Username under which you want to install the Applitools GitHub App. Typically, this will be your organization's GitHub username.

  3. The GitHub wizard asks you which repositories you want to allow the GitHub App to access and tells you what access type is being granted. It is recommended that you allow access to all current and future repositories, since this GitHub App installation serves your entire organization. If you select only specific repositories at this point, then when other repositories need to be added at later point in time, you will need to do this by configuration the Applitools GitHub directly on the GitHub site.

  4. If you have GitHub organization rights click you should see an Install button as shown in the screenshot. Click on the button. A dialog is displayed that confirms that the GitHub App is installed. Click the OK button to complete the process and you will see a confirmation dialog that the installation has completed successfully.
  5. If in the GitHub dialog you see a button Install and Request or Request instead of Install then there is an issue with your GitHub credentials. If you click on the Cancel button and exit from the GitHub installation dialog a dialog with the title Installation process has not been completed is displayed. This also displays a link that you can copy and send to your GitHub organization admin to complete the integration. Once this has done you can proceed with the instructions described in the section Configuring your repositories below.

Configuring your repositories

The next step is to configure the repositories that your team needs to integrate with:

  • Click the Manage repositories button.

  • Select the repositories on which your team runs tests and click the Apply button.

    If the list of repositories is large, you can type text in to the filter to display only repositories that include that text.

  • After you select the Apply button the repository selection dialog closes and a list of integrated repositories is displayed.

  • By default, GitHub integration closes a batch automatically once the CI build operation is completed. If your build runs concurrently or consists of multiple test and you want all of the results to appear together in a single batch, you need to disable automatic closing of the batch and close the batch explicitly when the build is complete in your CI.

    To control the batch closing operation, select the Show batch closing control checkbox and use the toggles that appear to the right of each repository to enable or disable auto-closing of the batch.

    If you disable auto closing of the batch, then you must notify closing of the batch explicitly, as described in Synchronizing multiple concurrent builds.

How to configure your CI

Apart from configuring the Test manager as described above, you also need to configure your CI as follows:

  • Setting up the start of the build: Setup that is done by the CI as part of the build before the Eyes tests are executed. This sets environment variables that are read by the Eyes SDK to obtain the information required by Eyes to execute the test.
  • Synchronizing multiple concurrent builds: If the CI runs a build using multiple concurrent instances, then it needs to notify Eyes when all of the instances have terminated.

For detailed instructions on how to configure a variety of popular CI platforms see Configuring your CI for the Eyes GitHub integration.

Working with GitHub

Once your system is set up as described above, you can use your CI and GitHub in the usual way. A CI job is normally triggered by a push to your repository, and this, in turn, runs your Eyes tests.

The Eyes GitHub integration is based on the GitHub all commits. When there is an open all commits, Eyes adds the following functionality to your GitHub workflow:

  • When your CI initiates a build that includes an Eyes test, Eyes detects that the run is associated with a GitHub pull request and searches for a baseline to use as a reference. Eyes checks for a matching baseline in the following order:
    • On a branch that corresponds to the repository name and source branch defined by the all commits.
    • If there is no such branch, Eyes creates a branch and populates it with the latest baseline for that test from the repository name and target branch defined by the all commits
    • If there is no such baseline, Eyes marks the test as New.
  • Eyes sends GitHub status information that is displayed on the GitHub all commits panel, as shown below. Note the icons in the lines with the Applitools logo . These icons represent the Eyes test status.
    • If the link at the top right-hand corner of the panel says Show all checks instead of Hide all checks (see the screenshot below), then no items are visible in the list. Click the link so that all of the items will be visible.

  • The item labeled tests/applitools indicates the results of the Eyes test that ran as part of this build. In this case, the icon indicates that some differences were found and need to be resolved. A icon would indicate that all of the Eyes tests passed. If you click on the Details link, then the browser switches to the Test results screen of the Eyes Test manager, showing the results of the tests in the most recent build related to that all commits. If you click Accept on all of the steps of all of the tests, the icon is replaced with the (passed) icon in the GitHub pull panel.
  • The item labeled scm/applitools represents the merge conflict status. In this example, the icon indicates that baseline conflicts have not yet been checked. A icon would indicate that there are merge conflicts. After you resolve any conflicts, Eyes sends an updated status to GitHub, and the next to the “scm/Applitools” item is replaced with the icon . If you click on the Details link, then your browser navigates to the Merge branches screen in the Test manager. See How to review branch differences and resolve them for more details.
  • You can merge the baseline changes of the source branch with the target branch in two ways:
    1. By clicking on the Merge button on the Merge branches screen in the Test manager (enabled only after all conflicts have been resolved). This operation is independent of any merge process in Git and occurs immediately, even if the GitHub merge has not been done.
    2. By merging the GitHub all commits as usual. This merges the Git files and, once this has completed successfully, Eyes is notified that it should merge the Eyes test baselines.

Similar information is also displayed in the GitHub code panel.

How to review branch differences and resolve them

Merging an Eyes source branch with a target branch results in adding new baselines to the target branch or updating its existing baselines according to baseline changes found in the source branch. Changes can include both updated baseline images and new, changed, or removed annotations (e.g. an added ignore region).

Eyes knows how to compare two baselines and determine whether they can be merged automatically or whether user intervention is required. For example, an ignore region added to a source baseline step that did not change in the target baseline can be automatically added to the target baseline, but if another ignore region was added to the same step in the target baseline, user intervention is required to determine whether to keep only one of the added regions or both of them.

As described above, the item on the GitHub pull request page marked “scm/applitools” indicates the merge status of the Eyes baselines in the source branch.

When you click on the Details link on this item, the browser switches to the Test manager Merge branches page preloaded with the source and target branches derived from the pull request. If there are no conflicts, then the Test manager displays a pop-up window to confirm this.

Click Got it! to remove the pop-up.

A full description of the merging process is not within the scope of this article; we will only briefly mention the main features available on the Merge branches page.

The Merge branches page shows all of the baselines that have changed in the source branch with respect to the target branch. That is, either they don't exist in the target branch, or they exist in the target branch but have changed in the source branch. Note that baselines that have changed in the target branch and were not changed in the source branch are not listed and are ignored in the merge process.

On the far left of each row, the merge status can show a value of Conflict, indicating that the baseline requires user intervention.

You can resolve conflicts by using the buttons on the right of each row, which allow you to choose whether to replace the target baseline with the source baseline or keep the target baseline unchanged. Choosing one of these changes the merge status from Conflict to Resolved. You can also compare the two baselines and manually edit the source baseline to include the changes from both branches.

An alternative to resolving each test individually is to use the checkboxes at the start of each row to select rows and then use the buttons in the toolbar at the top-left of the window to choose how to resolve the selected rows. The toolbar also has buttons that allow you to delete selected baselines in the source branch or undo the resolution made for the selected baselines.

Once you have resolved all conflicts, the merge status of the scm/applitools item in the GitHub pull request page changes to green (passed), and you can proceed to merge the branches, either from within the Test manager by clicking the Merge button, or via the GitHub pull request page, as described above.

Uninstalling the GitHub App

If the GitHub App is installed, ad you want to uninstall it then proceed as follows: