Using Eyes with GitHub
Introduction
The Eyes GitHub integration builds on the branching capabilities in Eyes, and on the GitHub pull request.
Eyes supports multiple branches, each of which can consist of multiple test baselines
Defines the sequence of images to which the sequence of images captured at checkpoints will be compared. A test can have multiple baselines, each of which is characterized by the execution environment it ran on (operating system, browser and viewport size). A baseline can have instances in multiple branches.. Eyes supports creating a new branch based on an existing branch, independent updates to the baselines in each branch, comparing branches to detect conflicts, and finally merging of the baselines in branches.
- Applitools visual UI tests are automatically configured to use and save baselines in branches whose name is derived from the Git branch names.
- All visual UI tests that are in a specific CI build, are automatically batched together, with a batch name that includes the Git commit information.
- The GitHub pull request panel includes a status indicating the latest visual UI test results and merge status.
- When the user initiates a GitHub pull-request merge, this will also trigger an Eyes branch merge.
This article focuses on the Eyes GitHub integration and assumes that you are already acquainted with GitHub and a CI system that is integrated with GitHub.
Getting started
How to configure the Eyes GitHub integration
- Adding a GitHub server. This only needs to be done on an account that has an on-premise Eyes or GitHub server. It only needs to be done once per account.
- Configure your team's repositories. This needs to be done by every team in the account.
- Use the Page navigatorThe Page navigator is a control at the top of the Test manager window that is used to switch between the main pages of the Test manager. It also displays the name of the current page. to display the Admin page. (You will only see this page in the Page navigator if you have Admin privileges).
- Select the Teams tile.
- Click on the row of the team you want to configure.
- Scroll down the page to find the GITHUB section:
- Add a github server to integrate with. If your organization uses an Eyes Cloud server and github.com, then you can skip this step. If your organization has a GitHub Enterprise server or if you have an on-premise Eyes Server then this step needs to be done once for all the teams in the account. In the server selection menu, look for, and select your GitHub server. If it is not on the list, then click on the Add button. This will open up a wizard which will guide you through the rest of the process; you will need the domain name of your GitHub server to start this process.
- The GitHub enterprise server is an on-premise server; therefore you need to white list the Eyes server. You should white list inbound HTTPS traffic from the Eyes server to the on-premise Enterprise GitHub server in your company's firewall. If you use the public cloud, then you need to white list eyesapi.applitools.com, and if you use a private cloud, then you need to white list https://myServerapi.applitools.com.
- Configure your team's repositories. Click the Manage repositories button. This will open up a wizard which will guide you through the rest of the process.
How to configure your CI
Your CI needs to be set up to configure the Eyes SDK with information related to the GitHub pull-request. For detailed instructions on how to configure a variety of popular CI platforms see Configuring your CI for the Eyes GitHub integration.
How to prepare your code
The SDK uses the value of the environment variable APPLITOOLS_API_KEY to obtain the Applitools API key and the value APPLITOOLS_BATCH_ID to obtain the batch ID. Older versions of the SDK do not do this. We recommend updating your SDK. If this is not possible then add the following code to your test after the eyes instance variable has been instantiated and before the test is opened.
Working with GitHub
Once your system is set up as described above, you use your CI and GitHub in the usual way. A CI job is normally triggered by a push to your repository, this, in turn, will then run your visual UI tests.
The Eyes GitHub integration is based on the GitHub pull request. When there is an open pull request, Eyes adds the following functionality to your GitHub workflow:
- When your CI initiates a build that includes an Eyes test, Eyes detects that the run is associated with a GitHub pull request and will search for a baseline to use as a reference. Eyes will check for a matching baseline in the following order:
- On a branch that corresponds to the repository name and source branch defined by the pull request.
- If there is no such branch, it will create a branch and populate it with the latest baseline for that test from the repository name and target branch defined by the pull request.
- If there is no such baseline the test will be marked as "New".
- Eyes sends GitHub status information that is displayed on the GitHub pull request panel as shown below. Note the two items with the Applitools logo
. These display icons based on the status values sent by Eyes.
- The item labeled “tests/applitools” indicates the results of the visual UI test that ran as part of this build. In this case, the
indicates that some differences were found and need to be resolved.
A
would indicate that all the visual UI tests passed.
If you click on the “Details” link, then the browser will switch to the Test results screen of the Eyes Test Manager, showing the results of the tests in the most recent build related to that pull request.
If you click "Accept" on all the steps of all the tests, then when you go back to the GitHub Pull panel the will be replaced by a (passed).
- The item, labeled “scm/applitools” represents the merge conflict status. In this example, the
indicates that baseline conflicts have not yet been checked.
A would indicate that there are merge conflicts. After you resolve any conflicts Eyes will send an updated status to GitHub, and you will see a
icon instead of the on the “scm/Applitools” item.
If you click on the “Details” link, then your browser will navigate to the Merge branches screen in the Test Manager.
See How to review branch differences and resolve them for more details. - You can merge the baseline changes of the source branch with the target branch in two ways:
- By clicking on the Merge button in the Merge Branches screen in the Test Manager (enabled only after all conflicts have been resolved). This operation is independent of any merge process in Git, and occurs immediately, even if the GitHub merge was not done.
- Merge the GitHub pull-request as usual. This will first merge the Git files and, once this has completed successfully, Eyes will be notified that it should merge the visual UI test baselines.
How to review branch differences and resolve them
Merging an Eyes source branch with a target branch results in adding new baselines to the target branch or updating its existing baselines according to baseline changes found in the source branch. Changes can include both updated baseline images as well as new, changed or removed annotations (e.g., an added ignore region). Eyes knows how to compare two baselines and determine if they can be merged automatically or if some sort of manual user resolution is required. For example, an ignore region added to a source baseline step that did not change in the target baseline can be automatically added to the target baseline, but if another ignore region was added to the same step in the target baseline, user intervention is required to determine whether to keep only one of the added regions, or to keep both of them.
As described above, the item on the GitHub pull-request page marked “scm/applitools” indicates the merge status of the Eyes baselines in the source branch.
When you click on the “Details” link on this item, the browser will switch to the Test Manager "Merge branch" page preloaded with the source and target branches derived from the pull-request. If there are no conflicts, then the Test Manager will first display a pop-up window stating this.
Press “Got it!” to remove the pop-up.
A full description of merging is not within the scope of this article; we will only briefly mention the main features available on the merge branches page.
The merge branches page shows all the baselines that have changed in the source branch with respect to the target branch. That is, either they don't exist in the target branch, or they exist in the target branch but have changed in the source branch. Note that baselines that have changed in the target branch and were not changed in the source branch will not be listed and are ignored in the merge process.
At the far left of each row, the status can show a value of “Conflict” indicating that the baseline needs resolving by the user.
You resolve conflicts by using the buttons at the right of each row, which allow you to choose if you want to replace the target baseline with the source baseline or to keep the target baseline unchanged. Choosing one of these will turn the “Conflict” status into a “Resolved” status. You can also compare the two baselines and manually edit the source baseline to include the changes from both branches.
An alternative to resolving each test individually is to use the checkbox at the start of each row to select rows and then use the buttons in the toolbar at the top left of the window to choose how to resolve the selected rows. The toolbar also has buttons that allow you to delete selected baselines in the source branch or to undo the resolution made for the selected baselines.
Once you have resolved all conflicts, the status of the scm/applitools item in the GitHub pull-request page will change to green (passed), and you can proceed to merge the branches, either from within test manager by clicking the Merge button, or via the GitHub pull-request page as described above.