Using Eyes with GitHub

Introduction

The Eyes GitHub integration builds on the branching capabilities in Eyes, and on the GitHub pull request.

Eyes supports multiple branches, each of which can consist of multiple test baselinesClosedDefines the sequence of images to which the sequence of images captured at checkpoints will be compared. A test can have multiple baselines, each of which is characterized by the execution environment it ran on (operating system, browser and viewport size). A baseline can have instances in multiple branches.. Eyes supports creating a new branch based on an existing branch, independent updates to the baselines in each branch, comparing branches to detect conflicts, and finally merging of the baselines in branches.

GitHub integrates with Eyes in the following aspects:
  • Applitools visual UI tests are automatically configured to use and save baselines in branches whose name is derived from the Git branch names.
  • All visual UI tests that are in a specific CI build, are automatically batched together, with a batch name that includes the Git commit information.
  • The GitHub pull request panel includes a status indicating the latest visual UI test results and merge status.
  • When the user initiates a GitHub pull-request merge, this will also trigger an Eyes branch merge.
The remainder of this article has two parts. The first three sections describe the one time setup required to start using the integration:
The remainder of the article describe how you can use this integration in your daily development workflow:

This article focuses on the Eyes GitHub integration and assumes that you are already acquainted with GitHub and a CI system that is integrated with GitHub.

Getting started

How to configure the Eyes GitHub integration

There are two steps to integrating Eyes with GitHub:
  • Adding a GitHub server. This only needs to be done on an account that has an on-premise Eyes or GitHub server. It only needs to be done once per account.
  • Configure your team's repositories. This needs to be done by every team in the account.
In any case, the first step is to navigate to your Admin/Team page:
  • Use the Page navigatorClosedThe Page navigator is a control at the top of the Test manager window that is used to switch between the main pages of the Test manager. It also displays the name of the current page. to display the Admin page. (You will only see this page in the Page navigator if you have Admin privileges).
  • Select the Teams  tile.
  • Click on the row of the team you want to configure.
  • Scroll down the page to find the GITHUB section:

  • Add a github server to integrate with. If your organization uses an Eyes Cloud server and github.com, then you can skip this step. If your organization has a GitHub Enterprise server or if you have an on-premise Eyes Server then this step needs to be done once for all the teams in the account. In the server selection menu, look for, and select your GitHub server. If it is not on the list, then click on the Add button. This will open up a wizard which will guide you through the rest of the process; you will need the domain name of your GitHub server to start this process.
  • The GitHub enterprise server is an on-premise server; therefore you need to white list the Eyes server. You should white list inbound HTTPS traffic from the Eyes server to the on-premise Enterprise GitHub server in your company's firewall. If you use the public cloud, then you need to white list eyesapi.applitools.com, and if you use a private cloud, then you need to white list https://myServerapi.applitools.com.
  • Configure your team's repositories. Click the Manage repositories button. This will open up a wizard which will guide you through the rest of the process.

How to configure your CI

Your CI needs to be set up to configure the Eyes SDK with information related to the GitHub pull-request. For detailed instructions on how to configure a variety of popular CI platforms see Configuring your CI for the Eyes GitHub integration.

How to prepare your code

The SDK uses the value of the environment variable APPLITOOLS_API_KEY to obtain the Applitools API key and the value APPLITOOLS_BATCH_ID to obtain the batch ID. Older versions of the SDK do not do this. We recommend updating your SDK. If this is not possible then add the following code to your test after the eyes instance variable has been instantiated and before the test is opened.


                                
Working with GitHub

                                
Once your system is set up as described above, you use your CI and GitHub in the usual way. A CI job is normally triggered by a push to your repository, this, in turn, will then run your visual UI tests. 

                                
The Eyes GitHub integration is based on the GitHub pull request.  When there is an open pull request, Eyes adds the following functionality to your GitHub workflow:

                                
    
                                    
  • When your CI initiates a build that includes an Eyes test, Eyes detects that the run is associated with a GitHub pull request and will search for a baseline to use as a reference. Eyes will check for a matching baseline in the following order:
    • 
                                    
      
                                        
    • On a branch that corresponds to the repository name and source branch defined by the pull request.
      • 
                                        
    • If there is no such branch, it will create a branch and populate it with the latest baseline for that test from the repository name and target branch defined by the pull request.
      • 
                                        
    • 
                   If there is no such baseline the test will be marked as "New".
                
      • 
                                    
    
                                    
  • Eyes sends GitHub status information that is displayed on the GitHub pull request panel as shown below.  Note the two items with the Applitools  logo  . These display icons based on the status values sent by Eyes. 
    • 
                                    
    (If the link at the top right says “Show all checks” instead of “Hide all checks” then click it so that the two items will be visible.) 
    
                                

                                

                                

                                
    
                                    
  • The item labeled “tests/applitools” indicates the results of the visual UI test that ran as part of this build. In this case, the  indicates that some differences were found and need to be resolved. 
				A 
			  would indicate that all the visual UI tests passed. 
				If you click on the “Details” link, then the browser will switch to the Test results screen of the Eyes Test Manager, showing the results of the tests in the most recent build related to that pull request. 
				If you click "Accept" on all the steps of all the tests, then when you go back to the GitHub Pull panel the  will be replaced by a  (passed).
				
    • 
                                    
  • The item, labeled “scm/applitools” represents the merge conflict status.  In this example, the   indicates that baseline conflicts have not yet been checked. 
					A  would indicate that there are merge conflicts. After you resolve any conflicts Eyes will send an updated status to GitHub, and you will see a 
					 icon instead of the  on the “scm/Applitools” item. 
					If you click on the “Details” link, then your browser will navigate to the Merge branches screen in the Test Manager. 
					See How to review branch differences and resolve them for more details. 
    • 
                                    
  • You can merge the baseline changes of the source branch with the target branch in two ways: 
    • 
                                    
      
                                        
    1. By clicking on the Merge button in the Merge Branches screen in the Test Manager (enabled only after all conflicts have been resolved). This operation is independent of any merge process  in Git, and occurs immediately, even if the GitHub merge was not done.
      2. 
                                        
    2. Merge the GitHub pull-request as usual. This will first merge the Git files and, once this has completed successfully, Eyes will be notified that it should merge the visual UI test baselines.
      3. 
                                    
    
                                

                                
How to review branch differences and resolve them

                                
Merging an Eyes source branch with a target branch results in adding new baselines  to the target branch or updating its existing baselines according to baseline changes found in the source branch. Changes can include both updated baseline images as well as new, changed or removed annotations (e.g., an added ignore region). Eyes knows how to compare two baselines and determine if they can be merged automatically or if some sort of manual user resolution is required. For example, an ignore region added to a source baseline step that did not change in the target baseline can be automatically added to the target baseline, but if another ignore region was added to the same step in the target baseline, user intervention is required to determine whether to keep only one of the added regions, or to keep both of them.

                                
As described above, the item on the GitHub pull-request page marked “scm/applitools” indicates the merge status of the Eyes baselines in the source branch.

                                

                                

                                
When you click on the “Details” link on this item, the browser will switch to the Test Manager "Merge branch" page  preloaded with the source and target branches derived from the pull-request. If there are no conflicts, then the Test Manager will first display a pop-up window stating this. 

                                

                                

                                
Press  “Got it!” to remove the pop-up.

                                
A full description of merging is not within the scope of this article; we will only briefly mention the main features available on the merge branches page.

                                
The merge branches page shows all the baselines that have changed in the source branch with respect to the target branch. That is, either they don't exist in the target branch, or they exist in the target branch but have changed in the source branch. Note that baselines that have changed in the target branch and were not changed in the source branch will not be listed and are ignored in the merge process. 

                                

                                

                                
At the far left of each row, the status can show a value of “Conflict” indicating that the baseline needs resolving by the user. 

                                
You resolve conflicts by using the buttons at the right of each row,  which allow you to choose if you want to replace the target baseline with the source baseline or to keep the target baseline unchanged.  Choosing one of these will turn the “Conflict” status into a “Resolved” status. You can also compare the two baselines and manually edit the source baseline to include the changes from both branches.

                                
An alternative to resolving each test individually is to use the checkbox at the start of each row to select rows and then use the buttons in the toolbar at the top left of the window to choose how to resolve the selected rows. The toolbar also has buttons that allow you to delete selected baselines in the source branch or to undo the resolution made for the selected baselines.

                                
Once you have resolved all conflicts, the status of the scm/applitools item in the GitHub pull-request page will change to green (passed), and you can proceed to merge  the branches, either from within test manager by clicking the Merge button, or via the GitHub pull-request page as described above.