Using Atlassian Bitbucket with Applitools Eyes

Introduction

The Eyes Bitbucket integration builds on the branching capabilities in Eyes, and on the Bitbucket pull request.

Eyes supports multiple branches, each of which can consist of multiple test baselinesDefines the sequence of images to which the sequence of images captured at checkpoints will be compared. A test can have multiple baselines, each of which is characterized by the execution environment it ran on (operating system, browser and viewport size). A baseline can have instances in multiple branches.. Eyes supports creating a new branch based on an existing branch, independent updates to the baselines in each branch, comparing branches to detect conflicts, and finally merging of the baselines in branches.

Bitbucket integrates with Eyes in the following aspects:
  • Applitools visual UI tests are automatically configured to use and save baselines in branches whose name is derived from the Git branch names.
  • All visual UI tests that are in a specific CI build, are automatically batched together, with a batch name that includes the Git commit information.
  • The Bitbucket pull request panel includes a status indicating the latest visual UI test results and merge status.
  • When the user initiates a Bitbucket pull-request merge, this will also trigger an Eyes branch merge.
The remainder of this article has two parts. The first few sections describe the one time setup required to start using the integration:
The remainder of the article describes how you can use this integration in your daily development workflow:

This article focuses on the Eyes Bitbucket integration and assumes that you are already acquainted with Bitbucket and a CI system that is integrated with Bitbucket.

Getting started

There are two ways to support the Eyes Bitbucket integration:
  1. Using the Bitbucket Cloud.
  2. Using the Bitbucket Server or Data Center.

The difference between the two lies in the way your CI application notifies Applitools Eyes that the visual testing build is complete. For Bitbucket Cloud integration, the notification is performed automatically in response to a build status update that is received from Bitbucket. For the Bitbucket Sever or Data Center integration, you need to explicitly configure your CI application to send a notification to Applitools Eyes.

How to configure the Eyes Bitbucket Cloud integration

There are two steps to integrating Eyes with Bitbucket:
  • Adding a Bitbucket server. This only needs to be done on an account that uses an on-premise Eyes setup. This only needs to be done once per account.
  • Configure your team's repositories. This needs to be done by every team in the account.
in both cases, the first step is to navigate to your Admin/Team page:
  • Use the Page navigatorThe Page navigator is a control at the top of the Test manager window that is used to switch between the main pages of the Test manager. It also displays the name of the current page. to display the Admin page. You will only see this page in the Page navigator if you have Admin privileges.
  • Select the Teams  tile.
  • Click on the row of the team you want to configure.
  • Scroll down the page to find the Bitbucket section:

  • Add a Bitbucket server to integrate with. If your organization uses an Eyes Cloud server and bitbucket.org, then you can skip this step. If your organization has an on-premise Eyes Server then this step needs to be done once for all the teams in the account. In the server selection menu, look for, and select your Bitbucket server. If it is not on the list, then click the Add button. This will open up a wizard which will guide you through the rest of the process; you will need the domain name of your Bitbucket server to start this process.
  • If your organization uses an Eyes on-premise setup, you need to white list the Bitbucket server. You should white list inbound HTTPS traffic from bitbucket.org to your Eyes server in your company's firewall.
  • Configure your team's repositories. Click the Manage repositories button. This will open up a wizard which will guide you through the rest of the process.

How to configure the Eyes Bitbucket Server or Data Center integration

There are two steps to integrating Eyes with Bitbucket:
  • Adding a Bitbucket server. This only needs to be done once per account.
  • Configure your team's repositories. This needs to be done for every team in the account.
in both cases, the first step is to navigate to your Admin/Team page:
  • Use the Page navigatorThe Page navigator is a control at the top of the Test manager window that is used to switch between the main pages of the Test manager. It also displays the name of the current page. to display the Admin page. (You will only see this page in the Page navigator if you have Admin privileges).
  • Select the Teams  tile.
  • Click on the row of the team you want to configure.
  • Scroll down the page to find the Bitbucket section:

  • Add a Bitbucket server to integrate with. In the server selection menu, look for, and select your Bitbucket server. If it is not on the list, then click the Add button. This will open up a wizard which will guide you through the rest of the process; you will need the domain name of your Bitbucket Server or Data Center to start this process.
  • The Bitbucket Server and Data Center are self hosted servers that are usually deployed behind the company firewall; therefore if you are using a cloud-based Eyes server you should white list inbound HTTPS traffic from the Eyes server to your on-prem Bitbucket server in your company firewall. If you use the Eyes public cloud, then you should white list eyesapi.applitools.com, and if you use a dedicated cloud, then you should white list the URL of that cloud, for example https://myServerapi.applitools.com.
  • Configure your team's repositories. Click the Manage repositories button. This will open up a wizard which will guide you through the rest of the process. As part of the process you need to select an authentication method. You can choose from using a Personal Access Token (PAT) or Basic Authorization.
  • If using a Personal Access Token, then when creating the PAT you must set Admin level permissions for accessing the repositories. For information on creating a PAT see generating Personal access tokens.
  • If using Basic Authorization to access the server, then you need only provide the User name and Password credential pair for accessing the server.

How to configure your CI

You need to set up your CI to configure the Eyes SDK with information related to the Bitbucket pull-request. For detailed instructions on how to configure a variety of popular CI platforms see Configuring your CI for the Eyes Bitbucket integration.

How to prepare your code

The SDK uses the value of the environment variable APPLITOOLS_API_KEY to obtain the Applitools API key and the value APPLITOOLS_BATCH_ID to obtain the batch ID. Older versions of the SDK do not do this. We recommend updating your SDK. If this is not possible then add the following code to your test after the eyes instance variable has been instantiated and before the test is opened.


                            

Working with Bitbucket

Once your system is set up as described above, you use your CI and Bitbucket in the usual way. A CI job is normally triggered by a push to your repository. This, in turn, will then run your visual UI tests.

The Eyes Bitbucket integration is based on the Bitbucket pull request. When there is an open pull request, Eyes adds the following functionality to your Bitbucket workflow:

  • When your CI initiates a build that includes an Eyes test, Eyes detects that the run is associated with a Bitbucket pull request and will search for a baseline to use as a reference. Eyes will check for a matching baseline in the following order:
    • On a branch that corresponds to the repository name and source branch defined by the pull request.
    • If there is no such branch, it will create a branch and populate it with the latest baseline for that test from the repository name and target branch defined by the pull request.
    • If there is no such baseline the test will be marked as "New".
  • Eyes sends Bitbucket status information that is displayed on the Bitbucket Builds panel as shown below. 

  • The item labeled tests/applitools indicates the results of the visual UI test that ran as part of this build. In this case, the indicates that all the visual UI tests passed. A indicates that either some differences were found and need to be resolved, that the test failed to complete successfully, or that the merge process identified conflicts. Whereas, a indicates that the test is in progress.
  • The item label, test/applitools, functions as a link. If you click the link, then the browser will switch to the Test results screen of the Eyes Test Manager, showing the results of the tests in the most recent build related to that pull request. When dealing with differences, if you Accept all the steps of all the tests, then when you go back to the Bitbucket Builds panel the (error/difference) will be replaced by a (passed).
  • The symbols adjacent to the item labeled scm/applitools represents the merge conflict status. An indicates that there are merge conflicts. After you resolve any conflicts Eyes will send an updated status to Bitbucket, and you will see a icon instead next to the scm/Applitools item.
  • If you click on scm/applitools, then your browser will navigate to the Merge branches screen in the Test Manager. See How to review branch differences and resolve them for more details.
  • You can merge the baseline changes of the source branch with the target branch in two ways:
    1. By clicking on the Merge button in the Merge Branches screen in the Test Manager (enabled only after all conflicts have been resolved). This operation is independent of any merge process in Git, and occurs immediately, even if the Bitbucket merge was not done.
    2. Merge the Bitbucket pull-request as usual. This will first merge the Git files and, once this has completed successfully, Eyes will be notified that it should merge the visual UI test baselines.

How to review branch differences and resolve them

Merging an Eyes source branch with a target branch results in adding new baselines to the target branch or updating its existing baselines according to baseline changes found in the source branch. Changes can include both updated baseline images as well as new, changed or removed annotations (e.g., an added ignore region). Eyes knows how to compare two baselines and determine if they can be merged automatically or if some sort of manual user resolution is required. For example, an ignore region added to a source baseline step that did not change in the target baseline can be automatically added to the target baseline, but if another ignore region was added to the same step in the target baseline, user intervention is required to determine whether to keep only one of the added regions, or to keep both of them.

As described above, the item on the Bitbucket pull-request page marked “scm/applitools” indicates the merge status of the Eyes baselines in the source branch.

When you click on the link on this item, the browser will switch to the Test Manager "Merge branch" page reloaded with the source and target branches derived from the pull-request. If there are no conflicts, then the Test Manager will first display a pop-up window stating this.

Press “Got it!” to remove the pop-up.

A full description of merging is not within the scope of this article; we will only briefly mention the main features available on the merge branches page.

The merge branches page shows all the baselines that have changed in the source branch with respect to the target branch. That is, either they don't exist in the target branch, or they exist in the target branch but have changed in the source branch. Note that baselines that have changed in the target branch and were not changed in the source branch will not be listed and are ignored in the merge process.

At the far left of each row, the status can show a value of “Conflict” indicating that the baseline needs resolving by the user.

You resolve conflicts by using the buttons at the right of each row, which allow you to choose if you want to replace the target baseline with the source baseline or to keep the target baseline unchanged. Choosing one of these will turn the “Conflict” status into a “Resolved” status. You can also compare the two baselines and manually edit the source baseline to include the changes from both branches.

An alternative to resolving each test individually is to use the checkbox at the start of each row to select rows and then use the buttons in the toolbar at the top left of the window to choose how to resolve the selected rows. The toolbar also has buttons that allow you to delete selected baselines in the source branch or to undo the resolution made for the selected baselines.

Once you have resolved all conflicts, the status of the scm/applitools item in the Bitbucket pull-request page will change to green (passed), and you can proceed to merge the branches, either from within test manager by clicking the Merge button, or via the Bitbucket pull-request page as described above.