Using Atlassian Bitbucket with Eyes


The Eyes Bitbucket integration builds on the branching capabilities in Eyes, and on the Bitbucket pull request and commit functionatity.

Eyes supports multiple branches, each of which can consist of multiple test baselines Defines the sequence of images to which the sequence of images captured at checkpoints will be compared. A test can have multiple baselines, each of which is characterized by the execution environment it ran on (operating system, browser and viewport size). A baseline can have instances in multiple branches.. Eyes supports creating a new branch based on an existing branch, independent updates to the baselines in each branch, comparing branches to detect conflicts, and finally merging of the baselines in branches.

Bitbucket integrates with Eyes in the following aspects:
  • Eyes tests are automatically configured to use and save baselines in branches whose name is derived from the Git branch names.
  • All visual UI tests that are in a specific CI build, are automatically batched together, with a batch name that includes the Git commit information.
  • The Bitbucket pull request panel includes a status indicating the latest visual UI test results and merge status.
  • The Bitbucket Commits panel also includes a status indicating the visual UI test result.

  • When the user initiates a Bitbucket pull-request merge, this will also trigger an Eyes branch merge.
  • When the user does a commit that triggers CI, the Applitools test status is updated to reflect the result of the test.
The remainder of this article has two parts. The first few sections describe the one time setup required to start using the integration:
The remainder of the article describes how you can use this integration in your daily development workflow:

This article focuses on the Eyes Bitbucket integration and assumes that you are already acquainted with Bitbucket and a CI system that is integrated with Bitbucket.

Getting started

How you integrate Eyes with Bitbucket depends on various factors which effect the communication of status information between Eyes and Bitbucket. There are three typical scenarios, depending on whether you use the Bitbucket Cloud or the Bitbucket Server (Data Center) and in the case of the server how the Eyes server interacts with the Bitbucket server:

  1. You use the Bitbucket Cloud - see section How to configure the Eyes Bitbucket Cloud integration

  2. You have a Bitbucket Server and the Eyes server can directly interact with Bitbucket - see How to configure the Eyes Bitbucket Server or Data Center integration :

  3. You have a Bitbucket Server that is on a secure network which the Eyes server can't access, but the Applitools supplied proxy service can execute and can act as a bridge between the two systems- this is described in the article Using the Applitools Bitbucket proxy service

How to configure the Eyes Bitbucket Cloud integration

Navigate to your Admin/Team page:

How to configure the Eyes Bitbucket Server or Data Center integration

Navigate to your Admin/Team page:
  • Use the Page navigator The Page navigator is a control at the top of the Test manager window that is used to switch between the main pages of the Test manager. It also displays the name of the current page. to display the Admin page. (You will only see this page in the Page navigator if you have Admin privileges).
  • Select the Teams  tile.
  • Click on the row of the team you want to configure to open the team configuration page.
  • Scroll down the page to find the Bitbucket section:

  • Add a Bitbucket server to integrate with. In the server selection menu, look for, and select your Bitbucket server. If it is not on the list, then click the Add button. This will open up a wizard which will guide you through the rest of the process; you will need the domain name of your Bitbucket Server or Data Center to start this process.
  • If the Bitbucket Server/Data Center are deployed behind a company firewall you need to whitelist inbound HTTPS traffic from the Eyes server to your Bitbucket server in your company firewall. The Eyes server will be if you use the Eyes public cloud or with your organizations name if you have a dedicated cloud or on-prem Eyes system. If you can't whitelist the Eyes server then you should use the Applitools proxy server as described in the article Using the Applitools Bitbucket proxy service instead of the configuration procedure described in this section.
  • Configure your team's repositories. Click the Manage repositories button. This will open up a wizard which will guide you through the rest of the process. As part of the process you need to select an authentication method. You can choose from using a Personal Access Token (PAT) or Basic Authorization.
  • If using a Personal Access Token, then when creating the PAT you must set Admin level permissions for accessing the repositories. For information on creating a PAT see generating Personal access tokens.
  • If using Basic Authorization to access the server, then you need only provide the User name and Password credential pair for accessing the server.

How to configure your CI

You need to set up your CI to configure the Eyes SDK with information related to the Bitbucket pull-request or commit request. This involves two steps:

  • Setting the environment variable APPLITOOLS_API_KEY to your API key.
  • Configuring the build so that before it runs the visual UI test, it sets the environment variable APPLITOOLS_BATCH_ID to a value generated by the CI.

In the BitBucket configuration file .bitbucket-pipelines.yml, add the following lines:

	- export APPLITOOLS_API_KEY="Your Api Key"

$BITBUCKET_COMMIT is the default environment variable for BitBucket commit SHA, which is used with the BitBucket CI/CD.

Replace the string "Your Api Key" with your API key.

How to prepare your code

The SDK uses the value of the environment variable APPLITOOLS_API_KEY to obtain the Applitools API key and the value APPLITOOLS_BATCH_ID to obtain the batch ID. Older versions of the SDK do not do this. We recommend updating your SDK.

Working with Bitbucket

Once your system is set up as described above, you use your CI and Bitbucket in the usual way. A CI job is normally triggered by a push to your repository. This, in turn, will then run your visual UI tests.

The Eyes Bitbucket integration is based on the Bitbucket pull request. When there is an open pull request, Eyes adds the following functionality to your Bitbucket workflow:

  • When your CI initiates a build that includes an Eyes test, Eyes detects that the run is associated with a Bitbucket pull request and will search for a baseline to use as a reference. Eyes will check for a matching baseline in the following order:
    • On a branch that corresponds to the repository name and source branch defined by the pull request.
    • If there is no such branch, it will create a branch and populate it with the latest baseline for that test from the repository name and target branch defined by the pull request.
    • If there is no such baseline the test will be marked as "New".
  • Eyes sends Bitbucket status information that is displayed on the Bitbucket Pull requests panel as shown below. Similar information is also displayed in the Bitbucket Commit panel.

  • The item labeled tests/applitools indicates the results of the visual UI test that ran as part of this build. In this case, the indicates that all the visual UI tests passed. A indicates that either some differences were found and need to be resolved, that the test failed to complete successfully, or that the merge process identified conflicts. Whereas, a indicates that the test is in progress.
  • The item label, test/applitools, functions as a link. If you click the link, then the browser will switch to the Test results screen of the Eyes Test Manager, showing the results of the tests in the most recent build related to that pull request. When dealing with differences, if you Accept all the steps of all the tests, then when you go back to the Bitbucket Builds panel the (error/difference) will be replaced by a (passed).
  • The symbols adjacent to the item labeled scm/applitools represents the merge conflict status. An indicates that there are merge conflicts. After you resolve any conflicts Eyes will send an updated status to Bitbucket, and you will see a icon instead next to the scm/Applitools item.
  • If you click on scm/applitools, then your browser will navigate to the Merge branches screen in the Test Manager. See How to review branch differences and resolve them for more details.
  • You can merge the baseline changes of the source branch with the target branch in two ways:
    1. By clicking on the Merge button in the Merge Branches screen in the Test Manager (enabled only after all conflicts have been resolved). This operation is independent of any merge process in Git, and occurs immediately, even if the Bitbucket merge was not done.
    2. Merge the Bitbucket pull-request as usual. This will first merge the Git files and, once this has completed successfully, Eyes will be notified that it should merge the visual UI test baselines.

How to review branch differences and resolve them

Merging an Eyes source branch with a target branch results in adding new baselines to the target branch or updating its existing baselines according to baseline changes found in the source branch. Changes can include both updated baseline images as well as new, changed or removed annotations (e.g., an added ignore region). Eyes knows how to compare two baselines and determine if they can be merged automatically or if some sort of manual user resolution is required. For example, an ignore region added to a source baseline step that did not change in the target baseline can be automatically added to the target baseline, but if another ignore region was added to the same step in the target baseline, user intervention is required to determine whether to keep only one of the added regions, or to keep both of them.

As described above, the item on the Bitbucket pull-request page marked “scm/applitools” indicates the merge status of the Eyes baselines in the source branch.

When you click on the link on this item, the browser will switch to the Test Manager "Merge branch" page reloaded with the source and target branches derived from the pull-request. If there are no conflicts, then the Test Manager will first display a pop-up window stating this.

Press “Got it!” to remove the pop-up.

A full description of merging is not within the scope of this article; we will only briefly mention the main features available on the merge branches page.

The merge branches page shows all the baselines that have changed in the source branch with respect to the target branch. That is, either they don't exist in the target branch, or they exist in the target branch but have changed in the source branch. Note that baselines that have changed in the target branch and were not changed in the source branch will not be listed and are ignored in the merge process.

At the far left of each row, the status can show a value of “Conflict” indicating that the baseline needs resolving by the user.

You resolve conflicts by using the buttons at the right of each row, which allow you to choose if you want to replace the target baseline with the source baseline or to keep the target baseline unchanged. Choosing one of these will turn the “Conflict” status into a “Resolved” status. You can also compare the two baselines and manually edit the source baseline to include the changes from both branches.

An alternative to resolving each test individually is to use the checkbox at the start of each row to select rows and then use the buttons in the toolbar at the top left of the window to choose how to resolve the selected rows. The toolbar also has buttons that allow you to delete selected baselines in the source branch or to undo the resolution made for the selected baselines.

Once you have resolved all conflicts, the status of the scm/applitools item in the Bitbucket pull-request page will change to green (passed), and you can proceed to merge the branches, either from within test manager by clicking the Merge button, or via the Bitbucket pull-request page as described above.