Cypress Tutorial

1. 🤖 How it works

The Applitools Eyes Cypress SDK is a simple plugin to Cypress. Once installed, it adds few commands to the main cy object. Specifically, it adds three main methods, cy.eyesOpen to start the test, cy.eyesCheckWindow to take screenshots (for each test step) and cy.eyesClose to close the test.

Note: Supports Chrome, Firefox and Mobile simulators

The interesting thing is that even though you run your Cypress tests using Chrome browser, you can still add visual validations using both Chrome and Firefox! This is because we upload the DOM and assets to our Visual Grid that was rendered in Chrome and open it in Firefox or Chrome as per the settings. The same thing happens in mobile simulators as well.

The picture below shows how it works

cypress-marketecture.png
  1. You run the Cypress test as usual

  2. Everytime you call cy.eyesCheckWindow, the SDK grabs the DOM and other assets asynchronously. And stores it locally until all the tests are done.

  3. Once all the Cypress tests are done, the SDK uploads all the assets to the Visual Grid server

  4. The Visual Grid server then opens up the assets in real browsers (Firefox or Chrome), or in mobile simulators, and with appropriate Viewport depending on what was set in the tests. Note that all these happen concurrently in the cloud.

  5. As and when the rendering happens, the Applitools Eyes server gets notification about it and the eyes server then compares the pictures using AI and advanced algorithms and shows the results.

1.1 Baseline vs. Checkpoint images

When you first run the test, our A.I. server simply stores those first set of screenshots as Baseline images. When you run the same test again (and everytime there after), the A.I. server will compare the new set of screenshots, aka Checkpoint images, with the corresponding Baseline images and higlights differences in pink color.

Baseline vs Checkpoint
The picture above is showing the Side-by-Side view of the baseline and checkpoint images

1.2 Marking the test as "Pass" or "Fail"

When the AI compares the baseline and the checkpoint image, if it finds a legitimate difference, it'll mark the test as Unresolved. This is because the AI doesn't know if the difference is because of a new feature or a real bug and will wait for you to manually mark it as a Pass/Fail for the 1st time.

If you mark the Unresolved checkpoint image as "Failed", it'll only mark the current test result as Failed. Mark the checkpoint as a fail

The picture above is showing how to mark the checkpoint image as Failed

Note:

To automatically mark the checkpoint as a "Fail" in the future test runs, you need to do the following:

  1. Annotate at least one of differences as a "bug region"
  2. Select the "Fail tests" checkbox in the popup window
  3. Press "Thumbs Up" (not "Thumbs Down") button in the checkpoint image's toolbar (Note: this is counter-intuitive. But what happens is that, we now create a new Baseline along with this bug and "Failed" metadata. So if the same image with the exact bug appears, it'll fail again)
  4. Press "Save" in the main toolbar

If you mark the unresolved checkpoint image as a "Pass", then it means that the difference is due to a new feature and so we update the new checkpoint image as the new baseline and mark the current test as Pass. And going forward we'll compare any future tests with this new baseline.

Mark the checkpoint as a Pass
The picture above is showing how to mark the checkpoint image as Passed

Note:

  • Applitools AI has been trained with 100s of millions of images. It doesn't do a pixel-by-pixel comparison because it leads to a lot of false positives, but instead simulates real human eyes and ignore normal differences that humans would ignore and only highlight those that humans would highlight as bugs.

  • ACCURACY: A.I's current accuracy rate is 99.9999%! Which means for most applications that odds that you'll see false-positives is 1 in a million!

A powerful test results dashboard

We provide a state-of-the-art dashboard that makes it very easy for you to analyze differences, report bugs straight from the dashboard and so on.

Seeing test result summary
The picture above is showing the summary view
2. 🖼 Analyzing differences

The following Gifs shows various tools Applitools provide to easily analyze various differences

Highlight differences between the baseline and checkpoint

Highlight diffs

Zoom into diffs

Zoom into diffs

Toggle differences between baseline and checkpoint

Toggle differences between baseline and checkpoint

Show both baseline and checkpoint side-by-side

Show both baseline and checkpoint side-by-side
3. 🐞 Reporting bugs (straight into Jira or Github) You can simply select a section on the image and directly file a bug in Jira or Github. No need to manually take screenshots, write steps and explain things! Reporting bugs to Jira or Github
4. ✅ Prerequisites
  1. Create a free Applitools account and get the Applitools API KEY API-key

  2. Please set the APPLITOOLS_API_KEY environment variable

    • Mac: export APPLITOOLS_API_KEY='YOUR_API_KEY'

    • Windows: set APPLITOOLS_API_KEY='YOUR_API_KEY'

    TIP

    1. You may skip this step if you want to hard code the API KEY inside the tutorial project's code.
    2. It's better to store APPLITOOLS_API_KEY in the system variables (in Windows) or in the ~/.bash_profile (in Mac) so that it is accessible from all Terminal shells
  3. Install node.js from https://nodejs.org​

  4. Install git from https://git-scm.com​

    TIP

    Installing git is optional. You need this mainly to clone the demo project from the Github repository. Instead of installing git, you can simply download the Zip file from the repo. Further, If you are on Mac, you already have git.

  5. Install Google Chrome browser from https://www.google.com/chrome/

5. 🚀 Run the demo app

1. Clone the demo app and install dependencies

  • git clone https://github.com/applitools/tutorial-cypress.git
  • cd tutorial-cypress
  • npm install
  • npx eyes-setup //this will install Applitools plugin to the project

2 Set the APPLITOOLS_API_KEY

You can set it as an environment variable like below:

Envirnomant variable:

  • Linux/Mac: export APPLITOOLS_API_KEY=<your_key>
  • Windows: set APPLITOOLS_API_KEY=<your_key>

You can also set all the Applitools related configuration in the applitools.config.js that's in the root directory. The below file shows how to setup concurrency, apiKey and the browser x viewports combinations. For full documentation, go here.

module.exports = {
    concurrency: 1,
    apiKey: 'APPLITOOLS_API_KEY',
    //
    browser: [
        // Add browsers with different viewports
        {width: 800, height: 600, name: 'chrome'},
        {width: 700, height: 500, name: 'firefox'},
        {width: 1600, height: 1200, name: 'ie11'},
        {width: 1024, height: 768, name: 'edgechromium'},
        {width: 800, height: 600, name: 'safari'},
        // Add mobile emulation devices in Portrait mode
        {deviceName: 'iPhone X', screenOrientation: 'portrait'},
        {deviceName: 'Pixel 2', screenOrientation: 'portrait'}
    ],
    // set batch name to the configuration
    batchName: 'Ultrafast Batch'
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

2. Run the test

  1. Interactive
./node_modules/cypress/bin/cypress open
1

Click on the ultraFastGrid.spec.js test. the tests using Chrome just once and sends the DOM snapshot to the Ultrafast Grid. The Grid will then render the DOM snapshot in the cloud. You need to wait 15-30 seconds.

  1. CLI
  npm test

  //or run this:
./node_modules/cypress/bin/cypress run --spec "cypress/integration/examples/ultraFastGrid.spec.js"
1
2
3
4

An example code

describe("AppTest", () => {

    it(`ultraFastTest`, function () {
        // Navigate to the url we want to test
        // ⭐️ Note to see visual bugs, run the test using the above URL for the 1st run.
        // but then change the above URL to https://demo.applitools.com/index_v2.html
        // (for the 2nd run)
        cy.visit('https://demo.applitools.com');

        // Call Open on eyes to initialize a test session
        cy.eyesOpen({
            appName: 'Demo App',
            testName: 'Ultrafast grid demo',
        })

        // check the login page with fluent api, see more info here
        // https://applitools.com/docs/topics/sdk/the-eyes-sdk-check-fluent-api.html
        cy.eyesCheckWindow({
            tag: "Login Window",
            target: 'window',
            fully: true
        });

        cy.get('#log-in').click()

        // Check the app page
        cy.eyesCheckWindow({
            tag: "App Window",
            target: 'window',
            fully: true
        });

        // Call Close on eyes to let the server know it should display the results
        cy.eyesClose()
    });

});
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


For full documentation, go here.

⚙️ 7. Troubleshooting Common Issues
  1. Forgetting to set your API key (or getting 401 exception).

  2. Applitools is not opening the app (appears to crash).

    • In order to make sure the screenshots are consistent across different tests runs. Applitools uses Viewport as an input. A viewport is simply the size of the screenshot you are taking (width and height). Applitools tries to resize the browser to that size before taking the screenshot. If you are running the app on a small laptop and/or have set the Viewport much larger than the laptop’s screen, then you’ll see this error.
    • Solution: Reduce the height or the width. For more see here.
  3. Not properly loading the API key from the environment variable into your IDE (like Eclipse).

    • After setting the APPLITOOLS_API_KEY in the environment variable to hold your Applitools API key, open the IDE from the command line terminal (and not from the IDE directly). On Mac, it’d look like this: Open a Terminal and then type: open ~/Applications/Eclipse.app. This will load Eclipse with all the environment variables.
  4. Trying to run the test but nothing happens

    • Check if you the APPLITOOLS_API_KEY is null
  5. Incompatibility between Chrome browser and Chrome driver.

  6. Debug logs

    • See this article to enable debug logs to help file support ticket
8. 📚 NPM Repo Docs and more

The @applitools/eyes-cypress npm package page contain more information about advanced configuration, best practices and troubleshooting. Visit eyes-cypress SDK repo to find out more about what's possible with Applitools' Cypress SDK.

Learn more

Applitools Eyes is a powerful platform for automated visual UI testing that supports full page screenshots, page layout matching, cross-device and browser testing, test batching, baseline branching and merging, automated baseline maintenance, collaboration features, and much more. Applitools has over 40 SDKs supporting a broad range of testing environments.

Reference documentation

To learn more, check out the Applitools Eyes documentation and tutorials for other testing environments.

Request a demo

If you want to see a demo of all our other features, you do request a demo

Knowledge base and Support

You can search our Knowldege base for more information. You can also file a contact our support team and file a Ticket.


Terms & Conditions Privacy Policy GDPR© 2020 Applitools. All rights reserved.

Last Updated: 5/22/2020, 8:21:31 PM