WebdriverIO 4 tutorial

1. ๐Ÿค– How it works

Applitools SDKs works with existing test frameworks and simply takes screenshots of the page, element, region or an iframe and uploads them along with DOM snapshots to our Eyes server. Our AI then compares them with previous test executions' screenshots (aka Baselines) and tells if there is a bug or not. It's that simple!

Applitools AI with RCA picture

1.1 Baseline vs. Checkpoint images

When you first run the test, our A.I. server simply stores those first set of screenshots as Baseline images. When you run the same test again (and everytime there after), the A.I. server will compare the new set of screenshots, aka Checkpoint images, with the corresponding Baseline images and higlights differences in pink color.

Baseline vs Checkpoint
The picture above is showing the Side-by-Side view of the baseline and checkpoint images

1.2 Marking the test as "Pass" or "Fail"

When the AI compares the baseline and the checkpoint image, if it finds a legitimate difference, it'll mark the test as Unresolved. This is because the AI doesn't know if the difference is because of a new feature or a real bug and will wait for you to manually mark it as a Pass/Fail for the 1st time.

If you mark the unresolved checkpoint image as a "Fail", any further runs with similar difference will be automatically marked as "Failed".

Mark the checkpoint as a fail
The picture above is showing how to mark the checkpoint image as Failed

If you mark the unresolved checkpoint image as a "Pass", then it means that the difference is due to a new feature and so we update the new checkpoint image as the new baseline and mark the current test as Pass. And going forward we'll compare any future tests with this new baseline.

Mark the checkpoint as a Pass
The picture above is showing how to mark the checkpoint image as Passed

Note:

  • Applitools AI has been trained with 100s of millions of images. It doesn't do a pixel-by-pixel comparison because it leads to a lot of false positives, but instead simulates real human eyes and ignore normal differences that humans would ignore and only highlight those that humans would highlight as bugs.

  • ACCURACY: A.I's current accuracy rate is 99.9999%! Which means for most applications that odds that you'll see false-positives is 1 in a million!

A powerful test results dashboard

We provide a state-of-the-art dashboard that makes it very easy for you to analyze differences, report bugs straight from the dashboard and so on.

Seeing test result summary
The picture above is showing the summary view
2. ๐Ÿ–ผ Analyzing differences

The following Gifs shows various tools Applitools provide to easily analyze various differences

Highlight differences between the baseline and checkpoint

Highlight diffs

Zoom into diffs

Zoom into diffs

Toggle differences between baseline and checkpoint

Toggle differences between baseline and checkpoint

Show both baseline and checkpoint side-by-side

Show both baseline and checkpoint side-by-side
3. ๐Ÿž Reporting bugs (straight into Jira or Github) You can simply select a section on the image and directly file a bug in Jira or Github. No need to manually take screenshots, write steps and explain things! Reporting bugs to Jira or Github
4. โœ… Prerequisites
  1. Create a free Applitools account and get the Applitools API KEY API-key

  2. Please set the APPLITOOLS_API_KEY environment variable

    • Mac: export APPLITOOLS_API_KEY='YOUR_API_KEY'

    • Windows: set APPLITOOLS_API_KEY='YOUR_API_KEY'

    TIP

    1. You may skip this step if you want to hard code the API KEY inside the tutorial project's code.
    2. It's better to store APPLITOOLS_API_KEY in the system variables (in Windows) or in the ~/.bash_profile (in Mac) so that it is accessible from all Terminal shells
  3. Download Selenium Standalone server [https://www.seleniumhq.org/download/]

    • Note that this is a jar file. So you should also have Java installed on your machine.
  4. Run the standalone server jar file - it should look something like below:

    • java -jar selenium-server-standalone-3.141.59.jar (Replace the jar file name with your jar file name)
    • This will run Selenium on localhost and on port 4444
  5. Install node.js from https://nodejs.orgโ€‹

  6. Install git from https://git-scm.comโ€‹

    TIP

    Installing git is optional. You need this mainly to clone the demo project from the Github repository. Instead of installing git, you can simply download the Zip file from the repo. Further, If you are Mac, you already have git.

  7. Install Google Chrome browser from https://www.google.com/chrome/

  8. Install ChromeDriver on your machine and make sure it's is in the PATH.

Below are some resources from the internet that'll help you

Tip: On Mac, place the chromedriver executable in /usr/local/bin folder so Eclipse and IntelliJ can find it.

5.1 ๐Ÿš€ - Run the existing demo app
  1. Get the code:

    • Option 1: git clone https://github.com/applitools/tutorial-webdriverio-basic.git
    • Option 2: Download it as a Zip file and unzip it.
  2. Run the Selenium standalone server jar file in a different CLI. It should look something like below:

    • java -jar selenium-server-standalone-3.141.59.jar (Replace the jar file name with your jar file name)
    • This will run Selenium on localhost and on port 4444
  3. cd tutorial-webdriverio-basic folder

  4. Note: Remember to set the APPLITOOLS_API_KEY env variable.

  5. npm install

  6. npm run test

  7. After you run one set of tests, you now have the baseline. Run the same test but with this URL: https://demo.applitools.com/index_v2.html. This version of the demo app has some visual bugs so you can see how it all works. And the second set of screenshots are called "Checkpoint images".

5.2 ๐Ÿค“ - Add Applitools to an existing project

1. Install the Eyes SDK

Note: This is for Webdriverio4. There is a different tutorial for WebdriverIO 5.

npm install @applitools/eyes-selenium@^4.31.9 --save-dev
npm install @applitools/eyes.webdriverio@^2.11.0 --save-dev
1
2

2. Import the SDK


const {
  By,
  ClassicRunner,
  Eyes,
  Target
} = require('@applitools/eyes.webdriverio');
const {Configuration} = require('@applitools/eyes-selenium');

1
2
3
4
5
6
7
8
9

3. Initialize the SDK


const runner = new ClassicRunner();

eyes = new Eyes(runner);

const configuration = new Configuration();
configuration.setAppName('Demo App');
configuration.setTestName('Smoke Test');
eyes.setConfiguration(configuration);

  // Add your API key 
  // -  To auto-load the API key from env, set it as APPLITOOLS_API_KEY env variable (RECOMMENDED)
  // -  To hardcode the API key, set it below
  // eyes.setApiKey('{APPLITOOLS_API_KEY}'); // ๐Ÿ‘ˆ๐Ÿผ REPLACE ME!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

4. Start the test


    //Start the test by calling eyes.open
    //Note that you can send additional parms like app name, viewport etc
    driver = await eyes.open(driver);


1
2
3
4
5
6

5. Add eyes.checkWindow to take a screenshot

Every checkWindow or (similar methods such as check, checkRegion, checkElement, checkFrame) take a screenshot and upload it to the server for analysis. And since they are part of the test, each screenshot essentially becomes a test step. Note that you can add as many check statements as you need after you open the tests.

    // Visual checkpoint #1.
    await eyes.check('Login Window', Target.window());

    // Click the "Log in" button.
    await driver.click(By.id('log-in'));

    // Visual checkpoint #2.
    await eyes.check('App Window', Target.window());
1
2
3
4
5
6
7
8

6. End test

  await eyes.closeAsync();
1

7. Get the test results

    // Close the browser.
    await driver.quit();

    // If the test was aborted before eyes.close was called, ends the test as aborted.
    await eyes.abortIfNotClosed();

    // Wait and collect all test results
    const allTestResults = await runner.getAllTestResults();
    console.log(allTestResults);

1
2
3
4
5
6
7
8
9
10

Everything put togeather

'use strict';

const chromedriver = require('chromedriver');
const webdriverio = require('webdriverio');
const {
  By,
  ClassicRunner,
  Eyes,
  Target
} = require('@applitools/eyes.webdriverio');
const {Configuration} = require('@applitools/eyes-selenium');


let driver;
let eyes;

describe('wdio', function () {

  before(async function () {
    chromedriver.start();
  });

  beforeEach(async () => {
    const chrome = {
      desiredCapabilities: {
        browserName: 'chrome'
      }
    };
    driver = webdriverio.remote(chrome);
    await driver.init();
  });

  afterEach(async () => {
    await driver.end();
    await eyes.abortIfNotClosed();

    const results = await eyes.getRunner().getAllTestResults(false);
    console.log(results);
  });

  after(async function () {
    chromedriver.stop();
  });

  it('Classic Runner Test', async () => {
    await driver.url('https://demo.applitools.com');

    const runner = new ClassicRunner();

    eyes = new Eyes(runner);

    const configuration = new Configuration();
    configuration.setAppName('Demo App');
    configuration.setTestName('Smoke Test');

    eyes.setConfiguration(configuration);
    driver = await eyes.open(driver);

    await driver.url('https://demo.applitools.com');

    // Visual checkpoint #1.
    await eyes.check('Login Window', Target.window());

    // Click the "Log in" button.
    await driver.click(By.id('log-in'));

    // Visual checkpoint #2.
    await eyes.check('App Window', Target.window());

    await eyes.closeAsync();
  });

});
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
6. ๐Ÿš€ Try Visual Grid ๐Ÿ”ฅ

In real world testing, you'll need to test your app on multiple browsers, viewports and mobile devices. So you'll need to run and re-run ALL the tests in different browsers and multiple viewports to ensure things are fine - which could take hours and hours to complete. You may also encounter browsers not opening properly, or hanging and what not.

What if you could run in just one browser, just once, and still do cross-browser tests across various browsers, viewports and multiple mobile emulators? That's where Visual Grid [BETA] comes in.

TL;DR

When you execute this tutorial, it'll run the test locally in Chrome browser but will show results in the following 5 combinations:

  1. Chrome browser (800 X 600 viewport)
  2. Chrome browser (700 X 500 viewport)
  3. Firefox browser (1200 X 800 viewport)
  4. Firefox browser (1600 X 1200 viewport)
  5. An iPhone4 emulator

... all in ~30 seconds!โฐ๐Ÿš€

Sounds interesting?

Go through our Visual Grid tutorial!๐Ÿš€




โš™๏ธ 7. Troubleshooting Common Issues
  1. Forgetting to set your API key (or getting 401 exception).

  2. Applitools is not opening the app (appears to crash).

    • In order to make sure the screenshots are consistent across different tests runs. Applitools uses Viewport as an input. A viewport is simply the size of the screenshot you are taking (width and height). Applitools tries to resize the browser to that size before taking the screenshot. If you are running the app on a small laptop and/or have set the Viewport much larger than the laptopโ€™s screen, then youโ€™ll see this error.
    • Solution: Reduce the height or the width. For more see here.
  3. Not properly loading the API key from the environment variable into your IDE (like Eclipse).

    • After setting the APPLITOOLS_API_KEY in the environment variable to hold your Applitools API key, open the IDE from the command line terminal (and not from the IDE directly). On Mac, itโ€™d look like this: Open a Terminal and then type: open ~/Applications/Eclipse.app. This will load Eclipse with all the environment variables.
  4. Trying to run the test but nothing happens

    • Check if you the APPLITOOLS_API_KEY is null
  5. Incompatibility between Chrome browser and Chrome driver.

  6. Debug logs

    • See this article to enable debug logs to help file support ticket

Resources




Terms & Conditions Privacy Policy GDPRยฉ 2019 Applitools. All rights reserved.

Last Updated: 11/5/2019, 5:10:59 AM