Calabash iOS tutorial
1. 🤖 How it works
Applitools SDKs work with existing test frameworks and takes screenshots of the page, an element, a region or an iframe and uploads them along with DOM snapshots to our Eyes server. Our AI then compares them with previous test executions' screenshots (aka Baselines) and reports if there is a bug or not. It's that simple!
1.1 Baseline vs. Checkpoint images
When you first run the test, our A.I. server stores those first set of screenshots as Baseline images. When you run the same test again (and everytime there after), the A.I. server compares the new set of screenshots, aka Checkpoint images, with the corresponding Baseline images and highlights differences in a pink color.
1.2 Marking the test as "Pass" or "Fail"
When the AI compares the baseline and the checkpoint image, if it finds a legitimate difference, it will mark the test as Unresolved. This is because the AI doesn't know if the difference is because of a new feature or a real bug and will wait for you to manually mark it as a Pass/Fail for the 1st time.
If you mark the unresolved checkpoint image as a "Fail", any further runs with similar difference will be automatically marked as "Failed".
If you mark the unresolved checkpoint image as a "Pass", then it means that the difference is due to a new feature so we set the new checkpoint image as the new baseline and mark the current test as Pass. Going forward we'll compare any future tests with this new baseline.
Applitools AI has been trained with 100s of millions of images. It doesn't do pixel-to-pixel comparison because this can lead to a lot of false positives. It instead simulates human eyes that ignore differences humans can't detect and highlight differences that humans can detect.
ACCURACY: Our A.I.'s current accuracy rate is 99.9999%! Which means for most applications the odds that you'll see false-positives are 1 in a million!
A powerful test results dashboard
We provide a state-of-the-art dashboard that makes it very easy for you to analyze differences, report bugs and much more. For more information on the Applitools dashboard check out these articles.
2. 🖼 Analyzing differences
The following Gifs show various tools Applitools provides to easily analyze various differences
Highlight differences between the baseline and checkpoint
Zoom into differences
Toggle between baseline and checkpoint
Show both the baseline and checkpoint side-by-side
3. 🐞 Reporting bugs (straight into Jira or Github)
You can select a section of the image and directly file a bug in Jira or Github. No need to manually take screenshots, write steps and explain things! To read more about bug regions check out this article.
4. ✅ Prerequisites
Create a free Applitools account and get your Applitools API KEY
- You may skip this step if you want to hard code the API KEY inside the tutorial project's code.
- It's better to store APPLITOOLS_API_KEY in the system variables (in Windows) or in the
~/.bash_profile(in Mac) so that it is accessible from all Terminal shells
Install git from https://git-scm.com
gitis optional. You need this mainly to clone the demo project from the Github repository. Instead of installing
git, you can simply download the Zip file from the repo. Further, If you are using Mac OSX, you already have
Build the application using XCode (ensure that -cal target is built)
5.1 🚀 - Run the existing demo app
Get the code:
- Option 1:
git clone https://github.com/applitools/calabash-test-ios-app
- Option 2: Download it as a Zip file and unzip it.
- Option 1:
From the command line:
- cd path/to/app/home
bundle exec cucumber
5.2 🤓 - Add Applitools to an existing project
Install the SDK
gem install eyes_calabash
Example Calabash iOS Feature
# Tag @eyes tag is used to mark scenarios which are eyes tests # Along with the @eyes tag next tags might be used: # @eyes_app_name "Application Name" # sets app_name for eyes. if skipped, the feature name will be used. # @eyes_api_key "DUMMY_KEY" # The API key is set implicitly from environment variable APPLITOOLS_API_KEY by default. The @eyes_api_key tag will override it. # # @eyes_test_name "a_test_name" # Sets the eyes test name for a particular scenario. If skipped, the scenario name will be used. # # Eyes test scenario should contain one or more those steps: # I check window # If current view contains a scrollable element, it's entire content will be captured # # I check window with description "description" # description is passed as a description of a screenshot # I check viewport window # Captures content of a current view "as is" # I check viewport window with description "description" # I check element "calabaish.element.selector" # Captures content of the element. If the element is scrollable, the entire content will be captured # I check element "calabaish.element.selector" with description "description" # I check viewport element "calabash.element.selector" # Captures the content of the element "as is" the currently visible part will be reported # I check viewport element "calabash.element.selector" with description "description" # Each scenario might have more than one 'check' steps #See lib/applitools/calabash/eyes_hooks.rb for details @eyes @eyes_app_name "CalabashSDK(iOS)" Feature: Sample Feature Scenario: The first screen When I wait Then I check window Scenario: Continue link Then I check element "button marked:'Continue'" @eyes_test_name "viewport window" Scenario: the second screen Given I touch the "Continue" button And I wait Then I check viewport window @eyes_test_name "explicit check of scrollable element" Scenario: the table view Given I touch the "Continue" button And I wait Then I check element "UITableView" @eyes_test_name "check window with scrollable element" Scenario: window (contains scrollable) Given I touch the "Continue" button And I wait Then I check window