Black Box Testing
5 Tools Used by Professional QA Teams
The Power of Black Box Testing
For many years, black box testing has been a powerful and popular approach in software testing. The appeal of black box testing is that it exposes where a system breaks from the user’s perspective. A black box test “doesn’t care” about the specifications behind the system, or how exactly it was implemented. It looks at the actions a user should be able to perform, and asks the question, “what happens when a user tries to do that? Does it really work?”
A big advantage of this approach is that it can expose gaps in the system. For example, take a simple login screen, and the specifications to which it was built:
A white box test plan would go over the specs and make sure they are implemented correctly:
- Check if username/password fields exist and show default values
- Type in username without password or vice verse user receives error
- Type in username that does not exist in system user receives error
- Type in password less than 6 chars user receives error
- Type in username that exists and password over 6 chars user logged in
You’ve probably noticed what’s missing here no validation that the password is correct! So while all 5 white box tests could pass, in reality the system would be broken.
A black box test plan would ignore the specs and just test the user story: “User signs in with username and should get access only if they type the right password.” The tests would be:
- Type in existing username and correct password user logged in
- Type in existing username and incorrect password user receives error
Test 1 would pass, but Test 2 would fail and expose that the system does not actually check the password at all.
(Note in reality there would be many more than 2 tests; we would need to test many possible usernames and passwords to see if and when the system falls down. This is a challenge of black box testing covering all relevant user workflows and inputs)
In this simple example it’s quite easy to see the gap in the system, because we are so familiar with login forms. But even in slightly more complex systems, it might not be immediately obvious that something in the spec or implementation is incorrect. White box tests can verify the implementation but can’t make sure the user story is fulfilled. That’s what black box tests are for.
Black Box and Agile Development
Although black box testing is not formally part of the agile methodology, it is strongly related. A basic tenet of agile development is that software should not be built to a set of meticulous specifications which are supposed to represent what users need. Rather, developers should quickly build something simple that provides value to a user, release it and allow users to actually use it. Then they can add more capabilities that provide even more value, in an iterative manner, verifying that each new capability really provides value for users.
Black box testing fits in perfectly with agile because the tests are planned in accordance with the user story. So they test the value users receive, rather than verifying the implementation, as in the white box approach.
Another way black box testing fits in with agile is that it can move fast:
- Tests are not reliant on a specific implementation, so if developers switch to a new implementation, as long as the user story remains the same, tests should remain valid.
- In some cases, black box testing enables you to plan a test and even implement an automated testing script before the new feature is developed (like in Test Driven Development). This can save a lot of time as you can build your tests in parallel to each development sprint, and not have to start working when development ends.
Black Box Test Automation
Because of the need to make sure that users are really receiving the value promised by the software, many powerful testing tools have emerged, which make it easy to do black box testing in an automated and repeatable manner.
Why is automation important? Because black box testing is very broad, and ideally, should test all possible inputs and usage conditions. In reality it’s never possible to test all the options. But in order to increase test coverage as much as possible, it is desirable to automate tests. When tests are automated, it’s possible to run many more tests covering many more inputs and conditions, with the same limited testing resources.
Today, there are three primary types of software applications that are developed and tested, and three main approaches to do black box testing:
Types of Software Applications
- Desktop applications the traditional form of application, which is installed and used on a local operating system such as Windows, Linux or iOS. For example, Microsoft Word.
- Web applications these are complex applications which are accessed via the Internet using regular web browsers. For example, Gmail, Facebook.
- Mobile applications these are applications which are either natively installed on a mobile device (apps) or web applications which are accessed by a web browser running on a mobile device. For example, Whatsapp, Waze, Amazon.com mobile site.
Technical approaches of black box testing tools
Coded Scripting
Some tools provide a scripting language that allow testers, with some technical knowledge, to write a series of instructions that describe what user operations should be performed in the test. For example, go to a certain URL, click the “login” button, and type in the username “john” and the password “123”. It is then possible to “run” or “execute” the test script, and those operations are automatically simulated on the application under test, as if a user performed them.
A test script must also perform validation checking if the interface functioned correctly and if the test passed or failed. To this end, testers add assertions to their test scripts code that checks if the expected result occurred. For example, a login test could end with an assertion that the title of the next page is “Welcome John”. If it isn’t, the test will fail.
Record/Playback
Some tools provide a graphic interface in which testers can define which user operations should be performed in the test. Testers either pick and choose elements of the software application and possible user actions, or perform the user operations directly themselves and “record” them, generating a testing script behind the scenes. Record/playback scripting is more accessible for testers with less technical knowledge, and makes it easier to build and test scripts, but is also less flexible than coded scripting.
Here too, like in coded scripting, testers need to implement assertions that verify the expected results of the test. In most cases the validation part of the test cannot be “recorded”, so it is necessary to specify it using a complex set of rules or by writing code.
Visual UI testing
A new set of tools go about it a different way: they take a graphic snapshot (screenshot) of the application when it is working properly. Then, when the system is under test, user inputs are simulated and another screenshot is taken. The visual UI testing tools then compare the first and second screenshots, and report differences between the two images, which might represent bugs.
Visual UI testing is different from the other two approaches because it provides a built-in system for validation, without requiring the user to explicitly define what output they expect or what might go wrong. A visual UI testing system can report on everything that goes wrong on a screen at one time even unexpected errors. On the other hand, testers need to explicitly define the simulation part how to click through to the right place in the interface and insert the user inputs.
That’s why scripted coding or record/playback systems and visual UI testing systems are complementary the former can be used to simulate a user action, and the latter can be used to perform powerful validation of bugs in the user interface or in the outputs returned by the system.
Black box testing tools by type of application and technical approach
There are hundreds of testing tools out there. In this post we cover 5 powerful and popular tools used by pro testers. The below breaks down these 5 tools and shows which technical approach each of them uses and which applications they are able to test.
| Desktop App | Web App | Mobile App/Site | |
|---|---|---|---|
| Coded Scripting |  Coded UI  |  Selenium |  |
| Record/Playback | Coded UI Test Builder | Selenium IDE | No established market leader |
| Visual UI Testing |  | | |
The 5 testing tools surveyed in this post, broken down by technical approach and apps tested
Pro Tool #1: Selenium Testing Web Apps Via Scripting or Record/Playback
What is it?
Selenium (AKA “Webdriver”) is a downloadable open source tool that does browser automation. Selenium contains a variety of drivers for almost all web browsers.
Selenium lets you write test scripts in the programming language of your choice (e.g. Javascript, Ruby/Rails, Python) these test scripts specify what the browser should do, for example, find a certain button in the HTML code of the page and click it.
When the user reaches a page on which there is a desired outcome, you need to perform validation, by writing custom code that checks what is on the page and performs an “assertion”, checking if the desired outcome is there. For example, after logging in with user “David”, you can assert whether the site shows “Welcome David” at the top of its interface. By design, validation is not part of the Selenium project, and can be performed by testers in their native programming language.
Writers of Selenium scripts need proficiency in HTML, CSS and object selector methods (such as XPATH) to find the relevant user interface elements on the web page. Selenium provides built-in commands that let the test scripts perform different user actions such as click, scroll, type text, navigate to different URL, wait, etc.
An easy way to get started with Selenium is the Selenium IDE. This is a GUI application that lets you press “record”, go to the web browser, perform user actions, and a test script will be produced automatically with all those user actions. Of course it’s not foolproof, and you’ll need to re-run your recorded scripts and tweak the code to make sure it does exactly what you want.
Selenium IDE also allows you to define assertions for validation of test results, by defining conditions on code elements that exist in the page.
The beauty of Selenium is that once you have a test script, the same script can be executed again and again on different browsers, facilitating cross browser testing of the same page or feature.
How does it facilitate black box testing?
Selenium can facilitate black box testing by simulating a user’s workflow in the browser, thus testing if a certain user story, represented by that workflow, works correctly. Selenium is a popular tool with millions of installations, and is probably the most common way black box testing is done today for web applications.
The concept of cross browser testing, which is a common use of Selenium, is closely related to black box testing. Selenium’s support for numerous browsers and platforms makes it possible to accurately simulate users’ journeys through the interface, on the real platforms they are likely to use. In today’s multi-platform environment it is not sufficient to test if we have implemented all the features of the websites correctly (white box testing); we need to make sure that real users on their actual computers are able to access and use those features (black box testing).
A main weakness of Selenium is that to use it, testers need to be proficient in writing code. Many professional testers are not expert coders and for them Selenium will have a steep learning curve. Some of this complexity can be overcome by using Selenium IDE.
Getting started
Want to do black box testing with Selenium? Here’s what you’ll need to get started:
- Download Selenium
- Installing and using Selenium IDE Selenium documentation
- Writing Selenium scripts from scratch Selenium documentation
- In-depth Selenium Tutorial by TutorialsPoint, covering both Selenium IDE and Selenium coded scripting
Pro Tool #2: Appium Testing of Mobile Sites Apps Via Scripting
What is it?
Appium is an open source tool that aims to be an open standard for automated testing of mobile websites and apps. It builds on the success of Selenium as a de-facto standard for browser automation, and allows testers to write scripts using the familiar Selenium API (with some additions for mobile such as a “swipe” command), which will work on mobile sites and apps. Like Selenium, Appium is built and supported by a large open source community.
Another important feature of Appium is that it is fully cross platform, supporting both Android and iOS, native apps and websites accessed through a variety of mobile browsers.
Appium allows testers to write scripts in almost any programming language. However it requires coding skills, and at the time of this writing there is no robust option for Play/Record scripting in Appium.
Appium does not provide validation or assertions as part of the project, but requires users to write custom code to validate that the simulated user actions succeeded or failed.
How does it facilitate black box testing?
Like Selenium, Appium can facilitate black box testing by simulating a user’s workflow in a mobile app or a website accessed via a mobile device browser. Due to Appium’s support of both major mobile operating systems, iOS and Android, and numerous browsers, it can be used to accurately test if users are able to perform actions on their actual devices.
Like Selenium, Appium requires testers to write code, and there is no robust option for play/record scripting. Another downside of Appium is that it is relatively difficult to set up. Appium is a relatively new project and this may improve as new versions are released.
Getting started
Want to do black box testing with Appium? Here’s what you’ll need to get started:
Pro Tool #3: Applitools Eyes Testing of Mobile, Web and Desktop Apps Using Visual Snapshots
What is it?
Applitools Eyes (disclosure we built it 🙂 is a low-cost SaaS solution with a perpetual free plan, which performs testing by capturing and comparing screenshots of user interfaces for mobile, web and desktop applications.
Unlike Selenium and the other tools we review on this page, Applitools Eyes focuses on the validation aspect of testing it provides a convenient way to check if, given a certain user workflow or input parameters, the user interface is functioning properly or returning the correct output.
Applitools Eyes works by recording an initial screenshot of the user interface when it is functioning correctly (called a “baseline”). Then, testers to simulate a user action, capture another screenshot of the application under test, and compare it to the baseline. It is possible to view all the differences between the current screen under test and the “baseline” representing the correct functionality any of these differences could represent bugs or incorrect functionality.
Unlike other types of test validation such as assertions in test scripts Applitools Eyes does not require the tester to explicitly define which problems they expect to find in the user interface. Rather, it exposes any changes visible to the user and lets the tester “approve” them, saying that these changes are acceptable, or “reject” them, indicating that these changes represent a bug.
It is possible to view changes in four different modes:
- Exact any change at the pixel level is indicated
- Strict – shows only significant changes, both at the layout and content level, using image analysis techniques to ignore minor changes that are not significant for the user.
- Content – only identifies differences that relate to the content, (e.g. the text of an article), ignoring layout differences (e.g. the presence of an ad).
- Layout – only identifies layout differences while ignoring content differences.
Applitools Eyes allows testers to manually simulate user actions and then capture a screenshot for comparison with the baseline. For automated testing, it integrates with tools like Selenium, Appium and QTP, enabling testers to simulate user actions via these other tools, and then trigger visual validation via the Applitools Eyes API.
How does it facilitate black box testing?
Visual UI testing is the most distilled form of black box testing, because it operates solely based on what the user sees, with no reliance or consideration for how the application is implemented. The power of black box testing is that it tests the system from the user’s perspective, making it possible to uncover unexpected failures. Visual UI testing does exactly that it identifies any difference in the user interface compared to a known baseline, even if that difference was not explicitly defined as an assertion in the test.
Visual UI testing enables testers to uncover hundreds of changes or possible bugs in one user interface screen using one validation step which would require manually coding or defining hundreds of individual assertions using other testing tools. This makes it possible to dramatically increase black box test coverage, testing many more possible user workflows, inputs and expected outputs.
Applitools Eyes makes visual UI testing practical by helping users focus on the visual differences that are most relevant for the current test, whether they are “cosmetic” or minor design changes (e.g. for browser compatibility tests), content changes (e.g. different messages shown in the user interface) or layout changes.
A weakness of the tool is that it cannot test responses of the application under test which are not visible to the user. For example, visual UI testing cannot validate that a user input was correctly written to a database.
Getting started
Want to do black box testing with Appium? Here’s what you’ll need to get started:
- Applitools Eyes Tutorial see instructions on how to connect it to your tests
- Video Tutorials
Pro Tool #4: HP QTP Testing of Web and Desktop Apps Via Record/Playback
What is it?
HP QuickTest Professional or QTP (now known as Unified Functional Testing), enables testers to perform a set of user actions on an application or website, “record” those user actions, and generate a test script that can “play back” or repeat those operations automatically.
QTP identifies objects in the user interface being tested, and enables testers to view the hierarchy of objects in the application under test. It is possible to edit and modify tests by selecting from these objects and specifying user actions that should be simulated on them.
Validation of tests is done within the QTP UI testers can select user interface elements and specify their expected state at the end of a test. QTP also enables reporting to relevant parties about test successes/failures with details about what exactly caused the test to fail.
However, like Selenium and Appium, in QTP testers must explicitly define their assertions and specify in advance each and every failure that can occur during the test. By integrating with a visual UI testing tool like Applitools Eyes (see tool #3 above), it is possible to perform one-step validation of an entire user interface screen, capturing any possible issues with any interface elements, including unexpected failures that were not explicitly defined as assertions.
The main strength of QTP is that testers can perform automated testing using a convenient GUI, without having to write code. QTP does generate test scripts behind the scenes these are written in VBScript, a relatively simple scripting language. More advanced QTP users can go in and manipulate the VBScript code directly to fine-tune how a test behaves.
How does it facilitate black box testing?
Like Selenium and Appium, QTP enables testers to define a workflow which represents a certain user story, simulate that workflow on the application or website being tested, and see if it operates as expected. If tests are designed around user stories, QTP becomes a black box testing platform. QTP is a very popular solution and probably the most common tool that enables black box testing without knowledge of coding.
However, there are two downsides to using QTP for black box testing:
- Browser/platform support In the past, QTP had limited support for browsers and mobile devices. This has improved in recent versions, but even in the latest version 12 there is no support for Opera and older versions of IE, Firefox and Chrome. Limited support for browsers and platforms means there is limited ability to simulate how users experience the system under test on their actual devices.
- Additional layer above the testing script because QTP provides a UI for defining tests, and generates scripts behind the scenes, there can be gaps between what the tester intended and how the test actually runs. This makes it a bit more difficult to achieve true black box testing, because in some cases the test might not accurately reflect the user story. If the test runs different than expected, or if certain user flows cannot be captured exactly, we are back to testing a specification and not the real way the user will experience the product. To a certain extent this is true in all testing technologies, because a test script cannot accurately capture 100% of a user’s expected actions. But the problem is more severe in systems like QTP because testers do not directly define the test scripts.
Getting started
Want to do black box testing with HP QTP (UFT)? Here’s what you’ll need to get started:
Pro Tool #5: Microsoft Coded UI Testing of Windows Apps Via Coded Scripting or Record/Playback
What is it?
Coded UI is a standard provided by Microsoft which allows testers to automate and simulate user actions on the Windows platform. The strength of Coded UI is that it is based on Microsoft’s knowledge of all possible user interface elements in its platform, and their behavior.
There are two ways to define tests in Coded UI:
- By writing coded scripts in languages such as C#, C++ or VB.
- Using the Coded UI Test Builder, provided as part of Visual Studio, which enables Record/Playback of Coded UI tests.
Unlike Selenium and Appium, Coded UI provides a mechanism, as part of its scripting language, for validating if a test passed or failed. Assertions can be added and executed using Coded UI operations or via the Test Builder interface.
How does it facilitate black box testing?
For desktop or mobile applications running on Windows platforms, Coded UI is an excellent choice for black box testing, because it provides the best simulation of how workflows will be executed by users on those platforms. For other platforms, or browser-based applications, testers will have to combine Coded UI with additional testing methods.
A main weakness of Coded UI is that it relies heavily on coded scripts and requires testers to be proficient in writing code. Some of this complexity can be overcome by using the Test Builder, but in most cases testers will have to “dive into” test scripts and modify them, requiring some level of coding skills.
Getting started
Want to do black box testing with HP QTP/UFT? Here’s what you’ll need to get started (all resources are from MSDN):
Summary
We surveyed 5 tools used by pro testers to do black box testing. The tools are powerful, but each has its limitations and works best in combination with other tools:
- Selenium can automate desktop browsers, but doesn’t support mobile, and doesn’t have a built-in way to do validation of test scripts.
- Appium automates native apps and mobile websites, but does not have a record/playback option, and also does not have a built-in way to do validation of tests.
- Applitools Eyes provides a powerful way to validate tests and visually inspect if an application is functioning correctly. But it needs to be combined with other tools to facilitate automating the test workflows in the UI of the system under test.
- QTP provides a convenient way to define automated tests without requiring testers to be proficient at coding. But it has limited platform support and often will not accurately simulate a test exactly as a tester intended, a downside for black box testing.
- Coded UI is an excellent way to automate user workflows for Windows applications, but it doesn’t support other platforms and also is heavily reliant on coded scripting.