How to Scale Mobile Automation Testing Effectively

Learn the best way to scale mobile automation testing today. We’ll look at a history of traditional mobile automation testing strategies and discuss why each one is and isn’t effective, and then see why a new tool, the Native Mobile Library, is different.

In today’s mobile testing world there are many different approaches to scaling our test automation for native mobile applications. Your options range from running locally with virtual devices (simulators/emulators) or real devices, to a local mobile grid/lab, to docker containers/virtual machines, to remote cloud test services.

As you’re probably aware, testing native mobile applications can sometimes be a difficult endeavor. There are a lot of moving parts and many points of failure involved. To successfully execute, everything needs to work in complete harmony.

For example, just executing a single Appium test you need:

An Appium server & all required dependencies installed
A mobile device or emulator/simulator
Valid test code logic
A compiled mobile application
Application web service APIs running and stable (if applicable)

Now let’s say you want to scale your tests across multiple devices for your cross-device validation needs. We’re now introducing more points of failure for each device that is tested. A test on one device may execute just fine but on another, it may fail for various unknown reasons. We then often have to spend a vast amount of time investigating and debugging these failures to find the root cause.

Let’s also consider that because we’re adding more devices we will likely need to add more conditional logic to our test code to accommodate these different devices. We may need to add conditionals for different device resolutions/screen sizes, OS versions, orientations, scroll views, locators/selectors, or maybe gestures for specific devices. This all adds more coded logic to our test suite or framework we need to maintain and thus refactor in the future when our application changes.

Because of these reasons and/or perhaps others, many people often don’t scale their mobile test coverage across different devices. Either it might introduce more test maintenance, more test flakiness, a longer test execution time, or access to different devices is not possible. Crossing fingers and hoping for the best has been the approach for many…

Scaling Mobile Automation Testing

Now let’s cover some common mobile test scaling approaches. I’ll go over the benefits and drawbacks of each of these approaches and finally introduce you to a new modern approach and technology, the Applitools Native Mobile Library.

Sequential Execution

The simplest but also the most inefficient and slowest approach. The example diagram below shows executing two tests (Test A & Test B) executing across three different mobile devices.

As stated before, this is the simplest approach but very inefficient and slow. Perhaps some people aren’t aware of different or better approaches. Or maybe some bad test practices are being used such as one test (Test B) is dependent on (Test A) to complete in order to proceed. Generally, a good test practice is to never have any tests dependent on another to execute. Test suites/frameworks should be architected in a way to run any test in any order by utilizing test hooks/annotations, database seeding, mocking, or API test data injection to set up each test independently.

For example, say I have an app that has a shopping cart and I have a test “Add items to shopping cart” which failed due to a bug in the app. If I relied on that test to then run my “shopping cart” specific tests, I couldn’t. That is where good test architecture comes into play by setting up each test independently by some of the methods mentioned above or other means.

Parallel Device Execution

We live in a fast-paced CI/CD world these days, and feedback loops that are as fast as possible are crucial. So what can you do to help get this test feedback ASAP? Parallelization!

In this scenario, we are parallelizing the devices needed for cross-device validation. It’s however still inefficient since the tests themselves are running sequentially. But still much better than the former sequential approach.

Parallel Test Execution

This approach to mobile automation testing is a bit more efficient as it’s parallelizing the tests in your test suite. Ultimately it should reduce the execution time and also promote good test practices keeping tests independent from one another to execute. However, the inefficiency now is that each device is sequentially tested.

Parallel Test & Parallel Device Execution

This is the most efficient and fastest traditional approach as it parallelizes both your test suite and devices for cross-device validations. However, it comes at a cost of machine resources, cloud concurrency or minutes usage on license limits (if applicable), and added complexity in your test framework having to manage different sets of parallelization. Some frameworks have this architecture “baked” into them but for many others, it’s up to the developer/tester (or whomever) to implement this logic themselves.

With that said, all of the approaches above come with but are not limited to some of the challenges below:

Susceptible to continual dependency and versioning conflicts.
Any single device (or devices) either locally or on a cloud test service can have issues at any given time resulting in test flakiness.
Network or latency timeout issues can occur.
Service crashes or issues occurring on any running parallel process or thread.
Added test code complexity for parallelization.
Added test code conditionals to accommodate different device form factors/resolutions and application layouts.

Applitools Native Mobile Library

Now that we’ve talked about the typical traditional approaches to mobile automation testing, let’s talk about the next generation of mobile cross-device testing using the Applitiools Native Mobile Library! Our Native Mobile Library (NML) uses a new technological approach to asynchronously validate your native mobile application in parallel and easily across many different devices in a single execution.

What this means is the parallelization of devices is handled on the Applitools NML. Since it’s asynchronous you are not waiting on the device to connect or the test results, which frees your tests to execute as fast as possible!

Some key benefits:

Execute your mobile tests on just one device (emulator, simulator, or a real device)!
- Supports Appium as well as XCUI, and Espresso frameworks for iOS and Android native applications.
Simplistic test authoring!
- Create tests with only one device and form factor/app layout in mind! You won’t need to add additional logic in your code for different devices or resolutions.
Less code maintenance.
- No extra coded conditionals for specific devices or supporting multiple execution environments (Local and Cloud). Tests only need to work on any single device and the environment you execute in.
- Visually Perfect with Applitools Visual AI to perform full-page mobile UI validations. No more coded UI assertion logic!
Fast and easy cross-device scaling without complicated multi-parallelization logic and achieve 10x faster executions compared to traditional approaches.
Fewer points of failure.
- No longer is there a need to execute the same test redundantly across different devices increasing test flakiness and execution time. Instead, run once on any real device or simulator/emulator and get validations across many devices.
Fast CI/CD feedback loop.
- Release faster with an assurance of cross-device coverage and your application UI is visually perfect.
No Appium version pinning.
- Use any version of Appium that works for you and your tests.
Secure!
- No need to upload an application to 3rd party cloud test vendors and risk exposing sensitive proprietary data and/or information.
- Tests can run locally on your network without opening firewall access or proxies.

Now let’s look at some example execution architectures using the Applitools Native Mobile Grid!

Asynchronous Parallel Device Execution

This next diagram is similar to the traditional Parallel Device Execution we talked about above. However, the parallelization is offloaded to the NMG to handle and no additional logic for parallel threads or processes is required. This approach is still not the most efficient since each test is running sequentially but it’s lightyears faster than the traditional approach.

Asynchronous Parallel Test Execution

Out of all the approaches for mobile test automation we’ve discussed thus far, this next example is by far the most superior, efficient, and fastest approach possible to scaling your native mobile test coverage! The added benefit of this approach is your entire test suite execution time will now only take as long as it takes to run your slowest test in your suite!

This example is similar to the traditional Parallel Test & Parallel Device Execution above but the device parallelization is now offloaded to the NMG and handled asynchronously.

Code Example using Mobile Native Library for Mobile Test Automation

The below code is an example using Appium Java, testing an iOS native application. For those of you familiar with the Applitools Ultrafast Test Cloud for desktop and mobile web applications, this should look very similar. All we need to do in the test code is define the devices we want to validate for our cross-device coverage needs. In this case, we’ve specified 15 iOS devices we’ll validate in the same execution time as it takes to run the tests on just one device! Additional settings such as setting the OS version and orientation per device can be specified (not shown).

public class iOSNativeUFGTest {
   private WebDriverWait wait;
   private IOSDriver driver;
   private VisualGridRunner runner;
   private Eyes eyes;

   private static IOSDriver startApp() throws Exception {
       DesiredCapabilities capabilities = new DesiredCapabilities();
       capabilities.setCapability("platformName", "iOS");
       capabilities.setCapability("automationName", "XCUITest");
       capabilities.setCapability("deviceName", "iPhone 12 Pro Max"); //Launch a single local simulator or real device
       capabilities.setCapability("platformVersion", "15.4");
       capabilities.setCapability("app", "/Users/justin/Desktop/MyApp.app");
       return new IOSDriver<>(new URL("http://localhost:4723/wd/hub"), capabilities);
   }

   @Before
   public void setup() throws Exception {
       runner = new VisualGridRunner(new RunnerOptions().testConcurrency(15));
       eyes = new Eyes(runner);
       eyes.setApiKey(System.getenv("APPLITOOLS_API_KEY"));;

       Configuration conf = eyes.getConfiguration(); //Configure the 15 devices we want to validate asynchronously
       conf.addMobileDevice(new IosDeviceInfo(IosDeviceName.iPhone_11));
       conf.addMobileDevice(new IosDeviceInfo(IosDeviceName.iPhone_11_Pro));
       conf.addMobileDevice(new IosDeviceInfo(IosDeviceName.iPhone_11_Pro_Max));
       conf.addMobileDevice(new IosDeviceInfo(IosDeviceName.iPhone_12));
       conf.addMobileDevice(new IosDeviceInfo(IosDeviceName.iPhone_12_Pro));
       conf.addMobileDevice(new IosDeviceInfo(IosDeviceName.iPhone_12_Pro_Max));
       conf.addMobileDevice(new IosDeviceInfo(IosDeviceName.iPhone_12_mini));
       conf.addMobileDevice(new IosDeviceInfo(IosDeviceName.iPhone_13_Pro));
       conf.addMobileDevice(new IosDeviceInfo(IosDeviceName.iPhone_13_Pro_Max));
       conf.addMobileDevice(new IosDeviceInfo(IosDeviceName.iPhone_XS));
       conf.addMobileDevice(new IosDeviceInfo(IosDeviceName.iPhone_X));
       conf.addMobileDevice(new IosDeviceInfo(IosDeviceName.iPhone_XR));
       conf.addMobileDevice(new IosDeviceInfo(IosDeviceName.iPhone_11));
       conf.addMobileDevice(new IosDeviceInfo(IosDeviceName.iPhone_8));
       conf.addMobileDevice(new IosDeviceInfo(IosDeviceName.iPhone_7));

       eyes.setConfiguration(conf);

       driver = startApp();
   }

   @Test
   public void iOSNativeTest() throws Exception {
       eyes.open(driver, "My Native App", "Login View"); //Start a visual test
       eyes.check("Login", Target.window().fully(true)); //Capture the mobile UI view
       eyes.closeAsync(); //End the visual test
   }

   @After
   public void tearDownTest(){
       /The /results object contains validation results on all 15 devices which can then be asserted for visual and accessibility checks
       TestResultsSummary results = runner.getAllTestResults(false); 
       eyes.abortAsync();
       driver.quit();
   }
}

Conclusion

Now for the first time ever, native mobile developers can perform continuous testing, running their entire test suites on every pull request or code push across many different devices to get immediate quality feedback at once. Just like web application developers have for many years now!

Hopefully, this article illustrated the benefits and superiority of the Applitools Native Mobile Library to other traditional approaches for mobile automation testing. As you now can see how simple and efficient it is to expand your cross-device coverage with the Native Mobile Library there are no more excuses not to. So what’s stopping you?!

How do you get started with the Native Mobile Library you may ask? To start using the Native Mobile Library, simply sign up at the link below to request access. You can read more about the Applitools Native Mobile Grid on our website.

Happy testing!

How to Scale Mobile Automation Testing Effectively