Quality is a challenge for all software development, but for mobile applications, it is particularly elusive. The rapid growth in mobile services such as SMS, mobile Web and mobile applications has created a variety of testing complications, not least of which is the fact that there are hundreds of available devices on, which a user might want to run the service (nearly 300 devices in the U.S. alone).
This makes it problematic for developers to maintain mobile applications across the wide variety of form factors, hardware, operating systems and other variables that fragment the mobile market. Further complicating the picture, release cycles are growing ever shorter (sometimes as little as six weeks). As a result, companies are finding it increasingly difficult to ensure quality while keeping costs under control.
The consequence of these dilemmas is that often device testing is incomplete or inconsistent, with testers hoping the program will work on devices similar to the ones where testing followed best practices. The result can be applications that crash and services/sites that do not format or operate as intended on all devices. In a worst-case scenario, code flaws can leave devices and their data vulnerable to attack.
Image via Shutterstock
On the customer side, users are becoming increasingly demanding, expecting near-perfect performance from their mobile services. A 2011 study by Equation Research found that 80 percent of users will try to use a mobile app twice (or less) before giving up if it doesn’t function properly. Thirteen percent won’t retry at all. Other similar studies abound—including one that indicates problematic mobile services affect user opinions not only of the service, but of the sponsoring company, as well.
Here’s the good news. An increasing number of providers are offering secure mobile testing in the cloud in affordable packages, enabling mobile developers to automate testing on a staggering array of devices with reliable results. Cloud-based solutions enable the broad-based, real-time and automated, on-device testing that many developers previously thought was out of reach.
The Mobile Testing Maturity Model
When companies begin developing mobile services, they generally go through a five-stage process in an effort to find that “sweet spot” where they are achieving reliable results without blowing their budgets. Orasi calls this the “Mobile Testing Maturity Model.” Your company may have been through one or more of these steps:
Stage 1 - Handheld: Firms purchase their own devices for testing, with staff manually testing code on the devices directly—a time consuming and cumbersome approach.
Stage 2 - Crowdsourcing: Firms crowdsource their applications, with alpha and beta testers helping to identify bugs and other problems. Due to lack of control over the environment and connection, the process tends to be unreliable.
Stage 3 - Emulators (aka Simulators): Companies turn to automated testing with device emulation, but this approach is too far removed from users and their physical devices.
Stage 4 - Tethered Devices: Tethering lets companies use in-house devices with some level of automation, but purchasing and maintaining testing automation tools and devices is expensive and complicated.
Stage 5 - Cloud Devices: As the need for additional device variations grows, companies move testing to cloud-based devices.
So, What’s So Great About the Cloud?
Cloud-based mobile testing has been around for a while, but new Software as a Service (SaaS) models are revolutionizing the cost effectiveness and reach of these solutions. With this model, third-party providers set up remote “labs” with hundreds of mobile handsets and tablets, tethered to cloud-based servers. Firms needing mobile testing log into a Web-based interface, from which they can run and test their services and software on the actual devices themselves. As with other SaaS delivery models, companies pay for the service, not the infrastructure. Some firms even specialize in offering “blocks” of service (e.g. a day), which makes the offering exceptionally cost effective.
The best of these solutions provides users with interactive handset control (keys, touch, flip/slide, accelerometer, power control), ability to perform all core functions available to device users (make calls, install apps, record video, send e-mail and messages, etc.) and a richly detailed handset screen that authentically replicates the user experience.
Robust automation support enables companies to run scripts on dozens—or even hundreds—of devices at once. And, because these solutions are cloud-based, firms can often engage in remote, real-time handset sharing with multiple participants.
The third-party provider generally handles all device updates, from firmware to OS to form factor, eliminating the need for testing companies to manage the lifecycles of their test devices. Because providers are spreading the cost across all their customers, they can offer this high level of service and keep the product affordable.
An added (and powerful) benefit of SaaS-driven mobile testing is that the cloud-based devices are connected to live mobile networks distributed across different locations, so developers and testers are assured of replicating as closely as possible the experience their customers will have.
Of course, I cannot guarantee all SaaS-based mobile testing providers offer this level of service, but the best ones certainly do. The key for companies is to be diligent in their research, seek out firms that align with leading testing-tool providers, and be prepared to take a leap of faith. The result can be a dramatic improvement in testing quality along with significant cost reductions.
Joseph D. Schulz is a seasoned developer with more than 25 years of professional application development and quality assurance experience. Joe is the Director of Mobile Testing at Orasi Software, one of the largest and most successful HP Software sales and services partners in the quality assurance space. With Orasi, he helps global organizations improve their development and QA processes around mobile device and application testing. For more information, visit www.orasi.com.
Edited by Alisen Downey