Tag Archives: mobile

Mobile Testing – How Much is Enough?

We all know that mobile apps can never be fully tested.  There are too many devices, too many OS versions, and too many different types of apps.  Testers continue to struggle with developing test plans and test cases the same way they did with traditional applications.  How do testers determine the correct scope of testing for an app to minimize quality and security risks for users and the organization?

First, look at organizational expectations.  Does your organization expect its software to work almost perfectly?  And does it fund and support development and testing to make that a realistic expectation?

If the answer to the first question is yes, but the second one is no, then the best you can do is set realistic expectations that may not be in line with what the organization wants.  If quality isn’t a high priority, then testing can be focused on high-priority areas only.

Second, is the app business-critical?  Does the organization depend on it to make money, service customers, or be more agile?  Or is it a marketing or public relations tool?  Does it provide valuable information to users, or is it simply nice to have?  If it is business-critical, then flaws could hurt the bottom line and quality become a higher priority.  And that’s true in all parts of the app, not just it’s operational aspects.  If users find poor quality in any part, they are unlikely to trust it to do business.

Third, consider the implications of a flaw to both use users and the organization.  A game or other entertainment app often has a high threshold of failures before most users think it’s not worth the effort.  Likewise a free app with limited extrinsic value.

But if the app performs an important function, or one that is counted on my users, a major flaw can have serious or even disastrous effects.  One example is the infamous iPhone time change bug in 2010, which failed to move from Daylight Savings Time on the appointed day.  Thousands of people were reportedly late for work or appointments as their alarm failed to go off on time.  It was an iOS flaw fixed by Apple a few weeks later.

Fourth, does the app use external services?  Many apps make use of other services within the enterprise, or commercial services for information such as weather or sports scores.  While testers don’t have to test the services, testers should read and understand what the service level agreements (SLAs) say about performance and capacity, and periodically test to make sure those are being fulfilled.

 

Now About Security

It goes without saying that security flaws are also quality flaws.  But many security flaws may have significant consequences to both user and organization.  If there is no data, or trivial data, to protect, security testing may not be important.  If it includes names, email addresses, or financial data, or any other identification information, that data needs to be protected at all cost.

In particular, testers have to know what data is being collected, and where data is stored.  On mobile devices, that can sometimes be a challenge, because it can be in internal storage or on a SIM, and it may not be readily apparent that data is being stored, and in any case often can’t easily be accessed.

In the era of hundreds of different devices and OS versions, as well as BYOD, it’s unrealistic to limit the kinds of devices that an app can be used on, even for internal users.  Testers have no control over the device or OS for testing or deployment purposes, and testers simply can’t test all combinations.

But teams likely have control over how and where the device stores data locally.  And they have control over how data is transferred to and from the device, and how it’s stored on the back end.  That’s what testers have to focus on.  Is any personal or identification data encrypted on the device, and is it encrypted during transmission?

Testing on mobile devices requires a combination of techniques, from traditional test cases to risk-based testing to device farm testing and if appropriate, crowdsourcing.  The test plan should be designed to use these techniques to test different aspects of the application.  Test cases, for example, can test traditional quality measures as well as security.  Device farms can be used for in-depth testing of popular devices.

Overall, test results should provide a clear picture of the quality of important aspects of the app given its purpose, and an overview of quality in general.

The Android Diaries, Part 3

1/1/2013   It’s a new year and a brave new world among the androids.   As I start to play, otherwise known as exploratory test my new phone, I discover that it’s like dancing.  I must learn a whole new series of movements.  When I turn the phone on, it says “Swipe to Unlock”.   So what’s a “swipe”?  So I go on the web and locate the definition of swipe.  Here’s Merriam Webester’s definition:   Definition of SWIPE. 1: a strong sweeping blow <a swipe of a paw> 2: a sharp often critical remark <took a parting swipe at management>.   Ok, so now what do I do…attempt to hit it with my “paw” or make a nasty remark at it?    After a few attempts, I find that an android “swipe” is rather like a doing a waltz with one’s finger.  I would define it as a glide.   However it is defined, so be it.  Now that I’ve mastered the “swipe”,  it’s time to learn to tap.   Hmmm, I wonder if my old tap shoes are still in the attic?

The Android Diaries, Part 2

12/31/2012   Having thoroughly researched both the iPhone and Galaxy S iii, and learned that the only real difference is size; screen of the Galaxy S ii is larger.  Since my arms won’t grow any longer and I don’t always wear my glasses, I even more confidently return to the store to purchase my Galaxy S iii.  Now the most important choice…color.  Of course, I want a white one that must be brought in from a store 40 minutes away.  When it finally arrives, the geek must transfer most of my contacts from old phone manually since the old phone is so outdated.  Then he shows me how to download music and I’m off to join the android world.

The Android Diaries, Part 1

12/30/2012  Having made  my new year’s resolution to become more technical,  even though it is not yet the new year, I decide to start early with the purchase of my new smart phone.  As I confidently head out to the AT&T store, I’m contemplating Apple vs Android.  At the store, I listen in as the phone geek behind the counter shows an older couple how to use a smart phone and shows them differences between the iPhone 5 and the Galaxy S iii.   After hearing some of the pros and cons of both, the couple decides on an iPhone.  I ask if I can download tunes on the Galaxy and the answer is not from my iPod but there are 6000 free songs available online.  Can I download the books from my Nook?  The answer is yes.  And seeing the line of customers behind me become longer and more impatient, I leave the store and return home to research both smartphones on the internet.