Tutorial #1: Introduction To Mobile Application Testing

There was a time when computers were a rage. They changed how we humans thought and learned. However, now mobile solutions have taken over the market. People don’t want to switch ON their laptops/PC for everything, rather they want their handheld devices to perform everything quickly.

Hence the mobile solutions which we deliver to our clients should be tested very well. This tutorial is intended for those people who are already in mobile testing or those who have switched to it in recent times. As we already have many tutorials on definitions of mobile testing-related terminologies, we will be directly dealing with the scope of this tutorial.

This tutorial will be both an introduction and your guide to Mobile Testing. So, let’s go ahead!

Types of Mobile Testing

There are broadly 2 types of testing that take place on mobile devices:

#1. Hardware testing:

The device includes internal processors, internal hardware, screen sizes, resolution, space and memory, camera, radio, Bluetooth, WIFI, etc. This is sometimes referred to as, simple “Mobile Testing”.

#2. Software or Application testing:

Applications that work on mobile devices and their functionality. It is called “Mobile Application Testing” to differentiate it from the earlier method. Even in mobile applications, there are a few basic differences that are important to understanding:

#1) Native apps: A native application is created for use on a platform like mobile and tablets.
#2) Mobile web apps are server-side apps to access website/s on mobile using different browsers like Chrome, Firefox by connecting to a mobile network or wireless network like WIFI.
#3) Hybrid apps are combinations of native apps and web apps. They run on devices or offline and are written using web technologies like HTML5 and CSS.

There are a few basic differences that set these apart:

• Native apps have a single-platform affinity while mobile web apps have a cross-platform affinity.
• Native apps are written in platforms like SDKs while Mobile web apps are written with web technologies like HTML, CSS, asp.net, Java, and PHP.
• For a native app, installation is required but for mobile web apps, no installation is required.
• Native apps can be updated from the Play Store or App Store while mobile web apps are centralized updates.
• Many native apps don’t require an Internet connection but for mobile web apps, it’s a must.
• Native apps work faster when compared to mobile web apps.
• Native apps are installed from app stores like Google Play Store or App Store where mobile web are websites and are only accessible through the Internet.

The rest of the article is about Mobile App Testing.

The significance of Mobile Application Testing

Testing applications on mobile devices is more challenging than testing web apps on the desktop due to

• Different range of mobile devices with different screen sizes and hardware configurations like a hard keypad, virtual keypad (touch screen) and trackball, etc.
• Wide variety of mobile devices like HTC, Samsung, Apple, and Nokia.
• Different mobile operating systems like Android, Symbian, Windows, Blackberry, and IOS.
• Different versions of operation systems like iOS 5.x, iOS 6.x, BB5.x, BB6.x, etc.
• Different mobile network operators like GSM and CDMA.
•  Frequent updates -(like Android- 4.2, 4.3, 4.4, iOS-5.x, 6.x) – with each update a new testing cycle is recommended to make sure no application functionality is impacted.

As with any application, Mobile application testing is also very important. The clientele is usually in the millions for a certain product and nobody wants a product with bugs. It may result in monetary losses, legal issues, and irreparable brand image damage.

Basic Differences Between Mobile and Desktop Application Testing:

A few obvious aspects that set mobile app testing apart from desktop testing

• On the desktop, the application is tested on a central processing unit. On a mobile device, the application is tested on handsets like Samsung, Nokia, Apple, and HTC.
• Mobile device screen size is smaller than a desktop.
• Mobile devices have less memory than a desktop.
• Mobiles use network connections like 2G, 3G, 4G, or WIFI whereas desktops use broadband or dial-up connections.
• The automation tool used for desktop application testing might not work on mobile applications.

Types of Mobile App Testing:

To address all the above technical aspects, the following types of testing are performed on Mobile applications.

• Usability testing: To make sure that the mobile app is easy to use and provides a satisfactory user experience to the customers
• Compatibility testing: Testing of the application on different mobile devices, browsers, screen sizes, and OS versions according to the requirements.
• Interface testing: Testing of menu options, buttons, bookmarks, history, settings, and navigation flow of the application.
• Services testing: Testing the services of the application online and offline.
• Low-level resource testing: Testing of memory usage, auto-deletion of temporary files, and local database growing issues known as low-level resource testing.
• Performance testing: Testing the performance of the application by changing the connection from 2G, 3G to WIFI, sharing the documents, battery consumption, etc.
• Operational testing: Testing of backups and recovery plans if a battery goes down or data is lost while upgrading the application from a store.
• Installation test: Validation of the application by installing /uninstalling it on the device.
• Security Testing: Testing an application to validate if the information system protects data or not.

Mobile Application Testing Strategy

The Test strategy should ensure that all quality and performance guidelines are met. A few pointers in this area:

#1) Selection of devices: Analyze the market and choose the devices that are widely used. (This decision mostly relies on the clients. The client or the app builders consider the popularity factor of certain devices as well as the marketing needs for the application to decide what handsets to use for testing.)

#2) Emulators: The use of these is extremely useful in theinitial stages of development, as they allow quick and efficient checking of the app. The emulator is a system that runs software from one environment to another without changing the software itself. It duplicates the features and works on the real system.

Types of Mobile Emulators

• Device Emulator- provided by device manufacturers
• Browser Emulator- simulates mobile browser environments.
• Operating Systems Emulator – Apple provides emulators for iPhones, Microsoft for Windows phones, and Google Android phones

---

Recommended Tool

Kobiton

Kobiton is an affordable and highly flexible cloud-based mobile experience platform that accelerates the testing and delivery of native, web, and hybrid apps on both Android and iOS using real devices. Their new scriptless test automation helps teams with no coding expertise to generate open standard Appium scripts with ease.

---

List of a few free and easy-to-use mobile device emulators

a) Mobile Phone Emulator: Used to test handsets like iPhone, Blackberry, HTC, Samsung, etc.

b) MobiReady: We can not only test the web app, but also check the code.

c) Responsivepx: It checks the responses of the web pages, appearances, and functionality of the websites.

d) Screenfly: It is a customizable tool used to test websites under different categories.

#3) After a satisfactory level of development is complete for the mobile app, you can move to testing on physical devices for more real-life scenario based testing.

#4) Consider cloud computing-based testing: Cloud computing is basically running devices on multiple systems or networks via the Internet where applications can be tested, updated, and managed. For testing purposes, it creates a web-based mobile environment on a simulator to access the mobile app.

Pros:

• Backup and recovery – Cloud computing automatically takes back up of your data from a remote location, making recovery and restoring of data easier. Also, the storage capacity is unlimited.
• Clouds can be accessed from different devices and anywhere.
• Cloud computing is cost-efficient, easy to use, maintain and update.
• Fast and quick deployment.
• Web-based interface.
• Can run the same script on several devices in parallel.

Cons

• Less control: Since the application runs on a remote or third-party environment, the user has limited control and access to the functions.
• Internet connectivity issues: the setup is on the Internet. Network issues affect the availability and functioning
• Security and privacy Issues: Cloud computing is Internet computing and nothing on the Internet is completely secure, so the chances of data hacking are greater.

#5) Automation vs. Manual testing

• If the application contains new functionality, test it manually.
• If the application requires testing once or twice, do it manually.
• Automate the scripts for regression test cases. If regression tests are repeated, automated testing is perfect for that.
• Automate the scripts for complex scenarios which are time-consuming if executed manually.

There are two kinds of automation tools available to test mobile apps:

Object-based mobile testing tools – automation by mapping elements on the device screen into objects. This approach is independent of screen size and is mainly used for Android devices.

• Examples: Ranorex, Jamo Solutions

Image-based mobile testing tools – create automation scripts based on screen coordinates of elements.

• Examples: Sikuli, Egg Plant, RoutineBot

#6) Network configuration is also a necessary part of mobile testing. It’s important to validate the application on different networks like 2G, 3G, 4G, or WIFI.

Test Cases for Testing a Mobile App

In addition to functionality-based test cases, Mobile application testing requires special test cases which should cover the following scenarios.

• Battery usage: It’s important to keep track of battery consumption while running applications on mobile devices.
• The speed of the application: the response time on different devices, with different memory parameters, with different network types, etc.
• Data requirements: For installation as well as to verify if the user with the limited data plan will be able to download it.
• Memory requirements: again, to download, install and run
• Functionality of the application: make sure the application is not crashing due to network failure or anything else.

Download Some Sample Test Cases for Testing Mobile Applications:

Typical Activities and Proceedings in Testing Mobile Applications

The scope of the testing depends on the number of requirements to be checked or the extent of changes made to the app. If there are a few changes, a round of sanity testing will do. In the case of major and/or complex changes, a full regression is recommended.

An example application testing project: ILL (International Learn Lab) is an application designed to help admins and publishers create websites in collaboration. Using a web browser, instructors choose from a set of features to create a class that meets their requirements.

Mobile Testing process:

#1) Identify the types of testing: As an ILL application is applicable for browsers, so it’s mandatory to test this application on all supported browsers using different mobile devices. We need to do usability, functionality, and compatibility testing on different browsers with the combination of manual and automation test cases.

#2) Manual and Automated testing: The methodology followed for this project is Agile with an iteration of two weeks. Every two weeks dev. the team releases a new build for the testing team and the testing team will run their test cases in the QA environment. The automation team creates scripts for a set of basic functionality and runs scripts that help determine if the new build is stable enough to test. Manual testing team will test the new functionality.

JIRA is used for writing acceptance criteria; maintaining test cases and logging /re-verification of defects. Once the iteration is over, an iteration planning meeting will be held where the dev. The team, product owner, business analyst, and QA team discuss what went well and what needs to improve.

#3) Beta Testing: Once the regression testing is completed by the QA team, the build moves to UAT. User Acceptance Testing is done by the client. They re-verify all the bugs to make sure every bug is fixed and the application is working as expected on every approved browser.

#4) Performance test: The performance testing team tests the performance of the web app using JMeter scripts and with different loads on the application.

#5) Browser testing: The web app gets tested across multiple browsers – both using different simulation tools as well as physically using real mobile devices.

#6) Launch plan: After every 4th week, the testing moves into staging, where a final round of end-to-end testing on these devices is performed to make sure the product is ready for production. Then it goes Live!

*****************************************

How to Test Mobile Applications on Both Android and iOS Platforms

It is very important for the testers who test their apps on both iOS and Android platforms to know the difference between them. iOS and Android have a lot of differences w.r.t to the look and feel, app views, encoding standards, performance, etc.

Basic Differences Between Android and iOS Testing

You may have gone through all the tutorials, I have put in some major differences here, which in turn will help you as part of your testing:

#1)  As we have a lot of Android devices available in the market and all of them come with different screen resolutions and sizes, hence this is one of the major differences.

For Example, the Samsung S2 size is too small when compared with Nexus 6. There is a high possibility that your app layout and design will get distorted on one of the devices. Probability is low in iOS as there are only countable devices available in the market and out of those many phones have similar resolutions.

For example, before the iPhone 6 and above came into existence all the older versions had a similar size only.

#2) Example to assert the above point is that in Android the developers must use 1x,2x,3x,4x and 5x images to support image resolutions for all devices whereas iOS uses just 1x,2x, and 3x. However, it becomes the tester’s responsibility to ensure that the images and the other UI elements are displayed correctly on all devices.

Please refer to the diagram below to understand the concept of image resolution:

#3) As we have a market flooded with Android devices, the code must be written in such a way that the performance remains steady. So it is quite probable that your app may behave slowly on lower-end devices.

#4) Another issue with Android is that software upgrades are not available for all devices at go. Device manufacturers decide when to upgrade their devices. It becomes a very difficult task to test everything both with the new OS and the old OS.

Also, it becomes a cumbersome task for the developers to modify their code to support both versions.

For example, when Android 6.0 came, there was a major change as this OS started supporting app-level permissions. To clarify further, the user can also change permissions (location, contacts) at the app level also.

The testing team owes the responsibility to make sure that it is showing the permissions screen on the app launched on Android 6.0 and above and does not show the permissions screen on the lower versions.

#5) From a testing perspective, pre-production build (i.e., beta version) testing is different on both platforms. In Android, if a user is added to the beta user list then he can see the updated beta build on the Play Store only if he is signed in to the play store with the same email ID which is added as a beta user.

Key Factors in Mobile Testing

I have been working in Mobile Testing for the last 2 years on both iOS and Android platforms. All the key points mentioned below in this tutorial are from my personal experience and some got derived from the issues encountered in the project.

Define your own scope of Testing

Everyone has their own style of testing. Some testers just focus on what they see in front of them while others are passionate about everything that works behind the scenes of any mobile application.

If you are an iOS/Android Tester, I would suggest you get yourself familiar with some common limitations/ basic functionalities of Android or iOS as it always adds value to our style of testing. I know things are difficult to understand without citing examples.

Given below are a few examples for your benefit:

• We cannot change the permissions like camera, storage, etc. on the app level in Android devices that are below the 6.0.1 version.
• For iOS below 10.0 version, the call kit is not there. Just to brief you in simple words, a call kit is used by a calling app and displays a full-screen view when a user is getting a call from a calling app such as WhatsApp, Skype, etc. Whereas for iOS versions below 10.0, we see those calls as a notification banner.
• Many of you may have come across problems in Paytm where your app is not redirecting you to the payment page of the bank in case you want to add money to your wallet. We think the above is an issue with our bank or Paytm server but it is just that our Android System WebView is not updated. A little knowledge about programming is always helpful for you to share with your team.
• In simple words, whenever an app is opening any web page in it, then Android System WebView should be updated.

Do Not Limit Your Testing

Testing should not just be limited to exploring the mobile app and logging bugs. We, as QA, should be aware of all the requests that we hit our server and the response that we get out of it.

Configure Putty to view logs or verify sumo logic for logs depending on what is being used in your project. It not only helps you in knowing the End-to-End flow of the application but also makes you a better tester as you get more ideas and scenarios now.

Reason: Any statement should have a valid reason behind it. The reason behind analyzing the logs is that many exceptions are observed in the logs but they don’t show any impact on the UI hence we don’t notice it.

So, should we ignore it?

No, we shouldn’t. It doesn’t have any impact on the UI but it may be a futuristic concern. We could potentially see our app crashing if these kinds of exceptions keep coming up. As we have mentioned about App Crash in the last sentence, this leads the QA to have access to crashlytics of the project.

Crashlytics is a tool where crashes are logged along with the time and device model.

Now the question is if the tester has seen the app crashing then why does he need to bother with crashlytics?

The answer to this is quite interesting. There are some crashes that may not be visible on the UI but they are logged on crashlytics. It could be an out of memory crash or some fatal exceptions which may impact the performance later.

Cross-Platform Testing

Cross-Platform Interaction Testing is very important.

To give a simple example, let’s say you are working on a chat application like WhatsApp which supports sending images and videos and the application is built on both iOS and Android platforms (Development may or may not be going in sync)

Ensure to test the communication of Android and iOS, the reason being that iOS uses “Objective C” whereas Android programming is Java-based and due to both of them being built on different platforms sometimes extra fixes need to be made on the app side to recognize strings coming from different language platforms.

Keep an eye on the size of your Mobile App

Another important piece of advice for mobile testers – Please keep checking the size of your app after each release.

We should ensure that the size of the app doesn’t reach a point where even we as end-user won’t wish to download this app due to its large size.

Testing App Upgrade Scenarios

For mobile testers, app upgrade testing is very important. Ensure your app doesn’t crash on the upgrade as the dev team may have mismatched the version number.

Data retention is also equally important as whatever preferences the user has saved in the previous version should be retained when he upgrades the app.

For example, a user might have saved his bank card details in apps like PayTm, etc.

Device OS may not Support the App

Sounds Interesting?

Yes, many devices may not support your app. Many of you must be knowing that vendors write their own wrappers on top of the US and it could be possible that any SQL query of your app is not compatible with the device hence it throws an exception and it may result in not even launching the app on that phone.

The point over here is – To try to use your app on your own devices except for the ones you use in the office. It is quite possible that you are seeing some issues with your app.

App Permission Testing

Next on the list is Permission Testing of mobile apps. Almost every second app asks its users for access to their phone’s contact, camera, Gallery, Location, etc. I have seen a few testers who make a mistake by not testing the proper combinations of these permissions.

I can recall a real-time example when we were testing a chat app that had all the features of sharing images and Audio files. Permission for Storage was set to NO.

Now, when a user clicks on the Camera option it never opens until the permission for storage is set to YES. This scenario was ignored as Android Marshmallow has this functionality that if storage permission is set to NO, the camera cannot be used for that app.

The scope extends further than what we have discussed in the above paragraph. We need to make sure that the app is not asking for any permissions that are not being used.

Any end user familiar with the software industry may not download the app in which too many permissions are asked. If you have removed any feature from your app, then make sure to remove the permission screen for the same.

Compare similar and popular Apps in the Market

Moral of the story – If ever you are in a doubt, then please don’t bring it to a conclusion. Comparing with other similar apps on the same platform can strengthen your argument that the functionality under testing will work or not.

Get an Overview of Apple’s Build Rejection Criterion

Lastly, many of you might have got your builds rejected by Apple. Let me make you aware of Apple’s rejection policies.

As a tester, it becomes difficult for us to cater to the technical aspects but still, there is some rejection criterion that the testers can take care of.

Always be on the Front Foot

Being a tester, don’t let things pass over to your court from the Dev Team/ Managers. If you are passionate about testing then “Always be on Front Foot”. Try to engage yourself in activities that take place well before the code comes to your bucket to test.

Most importantly, keep looking at JIRA, QC, MTM, or whatever is used in your project for all the latest updates on tickets from clients and the Business Analyst. Also, be ready to share your views if you require modifications. This applies to all testers who are working on various domains and platforms.

Until and unless we feel the product is our own, we never ought to give suggestions for new improvements or changes to the existing functionality.

Keep your app in the background for 12-24 hours

I know it sounds weird but there is a lot happening behind the scenes which we don’t understand.

I am sharing this because I saw the app crashing after launching it, say around 14 hours from the background state. The reason could be anything depending on how the developers have coded it.

Let me share a real-time example:

In my case token expiration was the cause behind it. One of the chat apps if launched after 12-14 hours will be stuck on the connecting banner and will not get connected until killed and relaunched. These kinds of things are very difficult to catch and in a way, it makes mobile testing more challenging and creative.

Performance Testing of your App

In the mobile world, the performance of your app impacts the extent to which your application is getting recognized worldwide. As a testing team, it becomes very important to check your app response and more importantly how it works when a large number of users are using it altogether.

Example:

Let’s talk about PayTm.

You must have all clicked on the ADD MONEY option in the PayTm app, which then displays the balance you have in your wallet. If we consider what is going on behind the scenes, then it is a request which is going on to the server with the PayTm UserID and the server sends back the response with the balance in your account.

The above case is only available when one user has hit the server. We need to make sure that even when 1000 users hit the server, they should get back the response well on time because end-user usability is our prime goal.

Conclusion

I would conclude this tutorial by re-iterating that mobile testing seems to be very easy in the beginning but as you get into the details you will understand that it is not easy to ensure that whatever has been developed will run smoothly on thousands of devices all over the world.

You will mostly see apps that are supported on the latest and last few versions of OS only. However, the testers should ensure that they don’t miss out on any scenarios. There are many other points that need to be taken into consideration but those are already iterated in the other tutorials.

Scenarios like battery consumption, interrupt testing, testing on different networks (3G, Wi-Fi), testing while switching networks, monkey testing of mobile apps, etc. are all useful when it comes to mobile testing.

The attitude of testers matters a lot when it comes to the real testing environment. Unless you love what you do, you won’t bother doing things that are mentioned in the tutorial.

I have been in this field for about 6 years now and I know that the tasks tend to get repetitive at times.  But, there are many other things that we can do on our own to make those monotonous tasks interesting.

Designing the right test strategy, choosing the right mobile simulators, devices, and mobile testing tools can ensure that we have 100% test coverage and help us include security, usability, performance, functionality, and compatibility-based tests into our test suites.

Well, this has been our effort to fulfil multiple requests from our readers in a mobile application testing guide.

About Author