Common Test Types Applicable for Mobile Application
Testers need to focus on installation, update and de-installation of the app using the following approaches:
- Application stores
The installation process may be different depending on the users of the app. The users could install the app from market place stores such as the Google Play Store or Apple’s App Store.
The users of enterprise apps will be required to perform installation tests via a link, or a distribution service such as HockeyApp or App Center.
- Sideloading (copying and installing app)
Some operating systems provide the option of installing the application by copying it to a mobile device and installing it from the file.
- Desktop applications
Desktop applications such as Apple iTunes (for iOS) or Android App Installer are available for installing apps on the smartphone. The tester needs to download the app in this application and use a cable to install it from there to the smartphone. Most of these desktop applications also allow de-installation of the app.
Installation can be performed using the following methods:
- OTA (Over-the-Air) via Wi-Fi or Cellular Data
- Data cable
Some of the test conditions that can be considered include:
- Installation, de-installation and upgrade on internal and external memory (if supported).
- Re-installation of app when the “retain app data” option was chosen during the previous de-installation.
- Re-installation of app when the “retain app data” option was not chosen during the previous de-installation.
- Cancelling or interrupting the installation or de-installation, for example, by shutting down the mobile device during the process or disconnecting from the internet.
- Resuming interrupted installation, de-installation and upgrade after cancelling or interrupting.
- Permissions-related testing. For example, some apps request permission to use the address book. This important test must verify app behavior if the user denies permission. For example, is there a corresponding message sent to the user?
- Update the app and verify that no data is lost.
Some apps require jailbroken (iOS) or rooted (Android) devices which give the user the administrative rights over the device. Most platform providers do not support jailbreaking/rooting as it may have legal consequences. An app not requiring jailbreaking/rooting may not need to be tested for the jailbreaking/rooting devices.
Stress testing is focused on determining the performance efficiency of the application when subjected to conditions beyond normal load. The stress test is this context is targeted only at the mobile device.
Some of the test conditions that can be considered for stress testing include:
- high CPU-usage
- low disk space
- battery stress
- poor bandwidth
- very high number of user interactions (real world network conditions may need to be simulated for this)
Some of these stressful conditions can be created using tools such as Monkey. This is a command line tool that runs over the ADB shell command line or, if possible, manually, e.g., by using big files or other apps with high CPU-usage or memory consumption.
Principal security issues for mobile apps include:
- Access to sensitive data on the device.
- Unencrypted information transfer or unsafe storage.
Some of the test conditions that can be considered for security testing include:
- Testing inputs for code injection and overflow.
- Encryption of transferred data.
- Encryption of locally stored data.
- Deletion of temporary data after use or after an abnormal end.
- Clearing text in password fields.
Top 10 mobile related vulnerabilities from the Open Web Application Security Project (OWASP) should also be explored.
If the user installs the app and it does not appear fast enough (e.g., less than or equal to 3 seconds) it may get de-installed in favor of another alternative app. Time and resource consumption aspects are important success factors for an app and performance testing is carried out to measure these aspects.
Performance efficiency needs to be tested on the device itself in addition to interaction with the backend system and other mobile devices.
Performance testing of the whole system should be performed as defined in the test strategy and is not mobile specific
The performance test of the app itself should contain chronometry for the most important workflows.
Some examples for the workflows of an online-banking app are: “Login”, “Change address” or “Bank transfer with PIN and TAN”. The tester should then compare this chronometry with similar apps.
Besides chronometric measures it is important to consider the perceived performance by the user.
User experience can have a huge impact on how long the user is willing to wait for a certain function to complete.
Usability is very important for mobile apps because data shows that a large number of users de-install their apps within a few minutes of installing because of poor usability or performance.
Due to this it is recommended that user experience (UX) design considers the look and feel of the platform which the app is to be used on. If the UX does not conform with the user’s expectations for their platform of choice, it can have a strong negative impact. Thus, a tester should be aware of the look and feel on the platform used.
Usability tests can be conducted by a tester using various available heuristics and test tours.
Considering personas is also a helpful support for usability testing. If required, a usability lab can also be used for this purpose.
In projects, findings identified during the usability test are mostly just findings and not defects. The tester must have the ability to explain the findings to the team, Product Owner or similar stakeholders.
To achieve satisfactory usability, an app should:
- be self-explanatory and intuitive.
- allow user mistakes.
- be consistent in wording and behavior.
- abide by the design guidelines of the platforms.
- make needed information visible and reachable in each screen size and type.
Many apps need to store data locally using various data storage mechanisms such as flat files or databases. Some of the test conditions to be considered for the database testing of mobile apps include:
- Validation of data storage issues:
- Upload conflicts
- Data security
- Constraints on the data
- CRUD (Create/Read/Update/Delete) functionality
- Data integration testing for data provided by the device (e.g., contacts) or by third-party apps
(e.g., pictures, videos and messages).
- Performance of storing the data on the device.
Globalization and Localization Testing
Internationalization (I18N) /Globalization testing of the application includes testing an app for different locations, formats for dates, numbers and currency, and replacing actual strings with pseudo-strings.
Localization (L10N) testing includes the testing of an app with localized strings, images and workflows for a particular region. For example, Russian and German words could be much longer than those in other languages. Since mobile devices have different screen sizes and resolutions, limited screen sizes may lead to problems with translated strings. These issues should be checked as standard globalization/localization tests.
A very important aspect to be checked is the date format used, such as YEAR – MONTH – DAY or DAY – MONTH – YEAR.
Accessibility testing is performed to determine the ease by which users with disabilities can use a component or system. For mobile apps this can be done using device accessibility settings and testing the app for each setting.
Accessibility guidelines are available from platform vendors and these should be used. For example, both Google and Apple have published accessibility guidelines for their respective platforms. Taking feedback from people who require accessibility is also helpful.
For mobile web an accessibility guide has been published by the W3C, which should be considered.