Automating the test execution

When separating manual and automation testing and choosing the latter, we conclude that various automation approaches and frameworks can be used in mobile application testing. The process will be determined partially by the application type.

Two standard test automation approaches are:

  • User-agent-based testing; uses the user-agent identifier sent by the browser to imitate a particular browser on a device. You can use this approach for mobile web applications. 
  • Device-based testing; involves running the application under test directly on the device. You can use this approach for all types of mobile applications.

The application type can also determine the test automation framework suitable for that application. Application testers can perform tests on mobile web applications using the usual web automation tools, while native apps might need specific tools. 

To automate software testing, some approaches used for conventional applications also often apply to mobile applications. These include capture/playback, data-driven, keyword-driven, and behavior-driven testing.

Methods to automate software testing

To automate software testing, the application testers must understand the automation script recording or creation mechanism and how to access and interact with the application’s graphical objects. Several methods for identifying a graphical object are used for mobile test automation. These include image recognition, OCR/text recognition, and object recognition.

Usually, manual scripting is required, at least to improve recorded scripts for readability and maintainability. Full GUI-based testing without the need for scripting.

It tends to be faster than Image/OCR comparison, especially when using native tools provided by the system manufacturer. However, it is slowed down by the need to compare the screen pixel by pixel with a baseline image.

Maintenance depends on the quality of the test scripts. Application testers mainly used it in providing changed baseline images.

Automation Tools Evaluation

The test automation teams must choose an appropriate set of tools to create test automation solutions. That’s why you should consider the key differences between the available tools and their suitability for the project requirements.

The evaluation parameters for test automation tools can be broken into two categories such as organizational fit and technical fit

  • • Assessment of the maturity of the own organization, its strengths and weaknesses
  • • Identification of opportunities for an improved test process supported by tools
  • • Understanding of the technologies used by the test object(s), in order to select a tool that is compatible with that technology
  • • Understanding the build and continuous integration tools already in use within the organization, in order to ensure tool compatibility and integration
  • • Evaluation of the tool against clear requirements and objective criteria
  • • Consideration of whether or not the tool is available for a free trial period
  • • Evaluation of the vendor or support for noncommercial tools
  • • Identification of internal requirements for coaching and mentoring in the use of the tool
  • • Evaluation of training needs, considering the testing skills of those who will be working directly with the tool(s)
  • • Consideration of pros and cons of various licensing models
  • • Estimation of a cost-benefit ratio based on a concrete business case
  • • Test automation requirements and complexities include using the app’s new features like FaceID, fingerprint, and chatbots.
  • • Test environment requirements include varying network conditions, importing or creating test data, and server-side virtualization.
  • • Test reporting and feedback loop capabilities.
  • • The ability of the framework to manage and drive execution on a large scale, either locally or in a test lab in the cloud.
  • • Integration of the test framework with other tools used in the organization.
  • • Support and documentation are available for current and future upgrades.

Approaches for setting up an Automation Test Lab

When performing mobile application testing, developers and testers can choose the device test lab they target their test automation against. It can be an on-premise device test lab or a remote device test lab. Various combinations of these approaches can be applied.

On-premise device test labs are generally difficult and time-consuming to maintain. Having devices locally in parallel with emulators and simulators would best serve the mobile app’s early development and testing phases.

When reaching a more advanced stage of app development, teams must perform full regression, functional, and non-functional tests. These tests are best executed on a full device lab. This is where a remote device test lab is managed, continuously updated, and maintained in the cloud.

Such remote device test labs complement on-premise ones and ensure that sufficient device and operating system combinations are available and updated. By using commonly available remote device test labs, teams can access a larger set of supported capabilities, including richer test reports and advanced test automation capabilities.

Lastly, when executing at scale through a test automation framework or a continuous integration job (CI), the stability of the overall test lab is key for test efficiency and reliability. Typically, such labs are designed to ensure that devices and operating systems are always available and stable.

Remote device test labs are not always necessary in the later development stages of the app. Well-designed and maintained on-premise labs can be as good as any remote lab.