Failure to plan for non-functional tests can result in the discovery of serious, sometimes disastrous, quality problems in a system after release. However, many types of non-functional tests are expensive so our Test Manager must select which non-functional tests to perform based on risk and constraints. In addition, there are many different types of non-functional tests, some of which might not be appropriate to a given application.
If our Test Manager may not have sufficient expertise to handle all the planning considerations, the Test Manager needs to delegate some of the test planning responsibilities to our Technical Test Analysts (and in some cases Test Analysts) assigned to the non-functional testing activities.
Our Test Manager will ask the analysts to consider the following general factors:
- Stakeholder requirements
- Required tooling
- Test environment
- Organizational factors
Another important consideration for test managers is how to integrate non-functional tests into the software development lifecycle. A common mistake is to wait until all functional tests are complete prior to starting non-functional tests, which can result in the late discovery of critical non-functional defects. Instead, non-functional tests are prioritized and sequenced according to risk. There are often ways to mitigate non-functional risks during early levels of testing or even during development. For example, usability reviews of user interface prototypes during system design are quite effective at identifying usability defects that would create significant schedule problems if discovered toward the end of system testing.
In iterative lifecycles, the pace of change and of iterations can make it difficult to focus on certain non-functional tests that require construction of sophisticated test frameworks. Test design and implementation activities that take longer than the timescales of a single iteration are organized as separate work activities outside of the iterations.