Our Test Managers are working to optimize testing so that good business value is delivered. Testing excessively does not deliver good business value, because the testing will impose unreasonable delays and cost more than it saves. Testing too little does not deliver good business value because too many defects will be delivered to users. The optimum lies between those two extremes. The Test Manager must help testing stakeholders understand this optimum and the value delivered by testing.
While most organizations consider testing valuable in some sense, few managers, including Test Managers, can quantify, describe, or articulate that value. In addition, many Test Managers, test leads, and testers focus on the tactical details of testing (aspects specific to the task or test level), while ignoring the larger strategic (higher level) issues related to testing that other project participants, especially managers, care about.
Testing delivers value to the organization, project, and/or operation in both quantitative and qualitative ways:
- Quantitative values include finding defects that are prevented or fixed prior to release, finding defects that are known prior to release (not fixed but documented, perhaps with workarounds), reducing risk by running tests, and delivering information on project, process, and product status
- Qualitative values include improved reputation for quality, smoother and more-predictable releases, increased confidence, protection from legal liability, and reducing risk of loss of whole missions or even lives
Test Managers should understand which of these values apply for their organization, project, and/or operation, and be able to communicate about testing in terms of these values.
A well-established method for measuring the quantitative value and efficiency of testing is called cost of quality (or, sometimes, cost of poor quality). Cost of quality involves classifying project and operational costs into four categories related to product defect costs:
- Costs of prevention, e.g., training developers to write more maintainable or secure code
- Costs of detection, e.g., writing test cases, configuring test environments, and reviewing requirements
- Costs of internal failure, e.g., fixing defects detected during testing or reviews, prior to delivery
- Costs of external failure, e.g., support costs associated with defective software delivered to customers
A portion of the testing budget is a cost of detection (i.e., money that would be spent even if testers find no defects, such as money spent developing tests), while the remainder is a cost of internal failure (i.e., the actual cost associated with the found defects). The total costs of detection and internal failure are typically well below the costs of external failure, which makes testing an excellent value. By determining the costs in these four categories, Test Managers can create a convincing business case for testing.