Test Improvement Processes

As organizations use testing to improve software, test improvement process techniques can be selected and used to improve the software development process and the resulting software deliverables. Test improvement processes can also be applied to the testing processes. Different ways and methods are available to improve the testing of software and of systems containing software. These methods aim at improving the process, and hence the deliverables, by providing guidelines and areas for improvement.

Testing often accounts for a major part of the total project costs. However, limited attention is given to the test process in the various software process improvement models, such as CMMI®.

Test improvement processes such as the Test Maturity Model integration (TMMi®), Systematic Test and Evaluation Process (STEP), Critical Testing Processes (CTP), and TPI Next® were developed to address the lack of attention to testing in most software process improvement models. Properly used, these models can provide a degree of cross-organization metrics that can be used for benchmark comparisons.

The models presented are not intended to be a recommendation for use but are presented here to provide a representative view of how the models work and what they include.

Introduction to Test Improvement Processes

Test improvement processes are relevant to the software development process as well as to the testing process. Learning from one’s mistakes makes it possible to improve the process organizations use to develop and test software. The Deming improvement cycle: Plan, Do, Check, Act, has been used for many decades and is still relevant when testers need to improve the process in use today.

One premise for process improvement is the belief that the quality of a system is highly influenced by the quality of the process used to develop the software. Improved quality in the software industry reduces the need for resources to maintain the software and thus provides more time for creating more and better solutions in the future. Process models provide a place to start improving by measuring the organization’s process capabilities against the model. The models also provide a framework for improving the organization’s processes based on the outcome of an assessment.

A process assessment leads to capability determination, which motivates process improvement. This may invoke a subsequent process assessment to measure the effect of the improvement.

Types of Test Improvement Processes

Assessment models are a common method that ensures a standardized approach to improving test processes using tried and trusted practices.

Process improvement models are categorized into two types:

  1. The process reference model provides a maturity measurement as part of the assessment to evaluate an organization’s capability compared with the model, to evaluate the organization within the framework, and to provide a roadmap for improving the process.
  2. The content reference model provides business-driven evaluations of an organization’s opportunities to improve, including, in some cases, benchmarking against industry averages using objective measurements. This evaluation can be used to create a roadmap for improving the process.

Improving the Testing Process

The IT industry can work with test improvement process models to reach a higher level of maturity and professionalism. Industry-standard models are helping to develop cross-organization metrics and measures that can be used for comparison. Several sets of recommended processes have materialized because of the need for process improvement in the testing industry. These include STEP, TMMi, TPI Next, and CTP. The staged models, such as TMMi and CMMI, provide standards for comparison across different companies and organizations. The continuous models, such as CTP, STEP, and TPI Next, allow an organization to address its highest priority issues with more freedom in the order of implementation. These are each discussed further in this section.

These models allow an organization to determine where it stands in terms of its current test processes. Once an assessment is performed, TMMi and TPI Next suggest a roadmap for improving the test process. Alternatively, STEP and CTP provide the organization with means to determine where its greatest process improvement return on investment will come from and leave it to the organization to select the appropriate roadmap.

The Testing Maturity Model integration (TMMi) comprises five maturity levels and is intended to complement CMMI. Each maturity level contains defined process areas that must be 85% complete by achieving specific and generic goals before the organization can advance to the next level.

The TMMi maturity levels are:

Level 1: Initial

The initial level represents a state with no formally documented or structured testing process. Tests are typically developed ad hoc after coding, and testing is seen as the same as debugging. The aim of testing is understood to be proving that the software works.

Level 2: Managed

The second level is attained when testing processes are clearly separated from debugging. It can be reached by setting testing policies and goals, introducing the steps in a fundamental test process, and implementing basic testing techniques and methods.

Level 3: Defined

The third level is reached when a testing process is integrated into the software development lifecycle and documented in formal standards, procedures, and methods. Reviews take place, and there should be a distinct software testing function that can be controlled and monitored.

Level 4: Measured

Level four is achieved when the testing process can be effectively measured and managed at an organizational level to benefit specific projects.

Level 5: Optimized

The final level represents a state of test process maturity where data from the testing process can be used to help prevent defects. The focus is on optimizing the established process.

The TPI Next model defines 16 key areas, each covering a specific aspect of the test process, such as test strategy, metrics, test tools, and test environment.

Four maturity levels are defined in the model: INITIAL, CONTROLLED, EFFICIENT, and OPTIMIZING. 

Specific checkpoints are defined to assess each key area at each maturity level. Findings are summarized and visualized through a maturity matrix that covers all key areas. The definition of improvement objectives and their implementation can be tailored according to the needs and capacity of the testing organization.

The generic approach makes TPI Next independent of any software process improvement model. It covers the test engineering aspects and supports managerial decision-making.

STEP (Systematic Test and Evaluation Process), like CTP and unlike TMMi and TPI Next, does not require that improvements occur in a specific order.

STEP is primarily a content reference model based on the idea that testing is a lifecycle activity that begins during requirements formulation and continues until the system’s retirement. The STEP methodology stresses “test then code” by using a requirements-based testing strategy to ensure that the early creation of test cases validates the requirements specification before design and coding.

Sometimes, the STEP assessment model is blended with the TPI Next maturity model.

The basic premise of the Critical Testing Processes (CTP) assessment model is that specific testing processes are critical. If carried out well, these critical processes will support successful test teams. Conversely, if these activities are carried out poorly, even talented individual testers and Test Managers are unlikely to be successful. The model identifies twelve critical testing processes. CTP is primarily a content reference model. The CTP model is a context-sensitive approach that allows for tailoring the model, including:

Identification of specific challenges

Recognition of attributes of good processes

Selection of the order and importance of implementation of process improvements

The CTP model is adaptable within the context of all software development lifecycle models.

In addition to participant interviews, the CTP model includes metrics to benchmark organizations against industry averages and best practices.

Once it has been agreed that test processes should be reviewed and improved, the process improvement implementation steps to be adopted for this activity could be as defined in the IDEALSM model:

Before the process improvement activities start, the stakeholders agree on the process improvements’ objectives, goals, scope, and coverage. The choice of the process improvement model is also made at this time. The model may either be selected from publically available options or developed internally. In addition, success criteria should be defined, and a method by which they will be measured throughout the improvement activity should be determined.

The agreed assessment approach is undertaken, and a test assessment report is created, which contains an appraisal of current testing practices and a list of possible process improvements.

The list of possible process improvements is prioritized. The prioritization could be based on return on investment, risks, alignment with organizational strategy, and/or measurable quantitative or qualitative benefits. Having established the priority order, a plan for delivering the improvements is developed.

The test process improvement plan for the delivery of the improvements is implemented. This could include any training or mentoring required, piloting of processes, and ultimately, their full deployment.

Having fully deployed the process improvements, verifying which benefits were received is essential. It is also important to check which success criteria for the process improvement activity have been met.

Depending on the process model used, this stage of the process is where monitoring of the next level of maturity starts, and a decision is made to either start the improvement process again or stop the activity at this point.