Test estimation, as a management activity, is the creation of an approximate target for costs and completion dates associated with the activities involved in a particular operation or project. The best estimates:
- Represent the collective wisdom of experienced practitioners and have the support of the participants involved
- Provide specific, detailed catalogs of the costs, resources, tasks, and people involved
- Present, for each activity estimated, the most likely cost, effort, and duration
Estimation of software and system engineering has long been known to be fraught with difficulties, both technical and political, though project management best practices for estimation are well established. Test estimation is the application of these best practices to the testing activities associated with a project or operation.
Test estimation should cover all activities involved in the test process. The estimated cost, effort, and, especially, duration of test execution is often of the most interest to management, as test execution is typically on the project’s critical path. However, test execution estimates tend to be difficult to generate and unreliable when the overall quality of the software is low or unknown. In addition, familiarity and experience with the system will likely affect the quality of the estimate. A common practice is to estimate the number of test cases required during test execution, but this only works if one can assume the number of defects in the software to be tested is low. Assumptions made during estimation should always be documented as part of the estimation.
Test estimation should consider all factors that can influence the cost, effort, and duration of the testing activities. These factors include (but are not limited to) the following:
- Required level of quality of the system
- Size of the system to be tested
- Historical data from testing for previous test projects which may be augmented with industry data or benchmark data from other organizations
- Process factors, including the test strategy, development or maintenance lifecycle, and process maturity, and the accuracy of the project estimate
- Material factors, including test automation and tools, test environment(s), test data, development environment(s), project documentation (e.g., requirements, designs, etc.), and reusable test work products
- People factors, including managers and technical leaders, executive and senior management commitment and expectations, skills, experience, and attitudes in the project team, the stability of the project team, project team relationships, test, and debugging environment support, availability of skilled contractors and consultants, and domain knowledge
- Complexity of the process, technology, organization, number of testing stakeholders, composition and location of sub-teams
- Significant ramp up, training, and orientation needs
- Assimilation or development of new tools, technology, processes, techniques, custom hardware, or a large quantity of testware
- Requirements for a high degree of detailed test specification, especially to comply with an unfamiliar standard of documentation
- Complex timing of component arrival, especially for integration testing and test development Fragile test data (e.g., data that is time-sensitive)
The quality of the software delivered for testing is also a major factor that our Test Managers consider in their estimation. For example, if the developers have embraced best practices such as automated unit testing and continuous integration, then as many as 50% of the defects will be removed prior to delivery of the code to the test team (for more on defect removal effectiveness of these practices). Some have reported that Agile methodologies, including test-driven development, result in higher levels of quality being delivered for testing.
Estimation can be done either bottom-up or top-down. Techniques which can be used in test estimation, singly or in combination, include the following:
- Intuition, guesses, and past experience
- Work Breakdown Structures (WBS)
- Team estimation sessions (e.g., Wide Band Delphi)
- Company standards and norms
- Percentages of the overall project effort or staffing levels (e.g., tester developer ratios) Organizational history and metrics, including metrics-derived models that estimate the number of defects, the number of test cycles, the number of test cases, each test’s average effort, and the number of regression cycles involved
- Industry averages and predictive models such as function points, lines of code, estimated developer effort, or other project parameters.
In most cases, the estimate, once prepared, must be delivered to management, along with a justification. Frequently, some negotiation ensues, often resulting in a reworking of the estimate. Ideally, the final test estimate represents the best possible balance of organizational and project goals in the areas of quality, schedule, budget, and features.
It is important to remember that any estimate is based on the information available at the time it was prepared. Early in a project, the information may be quite limited. In addition, information that is available early in a project may change over time. In order to remain accurate, estimates should be updated to reflect new and changed information.