Test Analysis

Test Analysis

Rather than considering test analysis and design together, they will be implemented as parallel, integrated, or iterative activities to facilitate the production of test design work products.

Test analysis is the activity that defines “what” is to be tested in the form of test conditions. Test conditions are identified by analysis of the test basis, test objectives, and product risks. They are viewed as the detailed measures and targets for success (e.g., as part of the exit criteria) and are traceable back to the test basis and defined strategic objectives, including test objectives and other project or stakeholder criteria for success. Test conditions are also traceable forward to test designs and other test work products as those work products are created.

Test analysis for a given level of testing are performed as soon as the basis for testing is established for that level. Formal test techniques and other general analytical techniques (e.g., analytical risk-based strategies and analytical requirements-based strategies) are used to identify test conditions. Test conditions may or may not specify values or variables depending on the level of testing, the information available at the time of carrying out the analysis, and the chosen level of detail (i.e., the degree of granularity of documentation).

There are a number of factors to consider when deciding on the level of detail at which to specify test conditions, including:

  • Level of testing
  • Level of detail and quality of the test basis
  • System/software complexity
  • Project and product risk
  • The relationship between the test basis, what is to be tested and how it is to be tested
  • Software development lifecycle in use
  • Test management tool being utilized
  • The level at which test design and other test work products are to be specified and documented
  • Skills and knowledge of the test analysts
  • The level of maturity of the test process and the organization itself (note that higher maturity requires a greater level of detail, or allow a lesser level
    of detail)
  • Availability of other project stakeholders for consultation

Specifying test conditions in a detailed fashion tends to result in a larger number of test conditions. For example, you might have a single general test condition, “Test checkout,” for an e-commerce application. However, in a detailed test condition document, this might be split into multiple test conditions, with one condition for each supported payment method, one condition for each possible destination country, and so forth.

Some advantages of specifying test conditions at a detailed level include:

  • Facilitates more flexibility in relating other test work products (e.g., test cases) to the test basis and test objectives, thus providing better and more detailed monitoring and control for our Test Manager
  • Contributes to defect prevention, by occurring early in a project for higher levels of testing, as soon as the test basis is established and potentially before system architecture and detailed design are available
  • Relates testing work products to stakeholders in terms that they can understand (often, test cases and other testing work products mean nothing to business stakeholders and simple metrics such as number of test cases executed mean nothing to the coverage requirements of stakeholders)
  • Helps influence and direct not just other testing activities, but also other development activities
  • Enables test design, implementation, and execution, together with the resulting work products to be optimized by more efficient coverage of detailed measures and targets
  • Provides the basis for clearer horizontal traceability within a test level

Some disadvantages of specifying test conditions at a detailed level include:

  • Potentially time-consuming
  • Maintainability can become difficult in a changing environment
  • Level of formality needs to be defined and implemented across the team

Specification of detailed test conditions are particularly effective in the following situations:

  • Lightweight test design documentation methods, such as checklists, are being used due to accommodate the development lifecycle, cost and/or time constraints or other factors
  • Little or no formal requirements or other development work products are available as the test basis
  • The project is a large-scale, complex or high risk and requires a level of monitoring and control that cannot be delivered by simply relating test cases to development work products

Test conditions are specified with less detail when the test basis is related easily and directly to test design work products. This is more likely to be the case for the following:

  • Component level testing
  • Less complex projects where simple hierarchical relationships exist between what is to be tested and how it is to be tested
  • Acceptance testing where use cases are utilized to help define tests