Once risk identification has occurred, risk assessment can begin, being the study of these identified risks. Specifically, risk assessment involves categorizing each risk and determining the likelihood and impact associated with each risk. Risk assessment may also involve evaluating or assigning other properties of each risk, such as risk owner.
Categorization of risk means placing each risk into an appropriate type, such as performance, reliability, functionality, and so forth. We used the ISO 9126 standard (which is being replaced by the ISO 25000 standard) quality characteristics for categorization, but many organizations use other categorization schemes. The same checklist used for risk identification is often used to categorize the risks. Generic quality risk checklists exist and we can customize these checklists. When using checklists as a foundation of risk identification, categorization of the risk often occurs during identification.
Determining the level of risk typically involves assessing, for each risk item, the likelihood of occurrence, and the impact upon occurrence. The likelihood of occurrence is the probability that the potential problem exists in the system under test. In other words, likelihood is an assessment of the level of technical risk. Factors influencing likelihood for product and project risks include:
- Complexity of technology and teams
- Personnel and training issues among the business analysts, designers, and programmers
- Conflict within the team
- Contractual problems with suppliers
- Geographically distributed team
- Legacy versus new approaches
- Tools and technology
- Weak managerial or technical leadership
- Time, resource, budget and management pressure
- Lack of earlier quality assurance activities
- High change rates
- High earlier defect rates
- Interfacing and integration issues
The impact upon occurrence is the severity of the effect on the users, customers, or other stakeholders. Factors influencing impact in project and product risks include:
- Frequency of use of the affected feature
- Criticality of the feature to accomplishing a business goal
- Damage to reputation
- Loss of business
- Potential financial, ecological or social losses or liability
- Civil or criminal legal sanctions
- Loss of license
- Lack of reasonable workarounds
- Visibility of failure leading to negative publicity
The level of risk is assessed either quantitatively or qualitatively. If likelihood and impact are ascertained quantitatively, one can multiply the two values together to calculate a quantitative risk priority number. Typically, though, the level of risk can only be ascertained qualitatively. That is, one can speak of likelihood being very high, high, medium, low, or very low, but one cannot express likelihood as a percentage with any real precision; similarly, one can speak of impact being very high, high, medium, low, or very low, but one cannot express impact in financial terms in a complete or precise fashion. This qualitative assessment of risk levels should not be seen as inferior to quantitative approaches; indeed, when quantitative assessments of risk levels are used inappropriately, the results mislead the stakeholders about the extent to which one actually understands and can manage risk. Qualitative assessments of risk levels are often combined, through multiplication or addition, to create an aggregate risk score. This aggregate risk score may be expressed as a risk priority number but should be interpreted as a qualitative, relative rating on an ordinal scale.
In addition, unless the risk analysis is based upon extensive and statistically valid risk data, the risk analysis will be based on the stakeholders’ subjective perceptions of likelihood and impact. Project managers, programmers, users, business analysts, architects, and testers typically have different perceptions and thus possibly different opinions on the level of risk for each risk item. The risk analysis process includes some way of reaching consensus or, in the worst case, establishing an agreed-upon level of risk (e.g., through management dictate or by taking the average, median, or mode level for the risk item). Furthermore, the risk levels are checked for good distribution through the range, to ensure that the risk rates provide meaningful guidance in terms of test sequencing, prioritization, and effort allocation. Otherwise, risk levels cannot be used as a guide for risk mitigation activities.