QA glossary
What is defect leakage?
Defect leakage is the share of defects that testing missed and a user or the live system found later. It is calculated as defects found after release divided by all defects found, before and after. A low figure means testing caught most problems before customers did.
An example from QA practice
In one release the team logs 90 defects during testing, and customers report 10 more in the first month. Defects found in total: 100. Defect leakage: 10 divided by 100, or 10%. The next release has 95 found in testing and 5 reported after release, which is 5%. The team then reads the 10 and 5 escaped bugs to see what kind of check was missing.
Using the number well
- Count one release at a time: a figure for a single release can be traced to its causes; an average over a year cannot.
- Agree what "after release" means: a defect reported by a customer, found in production monitoring or found during the next sprint are different things.
- Read the escaped bugs: the value of the number is in the list behind it. Ask what test would have caught each one.
- Do not turn it into a target: if testers are scored on it, fewer bugs get logged as defects.
Leakage versus density
Defect leakage measures how well testing works. Defect density measures how defective the code is. A module can have high density and low leakage if the team found and fixed the bugs before release.
How BetterQA does it
We take the figure from the bug tracker the team already uses, BugBoard or the client's own Jira, so there is no separate spreadsheet to keep in step. For each escaped defect our engineers write down which test would have caught it and add that test, which is how the number is meant to be used.