QA glossary
What is data drift?
Data drift is a change over time in the data a live machine learning model receives, so that it no longer looks like the data the model was trained on. The model has not changed, but its accuracy can fall without any error being raised.
An example from QA practice
A spam filter was trained on last year's email. This year attackers send a new kind of phishing message the training data never contained. Nothing in the code changed and every automated test still passes, but more spam reaches the inbox each month. A drift test compares the incoming emails, or the filter's predictions, with the training data and raises a flag when the difference crosses an agreed threshold.
Data drift versus concept drift
The ISTQB AI Testing syllabus (CT-AI v2.0, section 6.1.7) separates two forms. In data drift the statistical properties of the input change, for example through shifts in user behaviour or seasonality. In concept drift the right answer for the same input changes: after a new financial regulation, a transaction type that was low-risk may now count as high-risk.
Two ways to test for it
- Dynamic drift testing compares the model's output with fresh ground truth, such as user ratings or corrections, against a threshold.
- Static drift testing needs no ground truth: it compares the distributions of input or predicted output with the training data using a statistical test such as Kolmogorov-Smirnov.
- Run it after release: drift happens in production, so a test suite that only ran before launch cannot see it.
How BetterQA does it
We only test, we never build, so we do not retrain a client's model. What we do is report drift with the evidence: the reference inputs, the current outputs and the difference, logged in BugBoard so the client's data team can decide what to retrain. Where AI helps us analyse results, an engineer reviews every AI-generated result before it is reported.