QA glossary
What is non-determinism in AI testing?
Non-determinism is when a system can give different outputs for the same input. Most AI-based systems, and every large language model, behave this way, so one test run cannot prove they are right and a single failure cannot prove they are wrong.
An example from QA practice
A support chatbot is asked "How do I cancel my order?" ten times with the same wording. Eight answers give the right steps, one links to the wrong help page and one invents a cancellation fee. A test that asked the question once would have passed or failed depending on luck. The useful test asks it many times and checks that every answer stays within agreed limits: right steps, no made-up charges.
Why one run is not enough
The ISTQB AI Testing syllabus (CT-AI v2.0, section 4.1.2) gives non-determinism as the first reason AI-based systems need a statistical approach to testing: a single misclassification does not reflect the model's overall correctness, so the test set must be large enough to give statistically significant results. Section 4.1.3 adds that expected results then become ranges and tolerances, not single values.
What reduces the variation
- Lower the temperature: answers become more consistent, and less varied.
- Set a random seed where the LLM supports one, so runs can be repeated.
- Automate the checks on the output, so every run is judged by the same rule.
These come from the CT-GenAI v1.1 syllabus, section 3.1.4, which also warns that complete reproducibility cannot be guaranteed.
How BetterQA does it
Our own tools use LLMs, so their output varies too. That is one reason an engineer reviews every AI-generated result before it counts. In Flows, our browser test tool, a self-healing selector repair is a change to the test, so an engineer reviews it: a test should not quietly start checking a different element from the one it was written for.