QA glossary
What is an AI hallucination?
A hallucination is wrong information that a large language model (LLM) produces and presents as if it were true. The ISTQB glossary defines it as "Wrong information created by an LLM." The answer reads fluently, which is why it is easy to miss.
An example from QA practice
A team asks an AI assistant to write test cases from a user story about password reset. The assistant returns twelve cases. Ten are sound. One checks that the reset link expires after 24 hours, a rule that appears nowhere in the story or the product, and one calls an API endpoint that does not exist. Both look as convincing as the other ten. A tester who runs them without checking the source ends up reporting a "bug" against a requirement nobody wrote.
Why it matters when you buy testing
- In your product: if your app has a chatbot or an AI feature, a confident wrong answer reaches your customer with your name on it.
- In your testing: if a supplier uses AI to write tests or bug reports, a hallucinated test can pass for the wrong reason or flag a defect that is not there.
- Ask the supplier: who checks AI-generated test cases against the real requirements before they count?
Where the definition comes from
The quoted definition is from the ISTQB glossary. The ISTQB Testing with Generative AI syllabus (CT-GenAI v1.1, section 3.1.1) gives examples from testing work: fictitious test cases, test scripts that do not run, and tests for acceptance criteria that do not exist. Section 3.1.2 lists the ways to catch them: compare the output with documentation and known system behaviour, ask a subject matter expert, and check outputs for consistency with each other.
How BetterQA does it
Our own tools use AI, so we apply the same rule to them that we apply to a client's product: an engineer reviews every AI-generated result before it counts. A test case, a bug report or a finding that cannot be traced to the requirement or to observed behaviour does not go into BugBoard. When we test a client's AI feature, our engineers check its answers against the product's own documentation and data, the cross-verification the syllabus describes.