QA glossary
What is a test oracle?
A test oracle is whatever tells a tester what the right result is, so the actual result can be judged pass or fail. The ISTQB glossary defines it as "A source to determine an expected result." It can be a specification, an existing system, a calculation or an expert.
An example from QA practice
A payroll system is being replaced. For the new one, the old system is the oracle: the team runs last month's payroll through both and compares every payslip. Where they differ, someone checks the tax rules to decide which system is wrong. For a new bonus rule that the old system never had, the oracle is the written policy and a spreadsheet the finance team worked out by hand.
Common oracles
- The test basis: requirements, acceptance criteria and specifications.
- A previous version or a competing system that does the same job.
- People: a domain expert's knowledge of what is correct.
- An automated oracle: code that computes or checks expected results, used when inputs are generated or human checking costs too much.
The test oracle problem
Sometimes no affordable oracle exists. The ISTQB Test Analyst syllabus (CTAL-TA v4.0, section 1.3.4) names the usual causes: complex data, non-determinism such as in AI-based systems, probabilistic behaviour and missing or ambiguous requirements. For AI systems, the CT-AI v2.0 syllabus (section 4.1.3) suggests agreed ranges and tolerances in place of exact values.
How BetterQA does it
Every bug we file in BugBoard states the expected result next to the actual one, and the expected result names where it comes from: the requirement, the acceptance criterion or the behaviour the client confirmed. When an AI tool helps write a test or a report, an engineer reviews the result, and a test whose expected result cannot be traced to a source is not accepted.