When your AI test tool says passed, how do you know it is right?
For Heads of QA, CTOs and buyers who are about to rely on an AI test tool's green run.
A test tool that says "passed" is making a claim. We measured Flows, the browser test automation engine BetterQA builds, and found three ways that claim can be wrong. We now ask the same three questions of any AI test tool, our own included.
- AI test automation
- Our own test automation engine
- Reports
- Repairs
- Checks
- Mobile
What was at risk
A failure reported as a pass
32 of the 134 command types reported a failure by returning it, not raising an error. The step result was written as success without looking.
Self-healing nobody can explain
Only successful repairs were counted and nothing recorded which method made them, so nobody could say whether the AI helped.
A check that cannot fail
61 of the 87 automatic checks only confirmed that the page kept its domain or had a title, so a redirect to the login page passed.
An AI check that passes when it fails
The mobile engine's AI checks defaulted to optional, so a failed check still passed.
Five questions we ask of any AI test tool, ours included
- FailureHow does a failure reach the report? Can the vendor break something on purpose, in front of you?
- RepairWhich method made each repair, and how many did each make?
- CaughtWhat have your automatic checks ever caught? How many real defects?
- ExitCan I take my tests with me? Ours export to Playwright, Cypress and Selenium.
- ReaderWho reads the results? An engineer should, on every run.
How we found it
- Where
- Flows, the browser test automation engine BetterQA builds, and our mobile test engine.
- How
- A code reviewer found the failure-reporting gap, not a red test, because nothing was ever red. For repairs we built a fixed set of drift cases and ran it five times, twice. We counted the automatic assertions.
- What counted
- A tool that says "passed" is making a claim. Whether it is right depends on things most vendor demos never show you.
- How we workedThe cases we did not healWhere it showed upThree deliberately hard cases failed every time, and we report them instead of hiding them. A heal rate with no breakdown cannot tell you whether the AI helps.
What we found, and what it costs
- Finding1 of 3Failure as success: 32 of 134 command types reported a failure by returning itWhat it costs youWhen one of those failed, the step still passed, and nothing was ever red.
- Finding2 of 3Repairs with no breakdown: 75 of 90 healed, 65 by a plain rule and 10 by a page scorerWhat it costs youMost repairs are cheap, predictable rules, which is good for cost and stability, and is the baseline to judge any AI repair against.
- Finding3 of 3Automatic checks: 87 ran, 5 failed, all 5 were false alarmsWhat it costs youNone caught a real defect, and a redirect to the login page passed.
The mobile engine had the opposite trap, AI checks that defaulted to optional.
What changed
The mobile engine now refuses to create an AI check without an explicit choice on whether it may fail. We now ask the same three questions of any AI test tool we are asked to run, our own included.
The longer write-up: three questions to ask before you trust an AI test tool's green run.
Want test automation you can trust?
Our engineers build and run your test automation with these questions as the acceptance bar. The first month is a trial.
