Our AI automation engine builds and repairs tests. Here is what you need to know.
An automation engine that says a test passed is making a claim. Here is how often ours was wrong, and what we changed.
Flows, our test automation engine, builds browser tests, runs them, and repairs them when the page changes. It can also automate test cases written in BugBoard, which can turn a screenshot into test cases. We built both. Here is what we learned measuring Flows.
A failed step that said "passed"
Inside Flows, every command type hands back a result. 32 of the 134 command types reported a failure by returning it instead of raising an error. The code that built the step result wrote "success" without looking. So when one of those 32 failed, the step still passed.
A code reviewer found it, not a red test, because nothing was ever red. Failures now reach the report, and failure evidence is captured for them too.
Self-healing: what actually heals
When a selector stops matching, Flows tries to repair it. People assume the AI does that work. Before we measured, nobody could say, because only successful repairs were counted and nothing recorded which method made them.
So we built a fixed set of drift cases and ran it five times, twice. 75 of 90 attempts healed, with the same result every pass. A plain fallback rule made 65 of those repairs and a page scorer made the other 10. Three deliberately hard cases failed every time, and we report them rather than hide them.
The lesson: most repairs are cheap, predictable rules. That is good news for cost and stability, and it is the right baseline to judge the AI repair against.
Knowing what belongs to the tool
A healer looks at the page for a new candidate element. Flows also puts its own panel on that page. The healer has to know which elements are ours, so it never offers our own controls as a repair. Its first version decided that by element name, which any page could imitate. It now decides by identity, the same way the rest of the extension does.
AI assertions: measure them before you trust them
Flows can add checks automatically. Of 87 that ran, 5 failed, all false alarms, and none caught a real defect. 61 of the 87 checked only that the page kept its domain or had a title, so a redirect to the login page passed.
Our mobile engine had the opposite trap: its AI checks defaulted to optional, so a failed check still passed. Creating one without an explicit choice is now refused.
What you need to know before trusting any AI test engine
- Ask how a failure reaches the report. Then break something on purpose and watch it arrive.
- Ask which method made each repair. A heal rate with no breakdown cannot tell you whether the AI helps.
- Count what your automatic checks have caught. A check that cannot fail is not a check.
- Keep your tests portable. Flows tests export to Playwright, Cypress and Selenium, so you are never locked in.
- Keep an engineer reading the results. We do, on every engagement.
Want automated tests you can actually trust?
Our engineers build and run your test automation with tools we built ourselves, and read every result. The first month is a trial.
