AI QA vs human QA: who wins? What our own tools taught us

Green tests are not a verdict

Three security changes to Flows, our test automation tool, went through the full routine. Each came with tests we had watched fail before the fix and pass after it. Each passed the type checks, the linter and the whole suite.

In all three, an independent reviewer whose only job was to break the change found a critical defect:

• A cap read as a total. At a 1,000-row limit, a full test suite looked empty, so a delete guarded by "confirm how many items" could go through after confirming zero.
• A wildcard address slipped past an address rule. The new rule was weaker than the one it replaced, on one engine.
• Credentials without a scope check. Several tools used saved credentials for whatever project the caller named.

The tests were correct. They tested what the author thought of. The reviewer thought of something else.

The automatic check that never caught anything

Flows can add assertions to a test automatically. We measured them: 87 ran across 17 flows on 5 applications. 5 failed, and all 5 were false alarms. Not one caught a real defect.

61 of the 87 were close to empty. One kind only checked that the address kept the same domain, so a redirect to the login page still passed. The other only checked that the page had a title. They could not fail, which made them safe and useless. We fixed seven defects that came out of the study.

Two reviews passed it. The third did not.

Hireo, our recruiting platform, got a feature that lets a recruiter choose which screening questions a client sees. Two review rounds, one on correctness and one on design, found and fixed eight issues.

A third review, started with fresh context, found the one that mattered: saving any unrelated setting would have stored "show no questions" on every job that had no screening answers yet. That was 3,470 of 3,473 jobs. It was caught before release.

When every fix makes it worse

In JRNY, our sales CRM, we tried to change how job posts are classified. Each of four review rounds found a new critical defect, and each one was caused by the fix for the previous round.

After the fourth we stopped. We shipped the small, safe half and closed the rest unmerged. The prize was two possible leads, and that was not worth a fifth round of risk. Knowing when to stop is a judgement call, and no test suite makes it for you.

One rule, many answers

In BetterFlow, our time and leave platform, one business rule decides whether an unconfirmed holiday counts as a day off. We found it answered in more than a dozen places, with two different answers.

The answers split exactly where money is: leave days and overtime settlement. Every place had passing tests. The disagreement only shows when someone reads across all of them.

So who wins?

Neither, on its own.

AI agents give you volume. They write tests, run them and review every change at a pace no team can match. They also miss things with full confidence, and an AI check can pass while checking nothing.

What caught these bugs was independence. A reviewer who did not write the code and is told to break it. Fresh context every round. An engineer who reads the evidence and makes the call, including the call to stop.

To be precise: several of those independent reviews were run by AI agents that our engineers briefed and checked. The lesson is not human against machine. It is that nobody, person or model, should grade their own work. That is also why we only test and never build our clients' products.

What to take to your own team

• Treat green as a claim. Ask what a passing test would look like if the feature were broken.
• Measure your AI checks. Count how many have ever caught a real defect. Ours: zero of 87.
• Review with fresh context. The third look found what the first two could not.
• Keep a human decision at the end. Including the decision that a change is not worth shipping.