Test agents vs human testers: what 175 bugs showed us
Short answer
On two live client products, our test agents found 59 bugs and our human testers found 116. The two lists overlap less than you would expect. Agents are strongest where the problem lives in code, APIs and markup. People are strongest where the problem only shows up when a real person uses the product, in a real place, in their own language. Both sides found critical bugs, so dropping either one ships real problems.
From 27 September to 29 October 2026 we ran our own test agents (scripted browsers, API probes, code and accessibility audits) next to experienced testers on two client projects: a live online learning product and a booking marketplace. Neither side was told to stay out of the other's way. This article sets the two lists side by side.
59 bugs from agents, 116 from people
Both sides worked on both products at the same time. The split above is not about which side is better. The next section shows why: each side caught whole kinds of problems the other barely touched.
What each side found
We sorted all 175 bugs into twelve kinds. Where a bar is missing, the count is zero.
| Kind of bug | Human testers | Test agents |
|---|---|---|
| Features not working as intended | 40 |
8 |
| Data and integrations | 14 |
11 |
| Wording and translation | 13 |
3 |
| Business rules (pricing, booking, billing) | 12 |
12 |
| Confusing flows | 12 |
5 |
| Security | 6 |
9 |
| Form validation | 7 |
2 |
| Accessibility | 0 |
5 |
| Layout and visuals | 5 |
1 |
| Crashes and errors | 3 |
3 |
| Slow pages | 3 |
0 |
| Real devices and networks | 1 |
0 |
| Total | 116 |
59 |
Bar length is scaled to the largest single count (40). Counts come from the bug titles and labels, as described in the last section.
The agents are strongest where the problem lives in code, APIs and markup. People are strongest where the problem only shows up when a real person uses the product, in a real place, in their own language.
Problems you cannot see from the screen
These five were found by the agents and by nobody else. A tester clicking through the product would have had no reason to suspect any of them.
Reading the code showed that a host could mark a booking as paid without paying. No tester clicking through the site would see it.
A probe using the public key returned every user's address. Nothing on screen hinted at it.
Form errors and checkboxes that blind users could not hear. All 5 accessibility bugs came from the agents; the testers found none.
Comparing two internal reports showed that the admin's payable total left out compensation the providers could see. Tedious by hand, fast for an agent.
The same booking showed different times on two pages. The agent caught it by forcing the browser into another time zone.
Problems only a real user notices
These five were found by human testers and by nobody else. Each one needed a person in a real place, using the product the way customers do.
A learner in another country could not join the video session at all. It was only visible from a real network in a real place.
Found by sitting through a real multi-person call.
Signing in again from another tab left the pay button disabled. Real people use several tabs; scripts do not.
An email in another language addressed the reader with the wrong role's wording. Of the 16 wording and translation bugs, 13 came from people and 3 from agents.
A booking email showed a generic map instead of the real venue layout. A person compared the two; nothing else did.
Where the two lists meet
Business rules, 12 each. Booking, pricing and billing rules were caught from both directions, sometimes the same bug twice.
Time zones. Both sides found time-zone bugs on both products. One agent finding was closed as a duplicate of a tester's.
Agent first, person second. The billing gap the agents found was then confirmed by a tester's retest. That hand-off is how we run every engagement.
How serious were they?
| Found by | Critical | High | Medium | Low | Total | High or critical |
|---|---|---|---|---|---|---|
| Human testers | 2 | 28 | 47 | 39 | 116 | 26% |
| Test agents | 4 | 7 | 33 | 15 | 59 | 19% |
Both sides found critical bugs. Neither list is the "important" one: drop either and real, expensive problems ship.
What this means for a team shipping fast
If your code is written with AI assistants and checked by tests the same tools wrote, your checks share the code's blind spots.
Our agents cover the code-level side at the speed you ship: the exposed API, the unannounced error, the report that does not add up. Our testers cover the side your users actually experience: the wording, the second tab, the learner on a different network.
You need both, and you do not need to test every release by hand to get the second half. If you want to read about the two halves separately, see our pages on agentic QA testing, Flows, our browser test recorder, accessibility testing and security testing, or how outsourced QA adds testers to a team that already ships fast.
How we counted
We took all 175 bugs filed on two client projects between 27 September and 29 October 2026. We sorted them by who found them (the agents' account or a named tester's run) and by kind, from the bug title and labels. Severity is as filed. Client names are withheld.
Two projects over five weeks is a small sample, and the kind of each bug was assigned by us from titles and labels. Read the numbers as a picture of how the two sides differ, not as a benchmark you can compare to other teams.
Frequently asked questions
Try it on your product
Pick the one flow your customers use most. We run agents and a tester on it for a day, complimentary, and show you what each side finds.
Book a call Agentic QA testing
