Test agents vs human testers: what 175 bugs showed us
Short answer
On two live client products, our test agents found 59 bugs and our human testers found 116. The two lists overlap less than you would expect. Agents are strongest where the problem lives in code, APIs and markup. People are strongest where the problem only shows up when a real person uses the product, in a real place, in their own language. Both sides found critical bugs, so dropping either one ships real problems.
From 27 September to 29 October 2026 we ran our own test agents (scripted browsers, API probes, code and accessibility audits) next to experienced testers on two client projects: a live online learning product and a booking marketplace. Neither side was told to stay out of the other's way. This article sets the two lists side by side.

59 bugs from agents, 116 from people
Both sides worked on both products at the same time. The split above is not about which side is better. The next section shows why: each side caught whole kinds of problems the other barely touched.
What each side found
We sorted all 175 bugs into twelve kinds. Where a bar is missing, the count is zero.
| Kind of bug | Human testers | Test agents |
|---|---|---|
| Features not working as intended | 40 |
8 |
| Data and integrations | 14 |
11 |
| Wording and translation | 13 |
3 |
| Business rules (pricing, booking, billing) | 12 |
12 |
| Confusing flows | 12 |
5 |
| Security | 6 |
9 |
| Form validation | 7 |
2 |
| Accessibility | 0 |
5 |
| Layout and visuals | 5 |
1 |
| Crashes and errors | 3 |
3 |
| Slow pages | 3 |
0 |
| Real devices and networks | 1 |
0 |
| Total | 116 |
59 |
Bar length is scaled to the largest single count (40). Counts come from the bug titles and labels, as described in the last section.
The agents are strongest where the problem lives in code, APIs and markup. People are strongest where the problem only shows up when a real person uses the product, in a real place, in their own language.
Problems you cannot see from the screen
These five were found by the agents and by nobody else. A tester clicking through the product would have had no reason to suspect any of them.
A code audit showed a booking could be marked as paid without any payment going through. No tester clicking through the site would see it.
An API probe using only the app's public key returned other users' personal details. Nothing on screen hinted at it.
Form errors and checkboxes that blind users could not hear. All 5 accessibility bugs came from the agents; the testers found none.
Comparing two internal reports showed that the admin's payable total left out compensation the providers could see. Tedious by hand, fast for an agent.
The same booking showed different times on two pages. The agent caught it by forcing the browser into another time zone.
Problems only a real user notices
These five were found by human testers and by nobody else. Each one needed a person in a real place, using the product the way customers do.
A learner in another country could not join the video session at all. It was only visible from a real network in a real place.
Found by sitting through a real multi-person call.
Signing in again from another tab left the pay button disabled. Real people use several tabs; scripts do not.
An email in another language addressed the reader with the wrong role's wording. Of the 16 wording and translation bugs, 13 came from people and 3 from agents.
A booking email showed a generic map instead of the real venue layout. A person compared the two; nothing else did.
Where the two lists meet
Business rules, 12 each. Booking, pricing and billing rules were caught from both directions, sometimes the same bug twice.
Time zones. Both sides found time-zone bugs on both products. One agent finding was closed as a duplicate of a tester's.
Agent first, person second. The billing gap the agents found was then confirmed by a tester's retest. That hand-off is how we run every engagement.
How serious were they?
| Found by | Critical | High | Medium | Low | Total | High or critical |
|---|---|---|---|---|---|---|
| Human testers | 2 | 28 | 47 | 39 | 116 | 26% |
| Test agents | 4 | 7 | 33 | 15 | 59 | 19% |
Both sides found critical bugs. Neither list is the "important" one: drop either and real, expensive problems ship.
What this means for a team shipping fast
If your code is written with coding assistants and checked by tests the same tools wrote, your checks share the code's blind spots.
Our agents cover the code-level side at the speed you ship: the exposed API, the unannounced error, the report that does not add up. Our testers cover the side your users actually experience: the wording, the second tab, the learner on a different network.
You need both, and you do not need to test every release by hand to get the second half. If you want to read about the two halves separately, see our pages on agentic QA testing, Flows, our browser test recorder, accessibility testing and security testing, or how outsourced QA adds testers to a team that already ships fast.
The tools behind it
None of this works with an off-the-shelf stack and a spreadsheet. We built the tools we test with, and every engagement gets them included:
- BugBoard turns a screenshot or a screen recording into a structured bug report with test cases in minutes, and produces the release-decision report a team can show its auditors.
- Flows records a browser test and proposes a fix when selectors change, which an engineer approves. Tests export to Playwright, Cypress and Selenium, so they stay with the client when an engagement ends.
- The AI Security Toolkit runs 15 open-source scanners plus 20 checks we wrote, and correlates what they find in one pass. That is where the skipped-payment and open-data findings came from.
- Auditi handles accessibility and compliance audits, and BetterFlow keeps the team's time and checklists honest.
Here is the first half of that loop in practice, from a screenshot to a bug report to an automated test:
An engineer reviews everything our tools find before it reaches a client, because automated tools can be confidently wrong. Every one of the 59 agent findings above went through a person before it was filed.
How we counted
We took all 175 bugs filed on two client projects between 27 September and 29 October 2026. We sorted them by who found them (the agents' account or a named tester's run) and by kind, from the bug title and labels. Severity is as filed. Client names are withheld.
Two projects over five weeks is a small sample, and the kind of each bug was assigned by us from titles and labels. Read the numbers as a picture of how the two sides differ, not as a benchmark you can compare to other teams.
Frequently asked questions
Try it on your product
Pick the one flow your customers use most. We run agents and a tester on it for a day, complimentary, and show you what each side finds.
Book a call Agentic QA testing