Test agents vs human testers: what 175 bugs showed us

Test agents vs human testers: 175 bugs on two live products show what each side catches that the other misses.
Back to blog

Test agents vs human testers: what 175 bugs showed us

Short answer

On two live client products, our test agents found 59 bugs and our human testers found 116. The two lists overlap less than you would expect. Agents are strongest where the problem lives in code, APIs and markup. People are strongest where the problem only shows up when a real person uses the product, in a real place, in their own language. Both sides found critical bugs, so dropping either one ships real problems.

From 27 September to 29 October 2026 we ran our own test agents (scripted browsers, API probes, code and accessibility audits) next to experienced testers on two client projects: a live online learning product and a booking marketplace. Neither side was told to stay out of the other's way. This article sets the two lists side by side.

Tudor Brad, BetterQA founder, in a grey jacket over a dark shirt, smiling slightly
Article details
Bugs counted
175 across two live products
Period
27 September to 29 October 2026
Audience
Founders, CTOs and heads of product
BetterQA team
50+ engineers in Cluj-Napoca, Romania

59 bugs from agents, 116 from people

59 bugs found by test agents
116 bugs found by human testers on the same products
2 live products: a live online learning product and a booking marketplace

Both sides worked on both products at the same time. The split above is not about which side is better. The next section shows why: each side caught whole kinds of problems the other barely touched.

What each side found

We sorted all 175 bugs into twelve kinds. Where a bar is missing, the count is zero.

Number of bugs found by human testers and by test agents, by kind of bug
Kind of bug Human testers Test agents
Features not working as intended
40
8
Data and integrations
14
11
Wording and translation
13
3
Business rules (pricing, booking, billing)
12
12
Confusing flows
12
5
Security
6
9
Form validation
7
2
Accessibility
0
5
Layout and visuals
5
1
Crashes and errors
3
3
Slow pages
3
0
Real devices and networks
1
0
Total
116
59

Bar length is scaled to the largest single count (40). Counts come from the bug titles and labels, as described in the last section.

The pattern in one line

The agents are strongest where the problem lives in code, APIs and markup. People are strongest where the problem only shows up when a real person uses the product, in a real place, in their own language.

Problems you cannot see from the screen

These five were found by the agents and by nobody else. A tester clicking through the product would have had no reason to suspect any of them.

Code audit
Payments that could be skipped

A code audit showed a booking could be marked as paid without any payment going through. No tester clicking through the site would see it.

API probe
Personal data readable by anyone

An API probe using only the app's public key returned other users' personal details. Nothing on screen hinted at it.

Accessibility audit
Errors a screen reader never announced

Form errors and checkboxes that blind users could not hear. All 5 accessibility bugs came from the agents; the testers found none.

Report comparison
Billing that did not add up

Comparing two internal reports showed that the admin's payable total left out compensation the providers could see. Tedious by hand, fast for an agent.

Forced time zone
Times that disagreed

The same booking showed different times on two pages. The agent caught it by forcing the browser into another time zone.

Problems only a real user notices

These five were found by human testers and by nobody else. Each one needed a person in a real place, using the product the way customers do.

Critical
A live session that would not connect from abroad

A learner in another country could not join the video session at all. It was only visible from a real network in a real place.

Real call
Recordings that stopped at 10 minutes

Found by sitting through a real multi-person call.

Several tabs
A pay button stuck after a second login

Signing in again from another tab left the pay button disabled. Real people use several tabs; scripts do not.

Own language
The wrong words for the reader

An email in another language addressed the reader with the wrong role's wording. Of the 16 wording and translation bugs, 13 came from people and 3 from agents.

Email vs product
Emails that did not match the product

A booking email showed a generic map instead of the real venue layout. A person compared the two; nothing else did.

Where the two lists meet

Business rules, 12 each. Booking, pricing and billing rules were caught from both directions, sometimes the same bug twice.

Time zones. Both sides found time-zone bugs on both products. One agent finding was closed as a duplicate of a tester's.

Agent first, person second. The billing gap the agents found was then confirmed by a tester's retest. That hand-off is how we run every engagement.

How serious were they?

Bugs by severity, as filed, for human testers and test agents
Found by Critical High Medium Low Total High or critical
Human testers 2 28 47 39 116 26%
Test agents 4 7 33 15 59 19%

Both sides found critical bugs. Neither list is the "important" one: drop either and real, expensive problems ship.

What this means for a team shipping fast

If your code is written with coding assistants and checked by tests the same tools wrote, your checks share the code's blind spots.

Our agents cover the code-level side at the speed you ship: the exposed API, the unannounced error, the report that does not add up. Our testers cover the side your users actually experience: the wording, the second tab, the learner on a different network.

You need both, and you do not need to test every release by hand to get the second half. If you want to read about the two halves separately, see our pages on agentic QA testing, Flows, our browser test recorder, accessibility testing and security testing, or how outsourced QA adds testers to a team that already ships fast.

The tools behind it

None of this works with an off-the-shelf stack and a spreadsheet. We built the tools we test with, and every engagement gets them included:

  • BugBoard turns a screenshot or a screen recording into a structured bug report with test cases in minutes, and produces the release-decision report a team can show its auditors.
  • Flows records a browser test and proposes a fix when selectors change, which an engineer approves. Tests export to Playwright, Cypress and Selenium, so they stay with the client when an engagement ends.
  • The AI Security Toolkit runs 15 open-source scanners plus 20 checks we wrote, and correlates what they find in one pass. That is where the skipped-payment and open-data findings came from.
  • Auditi handles accessibility and compliance audits, and BetterFlow keeps the team's time and checklists honest.

Here is the first half of that loop in practice, from a screenshot to a bug report to an automated test:

An engineer reviews everything our tools find before it reaches a client, because automated tools can be confidently wrong. Every one of the 59 agent findings above went through a person before it was filed.

How we counted

We took all 175 bugs filed on two client projects between 27 September and 29 October 2026. We sorted them by who found them (the agents' account or a named tester's run) and by kind, from the bug title and labels. Severity is as filed. Client names are withheld.

Two projects over five weeks is a small sample, and the kind of each bug was assigned by us from titles and labels. Read the numbers as a picture of how the two sides differ, not as a benchmark you can compare to other teams.

Frequently asked questions

Not on the evidence of these two projects. Agents found 59 bugs, people found 116, and each side caught whole kinds of problems the other barely touched. Agents found all 5 accessibility bugs and the skipped-payment and open-data problems. People found the class that would not connect from abroad, the wrong wording in an email and the stuck pay button.
Problems that live in code, APIs and markup and do not show on screen: a booking that could be marked as paid without any payment going through, personal data an API returned to anyone who asked, form errors a screen reader never announced, billing totals that did not match between two reports, and times that disagreed between pages.
Problems that only show up when a real person uses the product in a real place: a video session that would not connect from another country, recordings that stopped at 10 minutes, a pay button that stayed disabled after signing in from a second tab, wording that was wrong for the reader, and emails that did not match the product.
No. Agents cover the code-level side at the speed you ship. A tester covers the side your users experience, and you do not need to do that by hand for every release to get the benefit.

Try it on your product

Pick the one flow your customers use most. We run agents and a tester on it for a day, complimentary, and show you what each side finds.

Book a call Agentic QA testing
Share the Post: