If you’re evaluating QA vendors right now, there’s a decent chance one of the pitches in your inbox is from a company selling agentic, AI-driven, end-to-end testing. No human testers, just an AI that writes and runs your test suite. We’ve watched a few of these demos ourselves, and the honest reaction is mixed: genuinely impressive for what it covers, and quietly worrying about what it doesn’t mention.
The coverage number is smaller than the pitch implies
Teams who’ve actually run these tools in production tend to land in the same range: somewhere around a third to just under half of what a real QA process needs gets covered. That’s a real number, not nothing. It’s genuinely useful for regression testing on features that are stable and haven’t changed in a while.
Where it falls apart is anything new. A feature that’s still changing, a workflow nobody’s fully speced out, an edge case that only makes sense once you understand why a customer would even attempt it. These tools are good at confirming that yesterday’s behavior still holds. They’re much weaker at figuring out what tomorrow’s behavior should even be checked for.
A test that heals itself can hide a real bug
Self-healing selectors are the feature every AI testing vendor leads with, and they demo well. When your UI changes, the tool finds the new element on its own and keeps the test running instead of failing.
Here’s the part that doesn’t make it into the demo: a test that heals past a real regression doesn’t fail. It keeps passing against something subtly different from what it was supposed to check. In a low-stakes app that’s a shrug. Somewhere money changes hands, a self-healing suite can wave a genuine bug straight through a green build, which is arguably worse than having no automated coverage at all. A green build is exactly what tells a team to stop looking.
The part no demo can show you
An AI tool will happily generate two hundred test cases for your login page. Generating tests stopped being the hard part a while ago.
The hard part is knowing which thirty flows in your product actually cost you money or trust when they break, and making sure those get covered first and covered well. That takes understanding your specific product and your specific customers, not a general capability a model ships with out of the box. It’s also exactly the piece a vendor can’t demo in thirty minutes, because it isn’t about the tool, it’s about your business.
Three questions before you sign anything
What percentage of our actual coverage need does this address today, not on the roadmap. What happens to a test when the UI changes: does it heal silently, or flag the change for a person to confirm. Who’s deciding which user flows matter most, the tool or someone who has read our product spec.
We’re not telling you to skip AI-driven testing. We’re telling you to find out exactly which 30 to 40 percent you’re buying, on purpose, and decide who’s covering the rest before the contract is signed rather than after the first incident.
If you want a second opinion on a vendor pitch, or you’re weighing AI-driven testing against a managed QA partnership, that’s a conversation we have with teams often. More on how we approach it on our testing services page.
Built by BetterQA.
Need help with software testing?
BetterQA provides independent QA services across manual testing, automation, security audits, and performance testing. ISO 27001, 9001, 14001 and 13485 certified.
