Your developers now ship with a coding agent beside them. Pull requests arrive faster, they are bigger, and some of the code in them was never read line by line by the person who opened them. The test suite that kept up last year does not keep up now.
This guide is for the people who decide how testing works: CTOs, VPs of engineering and QA leads. It covers what automated agents do well in testing, where they fail in ways that are easy to miss, how we run agentic test pipelines at BetterQA with our own tools and engineers, and what to ask any vendor before you sign.
What AI for QA means in practice
AI for QA means using language models and software agents for testing work: drafting test cases from requirements and bug history, writing and repairing automated tests, triaging failures and filing bug reports. It makes that work faster. It does not decide whether a release is safe to ship. A person who did not write the code still has to.
The useful question for a team lead is not "should we use it" (your developers already do) but "which parts of testing can an agent own, and which parts need someone independent of the code". The rest of this guide answers that, task by task.
What changes when code is written in hours
Three things change at once when a team adopts coding agents, and they compound.
Volume. More pull requests land per week, and each touches more files. A regression pass that took two days was fine for a fortnightly release. It is a bottleneck when the team merges every afternoon.
Authorship. Generated code looks finished. It compiles, it is formatted, it often comes with tests. But the person who opened the pull request may have reviewed its intent, not every branch. We wrote about the failure patterns this creates in why generated code fails in production and testing behind assisted development.
Shared blind spots. When the same model writes the feature and the tests for it, both carry the same assumptions. The tests pass because they check what the code does, not what the requirement asked for. That is developers testing their own code with extra steps: students grading their own exams.
Vibe coding ships 10x faster. You also get 10x the bugs. The answer is not to slow the developers down. It is a testing setup that runs at the same speed and is still independent of the people, and the models, that wrote the code.
Testing tasks an agent can own, and what a person still checks
Agents are good at work that is repetitive, well specified and easy to verify afterwards. They are weak at judgement about what matters to your users. Split the work along that line.
| Task | What automation does well | What a person still checks |
|---|---|---|
| Test case design | Drafts cases from requirements, tickets and past bugs, and flags near-duplicates | That the cases cover the business rule, not just the happy path |
| Regression from bug history | Turns each fixed bug into a test that would have caught it | That the test fails on the old code, so it guards something |
| Test scripting | Records a journey once and produces runnable tests | Assertions: is it checking the right value, or any value? |
| Selector repair | Fixes tests when a button moves or an ID changes | That the repair did not point the test at the wrong element |
| Bug reports | Turns a screenshot or log into a report with steps and test cases | Severity, and whether it is really one bug or three |
| Security scanning | Runs many scanners and correlates their output in one pass | Which findings are real and exploitable in your context |
| Accessibility | Checks whole user journeys, not single pages | Whether a screen reader user can actually finish the task |
Five failure modes to plan for
None of these show up as a red build. That is what makes them expensive: the dashboard looks healthier than the product.
1. Hallucinated assertions
A generated test asserts that a page "contains a total", not that the total is correct. Or it asserts against a value it read from the current, buggy output. The test passes forever and protects nothing. We have a whole piece on this pattern: every test passed, nothing worked.
2. Self-healing that heals the wrong thing
Automatic selector repair is one of the most useful things an agent does. It is also a way to hide a real bug. If the "Pay" button disappears and the test quietly re-attaches to a different button, the suite stays green while checkout is broken. Every repair needs a record and a human look.
3. Tests written from the code, not from the requirement
Ask a model to "write tests for this function" and it describes what the function does today, bugs included. Tests have to start from what the ticket or the user needs, which means someone has to own the requirement side.
4. Nobody checks the checker
If an agent writes the code, another agent writes the tests and a third summarises the results, there is no point at which anyone independent looked. A guard nobody has seen fail is not a guard. We go deeper in who validates the validators.
5. Data that should not leave
Pasting production logs, customer records or source code into a public chat tool is a data protection question before it is a testing question. Ask where every prompt goes. More on that in testing tools when data cannot leave.
How we run agentic test pipelines
We only test, we never build. We built our own QA tools, and our engineers, who test independently of your developers, drive them on your product. The tools do the volume. The engineers own the verdict.
| Step | What happens | Tool |
|---|---|---|
| 1 | Requirements and tickets come in and are traced to the test cases that cover them, so gaps in coverage are visible per ticket | BugBoard |
| 2 | Bug history becomes regression tests that would have caught each bug, checked against existing cases to avoid duplicates | BugBoard |
| 3 | Key journeys are recorded once as tests and re-run on every release, alongside the Playwright or Cypress suites in your own repository | Flows |
| 4 | Security is tested in the same pass: 15 open-source scanners plus 20 checks we wrote, correlated into one set of findings | AI Security Toolkit |
| 5 | Accessibility is tested across whole journeys, so the failure that stops a blind shopper halfway through paying gets caught | Auditi |
| 6 | A screenshot or video of a problem becomes a bug report with steps and test cases in minutes, linked to its Jira story | BugBoard |
| 7 | An engineer reviews every finding before you see it, and the release decision comes as a report your auditors can read | Our engineers |
BugBoard and Flows connect to coding agents and editors through MCP, so our engineers drive them from the same tools your developers use; the Flows MCP server is in private beta for client teams. Your tests and data stay yours: automated tests live in your repository from day one, and bugs and test cases export from BugBoard.
Proof from one engagement. On a fintech platform (client name withheld), a manual security round confirmed 38 findings. A second, wider round with the AI Security Toolkit confirmed 239, including 9 critical and 67 high. The second round covered more ground, so the toolkit is not the whole jump, but it shows what one pass over a large surface can surface for an engineer to verify.
The full picture, industry by industry, is on our future of QA page. The service itself is described under agentic QA testing, and the early thinking behind the pipelines is in agentic QA pipelines.
Build it, buy a tool, or bring in a team
Most engineering leaders end up choosing between three routes. Each one works for some teams.
Build it in-house
Your developers wire a coding agent into the test suite. It is the cheapest start and the hardest to keep honest, because the people writing the code also own the tests. It works best when you already have a QA lead with time to review what the agent produces.
Buy a testing platform
A platform gives you generation, execution and repair in one product. You still need people to decide what to test, to read the results and to own the release call. Check what happens to your tests if you leave.
Bring in a QA team that brings its own tools
A managed team, or QA engineers who join your team, run the pipeline for you and stay independent of your developers. This is how most of our clients work with us: through QA outsourcing for a whole product, or test automation for a suite that has stopped keeping up.
What to ask a vendor before you sign
Every testing vendor now says its product uses AI. These questions separate a method from a label. We expand on several of them in what testing vendors won't tell you before you sign.
- Who reviews what the agent produces? Ask for a name or a role, not "the platform".
- How do you show that a generated test can fail? A test that has never failed has not proved anything.
- What happens when a test repairs itself? Is the repair logged, and does a person approve it?
- Are your testers independent of the people who wrote the code? That includes the models that wrote it.
- Where do our code, logs and test data go? Which models see them, and under which agreement?
- What do we keep if we leave? Tests in a standard format, bug history and test cases, exported.
- What evidence do we get per release? Something your auditors or regulator can read, not a pass rate.
- Which certifications does your team hold today? Ask for the certificate, and check the dates.
Frequently asked questions
It can draft test cases from requirements and past bugs, record and repair automated tests, triage failed runs, turn screenshots into bug reports and run security and accessibility checks at scale. Each of those outputs still needs review by someone independent of the code, because an agent can be confidently wrong.
It replaces some of the typing, not the judgement. Deciding what to test, spotting an assertion that checks nothing and making the release call are the parts that matter most when code is generated at speed, and they need a person who did not write the code.
It is a testing setup where software agents carry out chained testing tasks, such as reading a ticket, generating test cases, running them and filing the bugs they find, with engineers reviewing the results. At BetterQA the agents drive our own tools, BugBoard, Flows, Auditi and the AI Security Toolkit, and our engineers own the verdict.
They save a lot of maintenance, but a repair can attach a test to the wrong element and hide a real bug. Treat every automatic repair as a change that someone reviews, and keep a record of what was repaired and why.
Yes. Automated tests live in your repository from day one, and your bugs and test cases export from BugBoard, so your test history stays with you when an engagement ends.
Engagements start from one month, with an NDA and DPA signed from day one. The first month is a trial period, the first two weeks are complimentary if you are not happy, and after that the notice period is 30 days. Rates run from €20 to €45 per hour.
Test generated code at the speed it is written
Our own tools run the pipelines; QA engineers who test independently of your developers check every finding. The first month is a trial, and the first two weeks are complimentary if you are not happy.
Talk to our teamNeed help with software testing?
BetterQA provides independent QA services across manual testing, automation, security audits, and performance testing. ISO 27001, 9001, 14001 and 13485 certified.
