LLMs can generate impressive unit tests. They’ll mock your database connections, stub your API calls, and even catch obvious edge cases like null values or empty strings. But when we audit codebases with AI-generated tests at BetterQA, we find the same gaps repeatedly – and they’re not random. They’re systematic blind spots in how LLMs approach testing.
Here’s what AI assistants consistently miss when generating backend tests, based on the production systems we audit.
The pattern: healthy unit coverage, almost no integration coverage
The shape repeats across the Node.js and Python backends we audit where developers report that AI wrote most of the tests. Unit coverage looks solid, often comfortably into the seventies or higher. Integration coverage, meaning tests that verify how components behave across a boundary, is a fraction of it.
This isn’t a coincidence, and it isn’t a criticism of any particular model. LLMs are trained on public repositories where unit tests vastly outnumber integration tests, because integration tests are harder to write, harder to maintain and less likely to be open-sourced. Models generate what they have seen most. The result is a dangerous illusion: a high coverage number sitting on top of untested failure modes.
Independent research points the same way. Cotroneo, Improta and Liguori’s 2025 analysis of more than 500,000 samples (arXiv:2508.21634) found AI-generated code simpler and more repetitive than human-written code while carrying more high-risk vulnerabilities – the profile you would expect from a generator that reproduces common shapes and does not reason about what happens between them.
What LLMs systematically don’t test
1. Multi-service integration boundaries
LLMs excel at testing a single service in isolation. They’ll mock your payment gateway, your email service, and your cache layer. But they rarely test what happens when these services interact in realistic sequences.
| What breaks in production |
|---|
| Payment succeeds but order confirmation email fails – user charges twice retrying |
| Cache invalidation happens before database write completes – stale data served |
| API gateway timeout differs from downstream service timeout – requests hang |
In one audit we found no test at all for the sequence: authenticate user → charge payment → send receipt → update inventory. Every step in it was well covered by unit tests. The sequence itself was covered by nothing.
2. Database transaction edge cases
AI-generated tests typically use transaction rollbacks for cleanup, which is good practice. But they rarely test scenarios where transactions themselves are the problem.
| Missing test cases |
|---|
| Deadlock detection when two transactions lock resources in opposite orders |
| Rollback cascade failures when a parent transaction aborts mid-child-transaction |
| Optimistic locking failures under concurrent updates |
| Connection pool exhaustion during traffic spikes |
One e-commerce platform we tested had comprehensive unit tests for order creation but zero tests for what happens when 500 users simultaneously purchase the last 3 items in stock. In production, this created duplicate orders and overselling – classic race condition, never tested.
3. Authentication and authorization edge cases
LLMs will test that valid tokens work and invalid tokens get rejected. They rarely test the boundary conditions that cause real security issues.
| What’s usually missing |
|---|
| Token expiration during a long-running request |
| Permission changes while user has an active session |
| Concurrent login from multiple devices with session limits |
| Token refresh race conditions |
| RBAC edge cases when user belongs to multiple roles with conflicting permissions |
On one SaaS platform the AI-generated auth tests reported strong coverage and missed the scenario where an admin demotes themselves while performing an admin action. The test suite passed. The production system let the demoted admin complete the action.
4. Rate limiting and throttling
Rate limiting logic is often tested in isolation – “does the rate limiter block after N requests?” But realistic scenarios involve distributed clients, varied request patterns, and timing edge cases.
| Real-world failures we’ve seen |
|---|
| Rate limit counters using Redis keys that expire mid-count |
| Throttling that doesn’t account for retries with exponential backoff |
| Per-IP limits that fail behind corporate NATs or VPNs |
| Burst allowances that interact incorrectly with sliding windows |
LLMs generate tests that verify rate limiting exists. They don’t test whether it actually protects your infrastructure under attack.
5. Concurrent access scenarios
This is perhaps the biggest gap. LLMs think sequentially – their training data is mostly sequential code and sequential tests. Concurrency bugs require thinking about interleaved execution timelines.
| What we find in audits |
|---|
| No tests for concurrent writes to the same resource |
| No simulation of race conditions in background job processing |
| No verification of thread-safety in shared caches |
| No testing of connection pool behavior under concurrent load |
One job queue system reported high coverage and had no test at all for two workers claiming the same job simultaneously. The bug surfaced in production months later, when traffic grew enough to make the race likely.
6. Data migration compatibility
When you ship a new backend version, you need to handle data created by the old version. LLMs rarely generate tests for this backward compatibility.
| Missing scenarios |
|---|
| New code reading old schema data during rolling deployments |
| Migration scripts running while the application is serving traffic |
| Rollback compatibility – can you safely roll back to the previous version? |
| Partial migration states when deployment fails mid-rollout |
One API migration we audited broke a substantial share of active sessions, because the new authentication logic could not parse tokens issued by the previous version. No test caught it.
7. Performance under load
LLMs test functional correctness: does the code produce the right output? They don’t test non-functional requirements: does it do so fast enough, under realistic load?
| What’s not tested |
|---|
| Query performance with tables containing 10M+ rows |
| Memory usage under concurrent request spikes |
| Response time degradation as queue depth increases |
| Resource cleanup after errors (connection leaks, memory leaks) |
An admin dashboard had full test coverage for its individual queries. Against production data volumes the main report query took most of a minute. No test caught it, because the fixtures held a hundred rows where production held tens of millions.
Why LLMs miss these gaps
Three structural reasons:
Training data bias: Public repositories have far more unit tests than integration tests. LLMs generate what they’ve seen most. Integration tests are harder to write, harder to maintain, and less likely to be open-sourced.
Context window limits: Testing integration scenarios requires understanding system architecture, deployment topology, and data flow across services. Even with 200K context windows, LLMs struggle to maintain the full system model needed to identify boundary issues.
Lack of runtime understanding: LLMs reason about code statically. Concurrency bugs, race conditions, and performance issues are runtime phenomena. You can’t reason about them without modeling execution timelines, resource contention, and timing – which requires testing infrastructure most LLMs never see in training.
What independent QA catches
| Professional testers approach systems differently. We model |
|---|
| Failure modes at component boundaries |
| State transitions under concurrent access |
| Real-world data volumes and traffic patterns |
| Degradation scenarios (partial outages, slow dependencies) |
| Security boundaries and privilege escalation paths |
We use techniques LLMs don’t: chaos engineering, property-based testing, concurrency testing frameworks, and load simulation that mirrors production traffic patterns.
A typical audit of a backend with “comprehensive AI-generated tests” turns up a handful of issues serious enough to have caused a production outage or data corruption. The coverage metrics always look good. The coverage itself has systematic blind spots.
The practical path forward
Use LLMs to generate unit tests – they’re excellent at this and save significant time. But recognize what they don’t test:
| But recognize what they don’t test |
|---|
| Bring in QA expertise for integration test design |
| Simulate realistic concurrency and load in staging environments |
| Test data migration paths explicitly before deploying schema changes |
| Review auth and permission logic with an adversarial mindset |
| Monitor production for the failure modes that weren’t tested |
AI assistants are powerful tools for generating boilerplate tests. They’re not replacements for systematic QA thinking about how systems fail in production.
The question isn’t “Can AI write tests?” It’s “Can AI anticipate the failure modes that matter?” Based on our audits, the answer is: not yet, and not without human expertise identifying the gaps.
Related reading
- Why AI-generated code fails in production – The 7 failure patterns we see repeatedly in production
- Why validation is not the same as testing – Understanding the critical difference between verification and validation
- Why dev-owned testing is not independent QA – Why organizational independence matters for quality assurance
Built by BetterQA – software testing experts who find what automated tools miss.
Need help with software testing?
BetterQA provides independent QA services across manual testing, automation, security audits, and performance testing. ISO 27001, 9001, 14001 and 13485 certified.
