We built our own security auditor. Here are our top 10 mistakes.

The ten mistakes

1. Our coverage table claimed tests we did not have. We read the source for every row marked "covered" in our own coverage table. 15 of 78 were wrong: 11 had no tool behind them at all, and 4 credited the wrong tool. We took the number off our website. A test now fails if a claim names a tool that does not exist, and we built the missing checks.
2. AI agents invented test IDs. Asked to tag each test with its standard identifier, two models produced five IDs, and three did not exist in the standard. One model used the same made-up ID for two different tests. The agents now get the real list and are told when an ID is not on it.
3. Checks that never ran were reported as clean. Eleven of our own checks were called with an option most of them rejected. They exited with an error on the wrong output stream, the runner saw an empty error message, and read it as "0 findings". A check that did not run now says "did not run", with the reason.
4. One check threw away every finding. It matched on a status value its own script never writes, so even when it ran and found something, the report stayed empty.
5. A failed login counted as logged in. If a login page set any cookie, such as an anti-forgery token, the login was treated as a success, and the report then claimed an authenticated scan. A login now needs positive evidence before it counts.
6. Healthy scans were killed after 90 seconds. Two scan types never reported activity to the watchdog, so a scan that was still working was declared frozen and failed. Every scan type now reports its activity.
7. Our scanner could be pointed at internal systems. Hosted scans checked an address once, by name, and then let the HTTP client look it up again and follow redirects. A public name that resolved to a private address, or a login page that redirected inside, got through. Every request is now judged on the address it actually connects to, at every hop.
8. A recon mode passed credentials to the AI agent. The mode that reads captured browser traffic, often from production, returned request bodies cut to 120 characters. A login body fits in 120 characters. Now no value leaves that code at all, only field names and types.
9. Silence was reported as readiness. A compliance readiness report printed "Ready" for any area with zero findings, so a crashed agent and a clean system looked the same. The matching export gave a perfect score with zero controls assessed. An area with nothing recorded is now "not assessed".
10. The change merged and the deploy broke. A change added a private dependency that only developer laptops could download. The pull request had no automated checks to catch it, the production build failed, and the old version quietly stayed live. The dependency now ships inside the package, and we ask every pull request which checks it actually has.

The pattern behind the ten

Almost none of these crash. Most are a tool reporting more certainty than it had: covered when nothing covered it, clean when nothing ran, logged in when it was not, ready when nobody looked.

That is the failure that matters in a security report, because nobody re-checks a clean result. So for every check we now ask the same question: what would this report look like if the check had silently done nothing? If the answer is "the same", the check is not finished.

Why we still trust it, and why a person still reads every report

On a public vulnerable-by-design benchmark, the toolkit produced 27 findings and 6 attack chains where the other tools we compared produced none. It also caught several of the "reported success while doing nothing" failures above in its own real scans, which is how we found them.

But a tool that can be confidently wrong needs a reader. Every finding our pipelines produce is reviewed by an engineer before it reaches a client.