Ten ways an automated security scan can make your system look clean

A clean security report can mean the scan did less than it claimed. Ten cases from our own scanner, and what to check in any scanner you buy.
A BQA mug in front of two laptops while a hand types

Ten ways an automated security scan can make your system look clean

The worst failure of a security scan is not a crash. It is a report that says "clean" when the scan did less than it claimed. Nobody re-checks a clean result. We found each of the ten cases below in our own AI Security Toolkit, which runs 15 open-source scanners and 20 checks we wrote, and each is fixed. They are also what to look for in any scanner you buy.

Ten ways a scan looks clean when it was not

  • The coverage table claims tests that do not exist. We read the source behind every row marked covered. 15 of 78 were wrong: 11 had no tool behind them and 4 credited the wrong tool. Ask to see the tool behind each claimed check.
  • AI agents invent test identifiers. Asked to tag tests with their standard identifier, two models produced five IDs and three did not exist. Ask how identifiers are validated.
  • Checks that never ran are reported as clean. Eleven of our checks were called with an option most of them rejected, the runner read the empty error as "0 findings". Ask what the report says when a check did not run.
  • A check drops every finding. One matched a status value its own script never writes, so the report stayed empty even when it found something.
  • A failed login counts as logged in. A login page that set any cookie, such as an anti-forgery token, was treated as a success, and the report then claimed an authenticated scan. Ask what proves the scan was logged in.
  • Healthy scans are killed as frozen. Two scan types never reported activity, so a scan still working was failed after 90 seconds. Ask what a timed-out scan looks like in the report.
  • The scanner can be pointed at internal systems. A hosted scan checked an address once, by name, then let the client look it up again and follow redirects. Ask how the tool restricts where it connects.
  • Recon mode leaks credentials to the AI agent. Request bodies cut to 120 characters can still hold a whole login body. Ask what data from your traffic reaches an AI model.
  • Silence is reported as readiness. A readiness report printed Ready for any area with zero findings, so a crashed agent and a clean system looked the same. Ask for "not assessed" as a state.
  • The release process breaks quietly. A change added a dependency only developer laptops could download, the pull request had no automated checks, and the old version stayed live. Ask which checks gate the tool's own releases.

The question behind all ten

For every check, ask what the report would look like if the check had silently done nothing. If the answer is "the same", the check is not finished.

Why a person still reads every report

On a public vulnerable-by-design benchmark, the toolkit produced 27 findings and 6 attack chains where the other tools we compared produced none. It also caught several of the failures above in its own real scans, which is how we found them. But a tool that can be confidently wrong needs a reader, so an engineer reviews every finding before it reaches a client.

Want a security scan that tells you what it did not check? See our security testing service, run with an engineer reading every result. Ask us about a complimentary scan.

Automate your security testing

Our AI Security Toolkit runs 20 checks we wrote ourselves alongside 15 open-source scanners to find vulnerabilities other scanners miss. SAST, DAST, SCA, secrets detection - unified in one scan.

Explore AI Security Toolkit
Share the Post: