Where Cloudflare’s free security skill stops, and what a BetterQA audit adds

We ran Cloudflare's free security skill and our scanner blind on public apps. Where the free tool stops, and what a paid audit adds.
Benchmark

Where Cloudflare's free security skill stops, and what a BetterQA audit adds

For founders, CTOs and engineering managers who pay for a security review, or are about to.

Cloudflare gives away an open-source security-audit skill, and it is good at what it does: an agent reads your source code and hunts for bugs. We ran it and our scanner, blind, on the source code of two public open-source apps. Both times the free skill found a little more of the known published bugs than we did. At pure code reading the free tool is about as good as ours, and we have to accept that. Where it stops is your running app. Its own instructions say it must not contact your deployed app or use real accounts, so it never signs in as your users, never sees your servers and never checks what your users actually receive. That is the part a paid audit adds. Below we set out where the free skill stops, what our audit adds, and what we changed because of the test.

A security report can fail in three ways: it lists bugs that are not real, it misses bugs that are, or it says "done" when part of the check never ran. We used those three as the yardstick for a second blind test on Snipe-IT 8.6.3, an app we had not tuned on and had never run either tool against. The free skill found 6 of 13 published bugs in a typical run and our scanner found 5; the retest below reports both tools on the first two, what we saw on the third, and where our fixes made no measurable difference.

  • Security audit
  • Free tool vs paid audit
  • Blind retest
  • Vendor claims
Four BetterQA colleagues in a meeting at a table with laptops, seen through the glass wall of the meeting room in the Cluj-Napoca office
What you get that the free skill does not give you
Live apptested from outside, as your users get it. The free skill must not contact deployed systems
Every rolesigned in as each user role you give us. The free skill must not use real accounts
Every criticalfinding reproduced by a person, filed in BugBoard, and retested after your fixes, in the price
Fixed priceagreed before testing starts, model usage included. The free skill's runs are billed to your own model account
The buyer's problem

What happens after the report

Finding a bug is one step. Before a security review has paid for itself, someone has to decide which findings are real, reproduce them, put them where developers will see them, and check afterwards that the fix holds. They also have to know which parts of the product nobody looked at.

A free tool covers some of those steps well. The rest lands on your own engineers. Whether that is a good trade depends on how much engineer time you have to spare.

Where the free tool stops

Five places the free skill stops, and what we do instead

Cards marked "By design" come from the skill's own README and instructions. "Not in its docs" means its documentation describes no such step. "Seen in our test" comes from the blind test further down. "Who signs it" is about who stands behind the report. We have not tried to make the skill do more than its docs describe.

By design

It never tests your running app

Its instructions say it must not contact deployed systems, shared infrastructure or real accounts. It works from your source code and a local sandbox. So whatever exists only once you deploy is out of its reach: your server and TLS setup, the headers your users receive, the access rules set in your hosting or database console, the JavaScript bundle as deployed, and the app file you publish to the store. Where a control lives outside the code, it lists it for you to check yourself. We do: our toolkit tests the running app from outside, including Nuclei, TLS checks, files the site exposes, secrets and source maps in the shipped JavaScript, and what a Supabase-style backend lets an anonymous visitor read. We also analyse the built Android or iOS app file. Then an engineer works through the product by hand.

By design

It never signs in as your users

It works out who may do what by reading the code, so it can spot a permission check missing from the source. It may confirm it with a local test using dummy accounts. It does not log in to your deployed app to check whether that check is enforced there, or what a second account can actually read there. We do: we sign in to your test environment as each role you give us and try what each role should not reach. On a test environment with two or more logins, we also check whether personal data created by one account shows up on another account's pages. That check is new this month and does not cover email yet.

Who signs it

The report comes from your own team

The skill runs inside your own coding agent. When your own engineers run it, its report is your team checking its own work. When an auditor, an investor or an enterprise customer asks for a security report, they usually want it from someone independent of the developers. We do: the report comes from BetterQA's engineers, who did not write your code, and we never sign off code we wrote.

Seen in our test

Free to download, not free to run

In our test, one run of the skill on its default setting started 65 to 153 helper agents, paid for from your own model account. Its README advises more than one run, because repeated runs find more, and in our test that held for our scanner too. The skill can be capped or run in a lighter, bounded mode, at the cost of coverage. We did not measure tokens or money. We do: the model usage of our audit sits inside the fixed price, and an engineer then tests by hand where scanners are weak.

Not in its docs

No person reproduces, tickets or retests for you

The skill writes report files: REPORT.md, FINDINGS-DETAIL.md and findings.json, with safe reproduction steps. Its agents try to disprove each finding, which is good practice and we do the same. A repeat run re-checks code that changed. Starting it, reading it, reproducing by hand and ticketing are on your team. We do: an engineer reproduces every critical finding and scores it with CVSS. Findings go into BugBoard with steps and evidence as they are confirmed, smaller findings that chain into a serious one are reported as one chain, and the retest of your fixes is in the price.

Side by side

What a free tool gives you, and what a paid audit adds

The free tool here is Cloudflare's open-source security-audit skill. Its own README calls it "a coding-agent skill that turns your agent into a security auditor". Every statement about it below comes from its README and instruction files at commit c1c8a8c1, read 15 November 2026, and we say where we have not checked.

Left column: Cloudflare's README and instructions at github.com/cloudflare/security-audit-skill, read 15 November 2026. Right column: how BetterQA runs a security audit today.
What you needFree tool: Cloudflare's skillBetterQA security audit
PriceFree, MIT licence. It runs inside your own coding agent, so the model usage is paid for from your own model account or plan.A written fixed price for your scope, agreed after a scoping call and before any testing starts.
Who does the workYou start it from your coding agent, pointed at your codebase. Your engineer reads what comes out.Our engineers run our toolkit, then test by hand where scanners are weak: permissions, business logic and findings that chain together.
Logged in as each roleWorks from the source code and a local sandbox. It maps what each user type may do from the code, and its instructions say it must not contact deployed systems or use real accounts. So it does not sign in to your running app.We sign in as each role you give us and check whether one can reach what only another should. Changes are made only on a test environment you confirm.
Data-leak checkHas hunting classes for tenant isolation and data lifecycle, worked from the source.On a test environment, with two or more logins, the audit also checks whether personal data created by one account shows up on another account's pages. It does not yet check email. We added this check this month.
Mobile appsHas hunting classes for native app and webview code, worked from the source.We also analyse the built app file (Android APK or AAB, iOS IPA) for what it stores, where it is set up to send data, and what it exposes. That part is static analysis of the file.
Is each finding checkedYes, by design: fresh agents try to disprove each candidate, and unproven leads stay as "needs_validation". The README says the agent that checks a finding is never the one that found it.The same idea, and on top an engineer reproduces every critical finding before it reaches your report.
TicketsWrites report files: REPORT.md, FINDINGS-DETAIL.md and a findings.json. Getting them into your tracker is up to you.Findings go into BugBoard as they are confirmed, with steps, evidence and severity.
Retest after your fixesRun it again after your fixes: its README says a repeat run re-checks changed code against the previous run. Starting it, and reading the result, is on your team.The retest of your fixes is included in the price.
What was not testedCovered. A coverage ledger and a NEEDS-VALIDATION.md record what stayed unresolved.Covered. The report's coverage statement lists every unit not examined and every access check that was not reached.

Both columns record what was not looked at. That is not something only we do.

What free costs

What running the free tool costs

The skill costs nothing. Running it does. It needs a coding agent with a model that supports parallel helper agents, and those sessions are paid for from your own model account or plan. Then someone on your team triages the output.

In our test, on its default setting, one run of Cloudflare's skill started 65 to 153 helper agents. You pay for those in your own model account. The skill lets you cap that number or pick a lighter, bounded mode, at the cost of coverage. We did not measure tokens or money. A paid audit's usage sits inside our price. Cloudflare's own README adds that in its test runs a single run found roughly half of the vulnerabilities that repeated runs found in total, so plan for more than one run. Our scanner is no different: in our test, one of our runs also found well short of what our three runs found together. We have no price to quote for this; it depends on your codebase and your model contract.

The blind test

The blind test: on a public app, the free skill was a little ahead

We ran our scanner and Cloudflare's skill on the source code of the same public open-source app, Flowise 3.0.13, with the same model for both. Every report was graded blind against 8 of the app's published security advisories, which we chose before the test. Cloudflare's skill finished 2 of 5 attempts; our scanner needed 4 attempts to get three finished runs.

Cloudflare's skill found 4 to 5 of the 8 per run (across its 2 finished runs) and our scanner found 2 to 4. The ranges overlap, and the free skill's is higher. Two of the 8 bugs were missed by every run, and a third was never fully found.

The model provider's safety filter stopped 3 of the free skill's 5 attempts during setup, and those runs did not finish. It hit our runs too. In one, four of our agents were stopped, one of them after it had saved its work; two were re-run inside the run and it finished. In another, a stopped agent's work was lost and we count that run as not finished. An unscored pilot run was repeated because the scanners were not installed on the worker machine.

We also ran both tools on two of our own products. We do not count those results here: we wrote those bug lists ourselves, so they cannot show which tool is better. They did show us our own mistakes, listed below.

Some of the setup favours us, and you should weigh it. We wrote the harness that runs our method. We tuned our method on an earlier codebase of ours. Our graders were automated, ran in our own harness using our own reviewer instructions, and came from the same model family as both tools. Flowise's advisories were public before the model's training cutoff, so both tools may have recalled some of them; that cuts both ways.

What the test says for a buyer: on reading the source code of a public app, the free tool was a little ahead of our scanner. The rest of a paid audit, testing your running app as each role and having a person reproduce what matters, was not part of this test, because the test was source-only and the free skill does not do it.

What we learned

Four mistakes of ours, and what we changed

The test showed us four mistakes. We wrote a fix for each before round two; the two new checks in the second row were merged on 15 November. Round two, below, was our chance to see them work. None of them moved a number we could measure.

  • Mistake 1We spotted a bug pattern and did not check where else it repeatsChangeIn our source review, every confirmed high or critical finding now triggers a hunt for its siblings. A run that skips this hunt cannot be marked complete.
  • Mistake 2Both tools stopped at the access check on a second public routeChangeThat route checks that a flow is marked public, then returns the stored data unfiltered, secrets included. All five runs, both tools, missed it. An access check decides who may call a route, not what it returns. We wrote two new checklist classes: what each public route returns, and which call sites bypass a safety wrapper.
  • Mistake 3A run said "done" with areas unreadChangeOne run finished in 16 minutes, skipped several of its own phases and still reported complete. Our source review now lists the entry points it can recognise, each examined or marked as not examined with a reason, and a completion check refuses a thin run. This is catching up: Cloudflare's skill already had validators and a coverage ledger.
  • Mistake 4Real findings were padded with minor adviceChangeOn one of our own codebases our reports carried 12 "good practice" items per run (range 8 to 13), against 1 for Cloudflare's skill (range 0 to 2). We added an appendix for good-practice advice, and a check of each candidate bug against the source, an idea that came from Cloudflare's skill. Round two below shows good-practice items still landing in the bug list.
Round two

The retest on the held-out app

On 16 November 2026 we ran the improved scanner against Cloudflare's skill again, on Snipe-IT 8.6.3, a public open-source asset-management app. Same rules as round one: source code only, same model for both, blind grading, and the scoring rule fixed before grading. Each tool ran 3 times. The answer key was 13 of the app's published security advisories, chosen before any scan. The app's maintainers published them, not us and not Cloudflare, after the model's training cutoff, so neither tool could have seen them in training. Both new checks (public pages, bypassed wrappers) were in the build we tested.

14 vs 3good-practice items per typical run in the bug list, which are not exploitable bugs: ours vs Cloudflare's
0, maybe 1false bugs across all six runs. The possible one was in one of ours
3 of 3 eachruns that finished at the first attempt, both tools
30 to 40 vs 99 to 153helper agents per run: ours vs Cloudflare's, on its default setting

Nothing we fixed moved a number we could measure. Every run, ours and theirs, finished at the first attempt. That does not show a fix working: we picked an app less likely to trip the model provider's safety filter, and it still stopped three of our agents in one run. The good-practice count did not move either: our bug lists still carried far more of those items than Cloudflare's, so we failed our own pass bar on that line. We are changing the scanner so that a bug must name who gains what. From one app we cannot tell whether the two new checks or the sibling hunt found anything they would otherwise have missed. Limits: it is one app and 3 runs per tool. One bug on the key was graded unevenly between runs, and the result is the same whichever way it is read. In one of our runs, two leads from agents the safety filter had stopped went into the findings without the usual re-check.

This comparison is not finished. We are preparing a further fair test on two more public open-source apps, comparing our full audit (the source code plus the running app, signed in as each user role) with the free skill. We will publish the results whichever way they go.

Who should use which

When the free tool is the right choice

If you have an engineer with time to start the runs, read the output and decide what is real, Cloudflare's skill is a good place to start, and it costs nothing to try. Run it more than once, as its own README advises.

A paid audit makes sense when you want someone else to own the whole job: testing as each of your user roles, a check on whether personal data created by one account shows up on another's pages, an engineer who has reproduced the critical findings, tickets in your tracker, and a retest once you have fixed things. It also makes sense when an auditor or a customer asks for a report from someone independent of your developers. You can also run the free tool in your own pipeline and still buy an audit on top.

Limits

Limits of this test, and questions to ask any vendor

Two public apps are still only two. In round one, two of the 8 public bugs were missed by every run. We have not checked every claim in Cloudflare's README against a run, and Cloudflare's skill is built not to touch a running app, so the role-login part of our audit was not compared. The data-leak check described above is new and we make no claim about its results. We cannot call a scan clean on the strength of a single test, ours or anyone's, and the grader bias stated above applies to round two as well.

  • AskHow many runs, and was the test blind?WhyOne run on a target the vendor chose proves little.
  • AskWhat did it not check?WhyA report that names its blind spots is easier to trust.
  • AskWho reproduced the findings, and who checks the fixes?WhyA list of candidates is not a list of confirmed bugs.
What this means for you: Free tools are good at finding candidates. Decide who will do the rest of the job, then choose.

Talk to us about a security audit

A security audit from BetterQA comes with a fixed price for your scope, findings in BugBoard, and a retest of your fixes. See how an engagement works, or talk to us.

Talk to our team

Automate your security testing

Our AI Security Toolkit runs 20 checks we wrote ourselves alongside 15 open-source scanners to find vulnerabilities other scanners miss. SAST, DAST, SCA, secrets detection - unified in one scan.

Explore AI Security Toolkit
Share the Post: