A benchmark built on a methodology from RMIT University has scored 16 leading AI models on the security of the code they write. Every one of them left holes, and the holes repeat. Codebases averaged 15 confirmed vulnerabilities each, 4.3 of them severe.
For the growing number of Australian teams shipping AI-assisted code, the useful finding is not that models make mistakes. It is that each model makes the same mistakes over and over, which means the risk can be predicted and managed rather than discovered in production.
What the benchmark measured
The SCW AI Trust Index asked each model to generate identical applications across 11 frameworks, repeating every combination ten times. Three independent static analysis tools scanned the output, and findings were verified to strip out false positives. That produced 1,760 complete codebases.
The approach started as a study with RMIT University covering six models, which Secure Code Warrior then extended to 16. The line-up includes Claude Sonnet 5, Claude Opus 4.8, Claude Fable 5, GPT 5.3 Codex, Gemini 3.1 Pro, Qwen3 Coder and Devstral 2, so it spans the tools most development teams are actually using, as Technology Decisions reported.
The same flaw, 8,543 times
Across the 1,760 codebases the researchers catalogued 86 distinct vulnerability types. The single most frequent was CWE-532, sensitive information written into log files, which turned up 8,543 times. Behind it came injection vulnerabilities, insecure design and broken access control.
“Every AI model we tested leaves a predictable, repeatable pattern of security gaps and weaknesses,” said Secure Code Warrior chief executive Pieter Danhieux. That predictability is the part worth acting on. A repeatable flaw is one a review process can be pointed at.
Claude leads, but nobody passes
Claude Sonnet 5 scored highest overall at 80.4 out of 100 on the AI Trust Index, which the authors describe as a living benchmark rather than a one-off report. Even so, top of the table still means shipping vulnerabilities by default.
The ranking also shifts by framework. The model that writes the safest code in one stack is not necessarily the safest in another, so a single organisation-wide default may be the wrong call. Teams already weighing these tools will find the performance side of that comparison in our breakdown of Claude Code versus Codex.

Why Australian businesses should care
The most common weakness is the one with the clearest local consequence. Sensitive information sitting in log files is exactly the exposure the Privacy Act 1988 makes an organisation answerable for, and the kind that turns into a notifiable data breach when logs are shared with a vendor or left on a misconfigured store. It is also close to invisible in code review, because the application works perfectly.
Australian teams have adopted AI coding assistants faster than they have adopted any process for checking their output. The research suggests the two need to move together: benchmark the specific models in use, put guardrails on the vulnerability classes those models are known to produce, and train reviewers to look for them. Where that turns into a procurement question rather than a developer one, it belongs in AI strategy work.
Model choice is a security control
The finding that should reset assumptions is that a more expensive model does not reliably produce safer code. Cost and popularity are not proxies for security, which means the choice has to be measured rather than assumed, and that applies to the assistant wrapped around the model as much as the model itself. We covered the tooling side of that in our comparison of Cursor and GitHub Copilot.
Picking a coding assistant has quietly become a security decision as much as a productivity one. We broke down the full 16 model table, the framework level winners and what to do about it in our guide to the most secure AI coding assistant. The teams that come out of this well will be the ones treating it that way before an audit does it for them.
