Every team I speak to in Melbourne has picked an AI coding assistant on one of two grounds: how fast it gets to a working change, or how much it costs per seat. I have never once been asked which one writes the safest code. That question now has an answer, and it is uncomfortable enough that it should change how a few of those choices get made.
A benchmark built on a methodology from RMIT University has scored 16 models on the security of the code they produce. I have spent the past week reading the ranking against what I see in client repositories, and the headline is not that one model is unsafe. It is that all 16 leave holes, the holes repeat per model, and the ranking covers models rather than the assistants most teams actually buy, so Cursor and GitHub Copilot are not in it at all.
This guide walks through the full ranking, the vulnerability that turns up more than any other, why paying more buys you nothing on this axis, and what Australian teams should actually do about it under the Privacy Act. The charts are the part I wish every vendor published next to their benchmark scores. If you read only the quick pick, you already have the structural answer.
What this AI code security benchmark measures
Most coding benchmarks ask whether a model can make the tests pass. This one asks something different: when the code works, is it safe? The SCW AI Trust Index asked each of 16 models to generate identical applications across 11 frameworks, repeating every combination ten times, then ran three independent static analysis tools over the output and verified the findings to strip false positives.
That produced 1,760 complete codebases. The methodology started as a study with RMIT University covering six models, which Secure Code Warrior extended to 16. The repetition matters more than the headline number: running the same prompt ten times is how you separate a model having a bad day from a model having a bad habit.
It is worth saying what this does not measure. A high score here means the model tends to produce fewer verified vulnerabilities in generated code. It says nothing about how good the model is at reasoning across a large repository or how fast it gets to a diff, which is the axis we weighed in our comparison of Claude Code versus Codex. Security is one dimension of a tool choice, not the whole of it.
The full ranking of 16 AI coding assistants
Claude Sonnet 5 tops the table on 80.4 out of 100. The spread underneath it is the story: the gap between first and last is nearly 59 points, which is far wider than the gap you see between the same models on a standard coding benchmark. On capability these tools are converging. On security they are not.
Code security scores on two benchmarks
Two independent benchmarks. They use different tasks and different scales, so the two blocks are shown separately and the numbers are not comparable across them.
SCW AI Trust Index
Score out of 100, higher is better. The eight highest scoring of the models we still track; the benchmark tested 16. Source: SCW AI Trust Index.
Endor Labs benchmark, includes GPT 5.6
Share of 200 tasks where the code was both functionally correct and secure, higher is better. A much harder bar: the leader clears fewer than 3 in 10. Source: Endor Labs, 17 July 2026.
Below that top group the numbers fall away fast, and the pattern is not random. Four Claude models sit in the top nine, and Claude Sonnet 4.6 sits thirteenth, inside the bottom four. That is the same vendor at both ends of the table, so vendor loyalty is a poor proxy for safety here. If you want the capability view of the Claude line-up rather than the security one, we cover it in which Claude model to use.
| # | Model | Trust Index score |
|---|---|---|
| 1 | Claude Sonnet 5 | 80.4 |
| 2 | Claude Fable 5 (now legacy) | 76.4 |
| 4 | Claude Opus 4.8 | 74.5 |
| 6 | Gemini 3.1 Pro | 66.6 |
| 7 | GPT 5.5 | 66.5 |
| 9 | Claude Sonnet 4.5 | 65.2 |
| 10 | Gemini 3.5 Flash | 59.1 |
| 11 | Devstral 2 | 53.1 |
| 12 | Claude Haiku 4.5 | 50.7 |
| 13 | Claude Sonnet 4.6 | 45.5 |
| 15 | Qwen3 Coder | 32.7 |
| 16 | GPT 5 Mini | 21.6 |
The benchmark ranked 16 models; this table keeps its positions and leaves out four older models we no longer track. Source: SCW AI Trust Index, July 2026.
Where Codex and GPT 5.6 sit on AI code security
Two questions come up the moment a reader scans that table. Codex is in the Endor Labs benchmark below, run with GPT 5.5 and GPT 5.6 Sol. GPT 5.6 is not in the Trust Index, and neither are its Sol, Terra and Luna variants, GPT 6 Astra or the GPT 6 Sol and Luna pair released on 22 September, so there is no security score there for the models many teams are running right now.
That gap is worth naming rather than glossing over. If you are on GPT 5.6, the closest evidence you have is the spread of the OpenAI models that were tested, and that spread is wide. For GPT 6 Astra this study has no score yet; the closest evidence is vendor cyber tests for Astra and Gemini 4 Carbon.
The OpenAI models we still track, on both benchmarks
Separate scales, shown apart. GPT 5.6 has no Trust Index score; its only measured result comes from Endor Labs. GPT 6 Sol, GPT 6 Luna and GPT 6.1 Sol are on neither benchmark yet.
SCW AI Trust Index
Score out of 100. Source: SCW AI Trust Index.
Endor Labs benchmark
Share of tasks correct and secure. Source: Endor Labs, 17 July 2026.
Between the two OpenAI models still on the Trust Index, GPT 5.5 on 66.5 and GPT 5 Mini on 21.6, the gap is tier rather than version. The three variants of GPT 5.6 differ in reasoning depth and speed rather than in safety posture, and we cover those differences in our guide to GPT 5.6 Sol, Terra and Luna.
There is a second benchmark that does cover it. The Endor Labs AI Coding Agent Security Benchmark runs 200 tasks drawn from 108 open source Python projects across 77 CWE classes, and scores a task only when the generated code passes both the functional tests and the security tests. On that measure GPT 5.6 Sol running in Codex scored 23.5 per cent on 17 July 2026.
Two things in the Endor Labs results are worth pulling out. GPT 5.6 Sol improves on GPT 5.5 inside Codex, but only by 1.1 points, and GPT 5.5 driven through Cursor beats both of them. The harness wrapped around the model moves the result about as much as the model does, which is the same lesson the framework table teaches from a different angle.
The practical read is narrow. Codex has a measured score on the Endor Labs benchmark, with GPT 5.5 and with GPT 5.6 Sol. GPT 5.6 has one measured data point, from a single harness and a single variant, so treat Sol as thinly measured and Terra and Luna as unmeasured until someone scores them.
Every AI coding assistant ships vulnerabilities
Top of the table still means shipping holes. Across the 1,760 codebases the researchers counted an average of 15 confirmed vulnerabilities per codebase, 4.3 of them severe. That is the average across all models, including the leaders. Scoring 80.4 does not mean Claude Sonnet 5 writes safe code. It means it writes less unsafe code than the alternatives. Vendor claims do not change that: Google says Gemini 4 Carbon patches flaws better than Opus 5.5.
The word the researchers keep using is predictable. “Every AI model we tested leaves a predictable, repeatable pattern of security gaps and weaknesses,” said Secure Code Warrior chief executive Pieter Danhieux. In my experience that is the most useful sentence in the whole study, because a repeatable flaw is a flaw you can build a control around. A random one is not.
This reframes the buying question. Instead of asking which assistant is safe, ask how much review budget each one costs you, and whether your team has that budget. That is a question about process, not about tooling, and it is the one most of the teams we talk to at Aivy have not yet worked through.
CWE-532, the flaw in AI generated code that repeats
One weakness dominates everything else. CWE-532, the insertion of sensitive information into log files, was confirmed 8,543 times across the study, nearly three times more often than the next most common issue. Cross site scripting came second at 2,949, and hard coded credentials third at 1,348.
Most common vulnerabilities in AI generated code
Verified true positives across 1,760 codebases. Source: SCW AI Trust Index.
What makes CWE-532 so awkward is that it is invisible in normal review. The application works. The tests pass. Nothing looks wrong until someone reads a log file and finds a token, an email address or a full customer record sitting in plain text. It is a bug you find during an incident, not during a pull request.
For Australian teams that specific weakness carries a specific consequence, because logs holding personal information are personal information under the Privacy Act 1988. We unpack that obligation in our Privacy Act and AI compliance playbook.
Anthropic added Claude Opus 5.5 to its range on 22 September 2026, at $5.62 and $28.08 per million tokens on the RBA rate of that day. Its model documentation now suggests starting there for most workloads and moving up to Claude Fable 5.1 only for demanding reasoning and long running agentic work, and Claude Opus 5 moves to the legacy list while staying available.
OpenAI added GPT 6 Sol and GPT 6 Luna on 22 September 2026, at $2.81 and $14.04 per million tokens for Sol and $0.14 and $0.70 for Luna, on the RBA rate of that day. OpenAI positions Sol for complex coding and agentic workflows and Luna as its most efficient model for focused, high volume tasks. There is no GPT 6 Terra. GPT 6.1 Sol, an upgrade released on 29 September at the same price, has no security score on either benchmark yet.
Google announced Gemini 4 Argon on 30 September 2026 with security results of its own: it says Argon ties for first on CWE bench v1 at 68 per cent and leads Gray Swan’s indirect prompt injection benchmark. It is going first to trusted cyber defenders, who get it without its cybersecurity safeguards. Those are Google’s figures; neither benchmark here has scored Argon, and nobody outside that programme can use it yet.
A pricier AI coding assistant is not a safer one
The finding that should reset a few procurement assumptions is that price predicts nothing here. The most expensive model tested, Claude Fable 5 at $248.41 per run, came second, while the budget tiers cluster at the bottom of the table.
That $248.41 figure is a Claude Fable 5 run, and the new release barely moves it. Fable 5.1 charges the same base rate as Fable 5, $14.04 per million input tokens and $70.20 per million output at the RBA rate of 22 September 2026, double what Claude Opus 5 costs and two and a half times Claude Opus 5.5. Only cache reads fall, by about 75 per cent to $0.35, so the 25 to 45 per cent saving Anthropic advertises turns up only when your workload reuses cache.
The pattern that does hold is tier, not vendor and not price. The budget tiers cluster at the bottom: Claude Haiku 4.5 on 50.7 and GPT 5 Mini on 21.6. If your team has been routing bulk work to a mini or flash tier to save money, that saving has a security cost attached that nobody put on the invoice. The capability side of that same trade off is in our comparison of Cursor, Windsurf and Devin Desktop.
No model writes the most secure code in every framework
The overall ranking hides something important. The leader changes depending on what you are writing. Claude Sonnet 5 tops the aggregate table, but it does not top a single one of the five frameworks the researchers broke out.
| Framework | Safest model for it |
|---|---|
| Java Spring | Claude Sonnet 4.5 |
| Python Django | Claude Opus 4.8 |
| C# (.NET) | GPT 5.5 |
| C | Claude Fable 5 |
Framework level leaders. The Java Enterprise API leader is an older model we no longer track, so it is left out. Source: SCW AI Trust Index.
Claude Sonnet 4.5 sits ninth overall and yet leads on Java Spring. GPT 5.5 sits seventh overall and leads on C# dot NET. If your stack is one of these, the aggregate ranking is actively misleading and the framework row is the number you want. This is the same reason a single organisation wide default tool rarely survives contact with a mixed codebase, a point that also came up when we compared Cursor and Claude Code.
A newer model version does not mean safer code
The most counterintuitive line in the data is inside one model family. Claude Sonnet 4.5 scores 65.2. Claude Sonnet 4.6, the later release, scores 45.5. Then Claude Sonnet 5 jumps to 80.4. Security did not improve monotonically with version number, it went down and then up again. Claude Sonnet 5.5, released on 28 September 2026, has no score yet, so choose it on capability and cost, as in our Sonnet 5.5 and Opus 5.5 comparison, then test its code yourself.
Claude Sonnet security score by release
Higher is better, axis 0 to 100. Source: SCW AI Trust Index.
Claude Sonnet 5.5, released on 28 September 2026, has no score on this index yet.
The practical lesson is that a model upgrade is not automatically a security upgrade, and nobody is going to tell you when it goes the other way. Release notes talk about capability. If security matters to your build, the version bump is worth re testing rather than assuming, which is the same discipline we apply when a new frontier model lands, as in Claude Opus 5 versus Fable 5.
Why AI code security lands harder in Australia
The single most common flaw in this study is personal information written into log files, and that is not a neutral technical detail here. Under the Privacy Act 1988, personal information in your logs is personal information you hold, with everything that follows: the security obligation, and the notifiable data breach regime if those logs leak or are handed to a vendor.
What makes it worse is where logs travel. They get shipped to observability platforms, attached to support tickets and copied into buckets, often outside Australia. So a CWE-532 in a generated logging call quietly becomes a data residency question as well as a security one, which is the territory we mapped in which AI tools keep your data in Australia.
Share of the three most common flaw types
Percentage of the 12,840 findings in the top three classes, axis 0 to 100. Derived from SCW AI Trust Index counts.
Australian teams have adopted these assistants faster than they have adopted any process for checking their output. I have not yet met a mid market team here with a review step designed specifically for generated code, and the study suggests that is the gap that matters more than the tool choice. We wrote up the study itself in more detail in our news coverage of the ranking.
How to pick the most secure AI coding assistant
The ranking is most useful as a routing rule rather than a shopping list. Frontier tier for anything touching customer data, credentials or payments. Budget tier for scaffolding, tests, throwaway prototypes and internal tooling where the blast radius is small. That single split captures most of the value in the table. A routing rule only holds if someone can look it up, which is why the approved tool list in an AI policy is where a split like this belongs.
After that, three things do more for you than swapping models. Point a static analysis tool at generated code specifically, not just at hand written code. Add CWE-532 to your review checklist, since it is the one that will not announce itself. And re test after a model version change rather than trusting the version number.
Pick per stack using the framework table, not per vendor. Route sensitive work to a frontier tier assistant. Put a static analysis pass in your pipeline that runs against AI generated code, and treat logging calls as the highest risk output the model produces. If you have moved to Claude Fable 5.1, Claude Opus 5.5 or Claude Sonnet 5.5, re test rather than assume they inherited an older result: only Endor Labs has scored the first two, at 37.4 and 33.5 per cent, and neither benchmark has scored Sonnet 5.5.
If you want a second pair of eyes on how your team is using these tools, that is the kind of thing we work through in AI consulting, or you can just book a discovery call and talk it through.
Frequently asked questions about AI coding assistant security
Which AI coding assistant writes the most secure code?
Claude Sonnet 5 ranks first on the SCW AI Trust Index with 80.4 out of 100, ahead of Claude Fable 5 on 76.4 and Claude Opus 4.8 on 74.5. No model scored clean, and the leader still produced verified vulnerabilities.
How many vulnerabilities does AI generated code typically contain?
The study found an average of 15 confirmed vulnerabilities per codebase, with 4.3 rated severe, across 1,760 codebases from 16 models. That average includes the top ranked models, so it is a floor rather than a worst case.
Is AI generated code safe to put into production?
Not without review. Every model tested produced verified vulnerabilities, so generated code typically needs the same static analysis and human review as any other contribution. The ranking tells you how much review to budget, not whether you can skip it.
Does a more expensive AI model write safer code?
No. The most expensive model in the study, Claude Fable 5 at $248.41 per run, ranked second, while the budget tiers sit at the bottom of the table. Price and security showed no reliable relationship in the results.
What is CWE-532 and why does it matter so much?
CWE-532 is the insertion of sensitive information into log files. It was confirmed 8,543 times in the study, roughly three times more often than the next issue, and it is difficult to catch in review because the application behaves normally.
Do Australian privacy rules apply to data in log files?
Personal information in logs is generally treated as personal information an organisation holds under the Privacy Act 1988, which brings security obligations and potential notifiable data breach duties. Check your own position with a qualified adviser, since specifics vary by organisation.
Is the best model the same for every programming language?
No. The framework level results show different leaders: Claude Sonnet 4.5 for Java Spring, Claude Opus 4.8 for Python Django, GPT 5.5 for C# dot NET and Claude Fable 5 for C. Pick per stack.
Are the cheaper mini and flash tiers safe enough for real work?
They typically sit at the bottom of the ranking, with GPT 5 Mini on 21.6 and Claude Haiku 4.5 on 50.7. They often suit scaffolding, tests and prototypes, but they are a poor default for code touching customer data or credentials.
Does upgrading to a newer model improve security?
Not always. Claude Sonnet 4.6 scored 45.5, below Claude Sonnet 4.5 on 65.2, before Sonnet 5 rose to 80.4. Security did not track the version number, so it is worth re testing after an upgrade rather than assuming improvement. Claude Sonnet 5.5, released on 28 September 2026, has no score yet, so test it before you switch.
Do I need a developer to act on this?
Not for the first step. Choosing which tier of assistant handles sensitive work is a policy decision. Adding static analysis against generated code and building a review checklist typically needs engineering input to implement well.
The bottom line on AI code security for Australian teams
The useful takeaway is not the name at the top of the table. It is that the spread between models is nearly 59 points on security while the spread on capability keeps narrowing, and that the flaws each model produces are repeatable enough to plan around. That makes tool choice a security control, which is a very different conversation from the productivity one most teams have had so far, and it only gets sharper as agentic tools write more code without a human watching each step.
A Melbourne team we work with had been routing bulk refactors to a budget tier assistant purely on cost, with no separate review path for generated code. Nothing had gone wrong. Reading the ranking, the thing that changed their mind was not the score but CWE-532: they went looking, and found tokens in their own application logs that no pull request had ever flagged.
If you are weighing this up for your own team, the honest starting point is an audit of what your assistants are already producing rather than a new vendor evaluation. We are Aivy, an AI and automation team working with Australian businesses, and for regulated work like professional services this is the kind of question we are happy to look at with you.
If you are weighing outside help, our guide on choosing an AI partner covers what to ask. Whatever you decide on tooling, the review step is the part that pays for itself.
Last updated: 1 October 2026. Scores reflect the SCW AI Trust Index as published in July 2026; the index is maintained as a living benchmark and rankings change as new models are added.
• OpenAI, Introducing GPT 6.1 Sol · the 29 September 2026 upgrade to GPT 6 Sol at 2 and 10 United States dollars per million tokens
• Google, Gemini 4 Argon announcement · the 30 September 2026 release to trusted cyber defenders only, without cybersecurity safeguards for them, and Google’s own results: tied first on CWE bench v1 at 68%, leader on Gray Swan’s indirect prompt injection benchmark (read 1 October 2026)
• Secure Code Warrior, AI Trust Index · full 16 model ranking (Claude Sonnet 5 80.4 to GPT 5 Mini 21.6), methodology (11 frameworks, 10 repetitions, 3 static analysis tools, 1,760 codebases), framework level leaders, CWE counts (532: 8,543; 79: 2,949; 798: 1,348), cost per run ($248.41 for Claude Fable 5)
• Technology Decisions · average of 15 confirmed vulnerabilities per codebase and 4.3 severe, 86 distinct CWE types, RMIT University’s role in the original methodology, Pieter Danhieux quote
• Endor Labs, AI Coding Agent Security Benchmark · GPT 5.6 Sol 23.5%, Claude Fable 5 29.0%, GPT 5.5 24.0% and 22.4%; 200 tasks across 108 open source Python projects and 77 CWE classes, 17 July 2026; read again on 29 September 2026: Claude Fable 5.1 37.4% (4 September 2026) and Claude Opus 5.5 33.5% (24 September 2026), both in Claude Code
• MITRE CWE-532 · definition of insertion of sensitive information into log file
• Anthropic, Claude pricing · Claude Fable 5.1 and Claude Opus 5.5 list prices, read 22 September 2026, converted at the Reserve Bank of Australia rate of that day, one Australian dollar buying 0.7123 United States dollars
• Anthropic, Introducing Claude Sonnet 5.5 · release date of 28 September 2026 and list price of 2 and 10 United States dollars per million tokens, converted at the same rate
