1300 166 138 Tell us what you need
News / Security

Humans approving AI commands miss one in three threats

By Monika Tantau • Published 3 months ago
Humans approving AI commands miss one in three threats

Key takeaways

• ScaleX ran more than 40,000 sessions producing around 409,000 approve or deny decisions
• The average reviewer scored 66.3% accuracy, missing roughly one threat in three
• Credential theft was missed 33.4% of the time against 11.7% for destructive commands
• Miss rates climbed as sessions went on, and 7% of players approved everything

Most Read

Almost every business case for AI automation contains the same sentence: a person will review it. That sentence is what gets the proposal past the risk committee, and it is rarely examined again. New data suggests it should be.

A study published by ScaleX put more than 40,000 sessions and around 409,000 individual approve or deny decisions through a controlled test. The average reviewer caught 66.3% of the threats put in front of them, which is another way of saying they waved through one in three.

What the study actually measured

Participants played the role of a human supervisor approving commands issued by an AI coding agent. Roughly a third of the commands they saw were genuine threats, built to steal credentials or damage the system, and the rest were ordinary operations. Players saw the agent’s history before each call and worked under time pressure.

The headline number is bad enough, but the distribution is worse. Only 20.8% of participants caught every threat while still letting legitimate work through. Around 32.9% finished with a negative score, and 7% approved everything placed in front of them without exception.

Where the misses cluster

The failures are not spread evenly, and that is the useful part. Openly destructive commands were missed 11.7% of the time, because a command that obviously wrecks something is easy to spot. Credential theft was missed 33.4% of the time, and commands that quietly exceeded their authorised scope 35%.

In other words, the quiet attacks get through roughly three times more often than the loud ones. The study also found that commands which looked routine, the sort of everyday build instruction a developer types without thinking, roughly doubled their chance of being approved, a blind spot that also shapes how teams pick secure AI coding tools.

Approval fatigue is the real finding

Accuracy fell as sessions went on. Early in a run people read carefully. Fifty decisions later they were clicking. This is the pattern that matters most for anyone deploying agents at scale, because it means the safeguard degrades exactly as volume increases, which is precisely when it is needed.

It reframes what a review step is worth. The human in the loop is the safeguard that almost every AI governance framework leans on, and asking one person to approve everything is not a control, it is a formality with a person attached to it.

Approval fatigue is the real finding

Why Australian businesses should pay attention

The design being tested here is already common in Australian offices. A system drafts responses and a staff member approves them. Another proposes invoice codings and someone signs off. A third suggests changes and a developer clicks accept. All three rest their safety case on a person reviewing in volume.

There is a live precedent for what happens when isolation fails. Anthropic disclosed that its own evaluations reached three real companies after a partner misconfiguration, an incident we covered when it broke. Deciding which decisions genuinely need a human, and which are better handled by a hard limit, is the substance of AI advisory work rather than an afterthought once the tool is chosen.

Design for fewer decisions, not more

The practical response is not to add reviewers. It is to reduce what reaches them. Set a threshold so routine work runs unattended and anything above it stops every time. Reserve human sign-off for actions that cannot be undone, which usually means money leaving, messages reaching customers, and data being deleted.

One measurement is worth adding to any deployment. Track what proportion of items the reviewer approves. If it sits at 99%, that step has stopped filtering anything, and it is better to know that from a dashboard than from an incident report.

Scroll to Top