1300 166 138 Tell us what you need
News / Generative AI

AI models decide their answer before they start thinking

By Monika Tantau • Published 7 months ago
AI reasoning model showing answer before generating chain of thought explanation

Key takeaways

• ServiceNow AI research found that reasoning models encode their final answer before generating any chain of thought tokens
• A linear probe predicted tool-calling decisions from pre-generation activations in most cases, before reasoning began
• Activation steering flipped model decisions between 7 and 79 per cent of the time depending on model and benchmark
• When decisions were flipped, the chain of thought rationalised the new outcome rather than resisting it
• The findings challenge the assumption that visible AI reasoning reflects the actual decision process

Most Read

When ChatGPT or Claude works through a question, most people assume it is genuinely thinking step by step before reaching an answer. New research from ServiceNow AI suggests that assumption may be wrong. A study published on arXiv found that reasoning models often have their answer locked in before they produce a single word of explanation — and that the reasoning shown to users is frequently constructed after the fact.

What the research actually found

Researchers developed a simple test to read the internal signals of reasoning models just before they began generating text. In most cases, the test could predict with high confidence which action or answer the model would choose, often before a single word of reasoning appeared on screen.

When researchers used activation steering to push the model toward a different decision, the chain of thought shifted to justify the new outcome rather than resist the change. In other words, the reasoning did not drive the decision. The decision was already set, and the reasoning followed.

How researchers tested the theory

The team then tested whether they could change a model’s decision by nudging its internal state before any reasoning began. They ran tests across multiple models and several standard benchmarks, finding they could successfully flip decisions between 7 and 79 per cent of the time depending on the model.

Critically, when a decision was changed, the model’s written reasoning adapted to match the new outcome rather than flagging any contradiction. The explanation adjusted as though the model had independently reached that conclusion on its own.

Research showing AI visible reasoning may not reflect actual internal decision process

The reasoning AI shows may not be the real reason

Most people who use AI tools assume the model works through a problem step by step before arriving at an answer. This study suggests that for many decisions, the outcome is already set internally before any explanation appears on screen.

That has significant implications for AI transparency. If the reasoning shown to users is constructed after the fact rather than driving the output, then reading an AI explanation does not tell you why it made a particular choice. The reasoning text and the actual decision process may be two separate things.

What it means for businesses using AI

Australian organisations increasingly use AI tools to support decisions in areas like customer service, document review, legal analysis and financial assessments. Many have adopted AI partly because the reasoning models provide visible explanations that look auditable.

This research suggests that visible reasoning may not be a reliable record of how a model reached its output. For Australian businesses operating under data governance frameworks or industry compliance requirements, the findings raise a practical question: if AI reasoning is constructed after the decision is already made, what exactly is being audited? Teams integrating AI into regulated or high-stakes workflows may need to reconsider how much weight they place on model explanations alone.

AI transparency has further to go

The research does not suggest AI tools are unreliable or unusable. It does suggest that the current generation of reasoning models operates in a way that is less transparent than the interface implies. A model that shows its working may not be showing how it actually works.

For the global AI industry, including the growing number of Australian companies building AI-powered products, this points to an emerging gap between what transparency looks like and what it actually provides. Interpretability research is still catching up with the scale and complexity of the models now in commercial use.

The paper arrives as regulators in Australia and globally move toward requiring explainability standards for AI used in consequential decisions. How those standards account for the gap between visible reasoning and internal decision processes will likely shape compliance requirements in the years ahead.

For Australian businesses building AI tools, understanding how these systems actually make decisions is the first step to deploying them responsibly. Aivy’s team specialises in AI chatbot development that is designed with transparency and business accountability in mind.

Scroll to Top