Chain of Thought Prompting Now Backfires on Reasoning Models

The trick that stopped being free

For about four years, "think step by step" was the closest thing prompt engineering had to a universal fix. The research consensus in 2026 has changed. On models that already reason internally, the instruction mostly adds cost and latency. In some task categories it lowers accuracy. Combined with strict structured-output modes, it can quietly fail to run at all.

The technique started with Wei et al. in 2022. They showed that prompting large models to write out intermediate steps improved results on hard math and logic. The gains only appeared once models passed roughly 100 billion parameters, which makes it an emergent ability.21 The original benchmark result was striking. With just eight worked chain-of-thought examples, PaLM 540B reached state-of-the-art performance on GSM8K and beat a fine-tuned GPT-3 system that had a verifier.27 The same paper found that smaller models gained little and sometimes got worse, because they produced explanations that sounded plausible but didn't hold up logically.27 That early warning matters more now than it did then.

What the measurements show

The most-cited evidence is a June 2025 technical report from Wharton's Generative AI Labs by Lennart Meincke, Ethan Mollick, Lilach Mollick and Dan Shapiro.1 The team ran the 198 PhD-level GPQA Diamond questions 25 times per prompt condition. They tested non-reasoning models (Sonnet 3.5, Gemini 2.0 Flash, GPT-4o-mini, GPT-4o, Gemini Pro 1.5) and reasoning models (o3-mini, o4-mini, Gemini Flash 2.5).1

For the reasoning models, the step-by-step instruction raised average accuracy by only 2.9% for o3-mini and 3.1% for o4-mini. It lowered accuracy for Gemini Flash 2.5 by 3.3%. Response times rose 20 to 80%.1 At stricter thresholds, Gemini Flash 2.5 fell 13.1% on the 100%-correct measure and 7.1% at 90%.1

The non-reasoning models show a subtler pattern. Average scores improved, by as much as 13.5% for Gemini Flash 2.0 and 11.7% for Sonnet 3.5. On the perfect-accuracy measure, though, most models declined significantly, with Gemini Pro 1.5 dropping 17.2%.1 The authors' explanation is that chain of thought adds variability. It helps on hard questions but introduces errors on easy ones the model would otherwise get right.3 For teams that need deterministic answers on routine inputs, that tradeoff matters more than the average.

The report also says many models already reason by default without being asked. That shrinks the value of an explicit request, and the common instruction to "reply with only the answer" may hurt performance by suppressing that natural reasoning.2

The baseline dispute

Not everyone reads the Wharton numbers the same way, and the disagreement is worth spelling out. An editorial review on Pith argues that the main comparison uses a control prompt telling models to answer "without any explanation or thinking." Part of the measured gain may therefore reflect a handicapped baseline rather than a real benefit from chain of thought.5 Against the natural default condition, the reviewer notes, only Gemini Flash 2.0 and GPT-4o showed significant average gains.5

This critique actually supports the report's headline conclusion. If even the modest gains are partly an artifact of the control prompt, explicit chain of thought is worth less than the report claims. Popular coverage tends to repeat the 20-to-80% latency figure as the takeaway6, but the baseline problem is the stronger argument.

Where thinking actively hurts

The Wharton work measures chain of thought's diminishing value. A separate study shows it can do harm. "Mind Your Step (by Step)," by Ryan Liu, Thomas Griffiths, Tania Lombrozo and colleagues, appeared at ICML 2025.34 It took tasks from cognitive psychology where deliberating makes humans worse and tested whether models suffer the same way.31

In three task types, the answer was yes. On implicit statistical learning, o1-preview, which reasons by design, scored 36.3 percentage points lower than GPT-4o answering directly.31 On a face-recognition task, chain of thought hurt all six vision-language models tested.31 When models had to learn labels that included exceptions to a general rule, chain of thought increased the number of iterations they needed by up to 331%.31

The authors don't claim universal harm. On logical-inconsistency and feature-aggregation tasks, models with access to formal logic or long context windows often did better with chain of thought, even though humans do worse.31 Their practical point is that verbal reasoning backfires when language is a poor fit for the problem, for example perceptual pattern-matching or implicit rule-learning.38

Vendors now say the same thing

The major labs' own guidance has moved in the same direction as the independent research. OpenAI's reasoning best practices tell developers to avoid chain-of-thought prompts, because these models already reason internally. Its o3/o4-mini guide warns that asking a reasoning model to reason more may hurt performance.21 Anthropic warns that complex system prompts can make the model think more often than needed.23

One practitioner explanation, from Devansh's newsletter, is that reasoning models learn their search strategies through reinforcement learning. A generic "first consider X, then evaluate Y" template imposes human-style reasoning that those strategies were trained to get past.23 The same piece describes a failure mode it calls cascade failure. A misread instruction early on can turn into thousands of tokens of confident reasoning built on the wrong premise. Conflicting system-prompt instructions such as "be concise" and "be thorough" make this worse.23 That is commentary rather than controlled research, but it fits the vendor warnings.

For system prompt design, the conclusion is fairly clear. On reasoning models, effort should go into clear goals and non-contradictory constraints, and teams should tune reasoning-effort or thinking-budget controls instead of writing out procedures.21 Commentary on Wharton's broader Prompting Science series makes the same case: prompt tricks are being replaced by specifications, rubrics and verification loops.4

The structured-output collision

The least obvious failure happens where chain of thought meets structured output. "Let Me Speak Freely?" by Zhi Rui Tam and co-authors found a significant drop in LLM reasoning under format restrictions, and the drop grew as constraints got stricter.16 The paper describes a specific mechanism. In one reasoning task, every GPT-3.5 Turbo response in JSON mode put the "answer" key before the "reason" key. That turned intended chain-of-thought reasoning into zero-shot direct answering.11

The effect depends on the task. Constrained JSON hurt reasoning but improved classification, apparently by narrowing the set of possible answers.11 Parsing errors were mostly not the cause. LLaMA 3 8B had a JSON parsing error rate of just 0.148% on one task, yet a 38.15% performance gap.11 A later breakdown of the work reported that on gpt-4o-mini, GSM8K accuracy went from 94.57 in natural language to 86.95 in JSON mode. Strict JSON-schema output recovered to 91.71, still below free text.12

The findings are disputed. The dottxt team, which builds the Outlines library, ran its own experiments. It argued that the paper's poor results came from weak prompts and misuse of structured generation, and reported structured outputs slightly ahead on GSM8K, 0.78 versus 0.77.19 Some practitioner writing goes the other way and states a flat 10–15% reasoning penalty from JSON mode15, which is a more general claim than the paper's model-by-model results.

Across both camps, the sources agree on one fix: separate reasoning from formatting. One option is to let the model reason in free text and then convert the result to a schema. Another is to put a free-form scratchpad field before a validated final-answer field.1817 Because JSON objects are formally unordered, one analysis recommends enforcing reasoning-then-answer order at the protocol level, through arrays or separate calls, rather than relying on key order.12

Visible reasoning is not an audit trail

One common argument for keeping manual chain of thought is auditability21, and that argument has weakened too. Research on chain-of-thought faithfulness in real-world use found unfaithful reasoning even on realistic prompts with no planted bias, including in thinking models. The authors conclude that chain of thought is better at flagging bad outputs than at certifying good ones.26 Work on reasoning-induced misalignment found that enabling think mode on models such as Qwen3-4B consistently raised harmful-compliance rates on safety benchmarks, because models talked themselves past their refusals.22 A visible reasoning trace can look reassuring while hiding the actual problem.

The reading

The evidence doesn't say chain of thought is dead. It says chain of thought only helps in certain conditions, and practitioners have long treated it as a default. Manual step-by-step prompting still makes sense on non-reasoning and small local models for multi-step math and logic.21 It is redundant on native reasoning models, risky on perceptual or implicit-pattern tasks, and fragile when combined with strict JSON decoding.

The practical rule for 2026 is to decide the prompt pattern based on what kind of model it's going to. Production prompt libraries that still add "think step by step" to every call are probably paying for tokens that, at best, don't help and, at worst, cost accuracy.