Chain of Thought Prompting Loses Ground to Built-In Reasoning

Built-in reasoning has absorbed the old prompt

For about four years, adding the phrase "let's think step by step" was one of the most widely repeated prompting tips. In 2026, current guidance says that tip is often unnecessary, and sometimes harmful, on the systems developers use most. Practitioner guides, vendor documentation and academic testing agree on the main point. Chain of thought did not fail. Instead, it moved inside the model. Models such as OpenAI's o-series and GPT-5 reasoning variants, Claude's extended-thinking modes and DeepSeek R1 now reason internally before they answer1. OpenAI's own best-practice guidance says that telling these models to reason step by step is unnecessary and may even reduce performance1.

The main change for people who write prompts is that reasoning depth is now set with an API parameter, not with wording. OpenAI exposes a reasoning-effort setting. Anthropic exposes a thinking-token budget. DeepSeek R1 reasons by default3. One 2026 guide argues that phrases such as "think harder" should not be used to get more deliberation. Developers should adjust the setting directly5. Taken together, these sources show that chain of thought is now built into the product rather than written into the prompt.

What the measurements show

The clearest evidence comes from Wharton's Generative AI Labs. Lennart Meincke, Ethan Mollick, Lilach Mollick and Dan Shapiro tested chain-of-thought prompting on GPQA Diamond, a set of 198 PhD-level multiple-choice questions in biology, physics and chemistry11. They ran each question 25 times under each condition. They then scored the results by average accuracy and by stricter thresholds, including whether a model answered a question correctly in every trial11.

For older non-reasoning models, the step-by-step prompt usually raised average scores. Gemini 2.0 Flash gained 13.5% and Claude Sonnet 3.5 gained 11.7%2. The cost was consistency. Under the perfect-accuracy measure, several models got worse, including Gemini Pro 1.5, which fell 17.2%2. The authors' explanation is that extra reasoning can introduce errors on easy questions the model would otherwise answer correctly11. Chain-of-thought requests on these models also took between 35% and 600% longer11.

For reasoning models, the gains were small. Average accuracy rose 2.9% for o3-mini and 3.1% for o4-mini, while Gemini Flash 2.5 dropped 3.3%2. Response times increased by 20% to 80%, or about 10 to 20 seconds per request11. The authors conclude that for dedicated reasoning models, explicit chain-of-thought prompting adds little and may not be worth the extra processing time13.

A caveat that strengthens the conclusion

An outside critique of the Wharton paper argues that its main comparison may overstate the benefit of chain of thought. The baseline prompt told models to answer "without any explanation or thinking," which actively blocked reasoning they might have done anyway14. When the researchers compared chain of thought with a neutral default prompt instead, the critique says only Gemini 2.0 Flash and GPT-4o showed significant average gains. Sonnet 3.5 and GPT-4o-mini showed none14.

This criticism makes the overall conclusion stronger. The authors themselves reported that many recent models already reason step by step when nobody asks them to12. If the gains look even smaller against a fair baseline, the trigger phrase does less than its reputation suggests, even on non-reasoning models. The reliability problem is also older than these models. A 2024 NeurIPS study of the Blocksworld planning problem found that gains from worked reasoning examples largely disappeared when test problems differed in generality from the examples20. The original 2022 chain-of-thought research had also found that smaller models gained little, and sometimes produced reasoning that sounded convincing but was logically wrong18.

Where the guidance disagrees

The disagreement in 2026 is about how far the retreat from explicit chain of thought should go. One practitioner guide lists chain of thought, few-shot examples, self-consistency, least-to-most prompting and skeleton-of-thought as techniques that actively make reasoning models worse. Its argument is that a prescribed step-by-step path can duplicate or conflict with the model's own process9. Another guide suggests a middle approach for hard tasks: run chain of thought inside a reasoning model, sample several reasoning traces and take a majority vote, and use tree-of-thought methods when the problem involves searching through options3. Another argues that explicit chain-of-thought strategies still give a level of control and auditability that built-in reasoning cannot provide4.

My reading is that the skeptics are right about the default and the defenders are right about specific cases. Several guides agree on where manual chain of thought still helps: non-reasoning models, small or locally run open-weight models, and cases that need one fixed reasoning path that people can inspect16. DeepSeek V4 is a notable exception to the "don't ask" rule. Because its thinking trace is fully visible and streamed as part of the output, one guide recommends explicitly asking it to show every step5. On reasoning models with hidden or summarized thinking, these guides converge on removing visible step-by-step instructions, which mostly add to billed output6.

Another claim cuts against the assumption that longer reasoning means better answers. One analysis cites a February 2026 paper finding that removing filler words such as "Hmm" and "Wait" from reasoning traces shortened them by 27% to 51% without changing accuracy19. If that result holds up, trace length is a poor sign of quality, and paying for more visible reasoning makes even less sense.

How system prompt design is changing

As reasoning moves inside the model, system prompts are shifting from instructions about how to think to instructions about what a good result looks like. One 2026 guide describes a common structure for prompts to reasoning models: the goal, constraints, context, a reasoning budget set through the API, and evaluation criteria. It also advises developers to state the desired outcome rather than a procedure5. Its reasoning is that procedural prompts substitute the developer's reasoning for the model's and limit quality to the level of that script5. One best-practices checklist makes the same point and warns against "chain-of-thought everywhere" because of the extra cost in tokens and time25.

At the same time, the structural basics of system prompts have become largely settled. Common guidance includes:

• Separating sections clearly. Guides recommend XML-style tags or markdown headers to divide instructions, context and user data2128. Claude is widely reported to respond best to XML tags, while GPT models handle markdown headers and JSON schemas well28.
• Placing key instructions carefully. Microsoft's Foundry documentation advises stating the task first and testing whether repeating instructions at the end helps. Its reason is that models can be influenced more by recent text27. A reasoning-model guide gives similar advice: put binding requirements near the end so they are still prominent when the model finishes thinking5.
• Giving the model a fallback. Microsoft recommends telling the model what to return, such as "not found," when it cannot answer, to reduce invented responses27.
• Defending against prompt injection. Recommended defenses include stating that system instructions override user requests and wrapping user input in tags that the model treats as data, not commands2822.
• Managing prompts like code. One guide suggests keeping prompts in version control and running a test set of 50 or more input-output pairs whenever a prompt changes28. LaunchDarkly goes further and treats prompts as runtime configuration that teams can A/B test and roll back without a new code deployment21.

The older list of prompt patterns still exists. Zero-shot, few-shot, role prompting, prompt chaining, reflection and tree-of-thought prompting all appear in current taxonomies22. What has changed is that choosing a pattern now depends on the model. Guides that once recommended stacking techniques now check model fit first25.

Transparency is the next problem

Building reasoning into models creates a new problem. If the reasoning happens out of view, people may not be able to inspect it. One October 2026 report argues that architectures that move part of a model's reasoning into its internal representations reduce the visible chain of thought that safety monitors can read7. The same report says that the reasoning traces models produce are sometimes faithful to their actual process and sometimes not7. Its claims about specific unreleased model designs are harder to verify than the benchmark findings above. Still, the direction it describes is consistent with the shift reported across the field, from reasoning that users can read toward reasoning that runs out of sight.

This also changes one argument for chain-of-thought prompting. Practitioners who rely on it for auditability should know that an explicitly requested reasoning path may differ from the model's actual internal reasoning7. For most production work in 2026, the guidance across sources is consistent: choose the right model tier, describe the goal and constraints clearly, set the reasoning effort deliberately, and judge the answer by measured results rather than by how convincing its reasoning looks511.