Chain-of-Thought Prompting Shifts From Prompt Trick to Schema Design

A technique that turned into a default

Chain-of-thought (CoT) prompting is one of the most widely taught ideas in prompt engineering. The basic version is easy to explain: you ask the model for the steps it took to reach an answer, not just the answer. Industry explainers describe it almost the same way. IBM calls it a technique for multistep reasoning tasks, built by guiding a model through a logical series of intermediate steps1. AWS describes it as splitting a complex task into simpler steps that lead to an answer2. Coursera describes it as a way to increase a model's reasoning power with a step-by-step approach7.

The idea comes from a Google research paper, arXiv 2201.11903. It defined a chain of thought as a series of intermediate natural-language reasoning steps that lead to a final output4. Jason Wei, a research scientist at Google, described the aim as having the model's reasoning chain mimic an intuitive thought process on multi-step problems10. A freeCodeCamp review of that paper highlights two findings. The gains grew as problems got harder, and on single-step tasks standard prompting was already good enough30.

Today the picture is more complicated than the explainers suggest. CoT now matters less as a phrase you type and more as a design decision in system prompts, output schemas and model choice. In some cases it is worth leaving out entirely.

The pattern catalogue

The coverage agrees on a basic taxonomy. Zero-shot CoT adds an instruction such as "let's think step by step" without examples. Few-shot CoT includes worked examples whose reasoning steps resemble the target problem9. Splunk notes that few-shot CoT literally makes the prompt longer, using example question-and-answer pairs to show the intermediate steps the model should follow5. Auto-CoT automates this. It picks diverse questions from a dataset, generates reasoning chains for them with zero-shot CoT, and inserts those chains as demonstrations89.

Several specialised variants extend the family. AWS lists least-to-most prompting, which breaks a hard question into smaller sub-questions and solves those first2. Faithful CoT targets a known weakness: the stated reasoning and the final answer can disagree. It converts the query into a symbolic chain and hands it to a deterministic solver9. Tabular CoT puts the reasoning steps in a markdown table so they are easier to read9.

CoT also needs to be distinguished from prompt chaining. IBM's distinction is the cleanest: chaining spreads a task across several sequential prompts, while CoT draws out the reasoning inside one prompt1. Tricentis calls chaining a workflow design pattern, not a reasoning technique. It argues the strongest setups combine CoT, few-shot examples and chained pipelines10.

For writing system prompts, AWS offers concrete phrasings that work as templates. These include "explain your reasoning," "first determine X, then identify Y," and explicit if/then/else branches2. Practitioner forums share similar scaffolds. These are numbered lists of sub-questions the model must answer before its final recommendation3.

The case for: transparency and debuggability

The explainers agree on why CoT helps. One reason is accuracy on multistep problems such as arithmetic, commonsense and symbolic reasoning12. The other reason, arguably more important for production teams, is observability. Codecademy argues that a visible reasoning sequence lets developers find the exact step where a model went wrong and ask for a correction there8. IBM also cites observability and debugging as benefits1. Some vendor claims are more aggressive. K2view says CoT improved solve rates on the GSM8K math benchmark by more than 300% over standard methods6. That figure describes the original research era, not current models.

The weakness here is that most of these explainers present CoT as a reliable gain. Only a few, such as AWS, list limitations. AWS notes that CoT needs larger models, may not suit every task, and can produce a final answer that doesn't follow from the stated reasoning2.

The case against: diminishing returns on modern models

The strongest counterevidence comes from Wharton's Generative AI Labs. Their report, "The Decreasing Value of Chain of Thought in Prompting," tested CoT on GPQA Diamond, a set of 198 PhD-level multiple-choice questions in biology, physics and chemistry21. For non-reasoning models, CoT raised average accuracy. Gemini 2.0 Flash gained 13.5% and Sonnet 3.5 gained 11.7%, while GPT-4o-mini's 4.4% gain was not statistically significant21. But the share of questions answered correctly in every trial fell for several models, including a 17.2% drop for Gemini Pro 1.5. The authors' reading is that CoT adds variability, which can cause errors on easy questions the model would otherwise get right21.

For dedicated reasoning models, the returns were small. o3-mini gained 2.9%, o4-mini gained 3.1%, and Gemini Flash 2.5 lost 3.3%. Meanwhile CoT requests took 20–80% longer21. The authors conclude that for these models, explicit CoT prompting adds little and may not justify the extra processing time22. They also found that many recent models already reason step by step without being asked, so an explicit request changed little24.

Not everyone accepts the headline numbers. A critique on the review site Pith argues that the main comparison is skewed. The baseline prompt told models to answer "without any explanation or thinking," so some of the measured CoT gain may reflect a handicapped control25. Against the paper's more neutral Default condition, Pith notes, only Gemini Flash 2.0 and GPT-4o showed significant average gains25. This critique doesn't rescue CoT. It strengthens the report's main conclusion, because it suggests even the modest gains for non-reasoning models are overstated.

A separate paper, "Mind Your Step (by Step)," goes further. It finds that on tasks where verbal deliberation also hurts humans, such as implicit statistical learning, reasoning can actively reduce accuracy. OpenAI's o1-preview scored 36.3 percentage points below GPT-4o's zero-shot result on one subset28. Commentators have drawn sharp conclusions from this. One newsletter calls CoT an anti-pattern on reasoning models29, and Vellum advises teams using reasoning-native models to test carefully before adding it27.

The defensible conclusion is that "think step by step" no longer helps by default. It helps most on older or smaller models that don't reason unprompted, on hard multistep problems, and when latency and token costs are acceptable. Applied to everything, it costs money and can make answers less consistent.

Where CoT survives: inside the schema

CoT now matters most in structured output design. Developers increasingly need LLM responses that match a JSON schema, and vendors now enforce schemas at the decoding level. OpenAI's structured outputs feature uses constrained decoding to guarantee schema compliance. Google Gemini and Anthropic offer their own schema options12. The catch is that a schema allowing only {"answer": ...} leaves the model no room to reason before it commits15. One guide cites research (Tam et al., 2024) finding that reasoning gets worse under strict format rules17.

The common fix is to put the reasoning inside the schema. OpenAI's own documentation includes a chain-of-thought example. Its schema contains an array of steps, each with an explanation and an output, followed by a final_answer field11. Field order matters. One practitioner warns that if the answer field comes before the reasoning field, the model commits to an answer before it reasons. The result is valid JSON with a wrong answer, and validation won't catch it12. A community test on GSM8K with GPT-4o compared reasoning-first and answer-first orderings to show the effect19. Taskade adds a vendor-specific detail. OpenAI emits keys in schema order, while Anthropic emits required properties first, so the reasoning field must be listed first for one and marked required for the other17.

The pattern has critics. Computational linguist Yoav Goldberg argues that putting reasoning inside a data structure breaks basic language-use principles and may make models more brittle. He proposes JSON5-style comments as a more natural place for reasoning14. Separately, research on Schema Reinforcement Learning trains models to write reasoning comments in JSON5 before the structured output, and reports up to a 16% gain in JSON generation accuracy13. That convergence suggests the comment-based approach may be more than a stylistic preference.

The other answer is to keep reasoning out of the schema altogether. With Anthropic's extended thinking, the grammar applies only to the final output, not to the thinking block17. vLLM's reasoning parser likewise returns the model's reasoning separately from the schema-constrained content20. Developers without that support often chain two calls. A reasoning model does the analysis, and a faster model formats the result into JSON1615.

What it means for prompt engineers

Prompt engineering has moved on from simple instructions. On reasoning models, the reasoning step is increasingly built in, and adding generic CoT on top mostly adds latency21. On cheaper non-reasoning models, CoT can still help with hard problems, at the cost of tokens and consistency22. In structured pipelines, CoT is no longer an instruction in the prompt. It's a property of the schema, and something as minor as field order can decide whether the output is correct12.

That makes CoT a design choice that depends on the model, the task, the accuracy threshold and the latency budget. Teams should test it against their own workloads rather than using it everywhere.