LLM Structured Output Shifts From Regex to Constrained Decoding

The regex era is ending

For most of the time developers have built on large language models, they have had a quiet problem. They ask the model for JSON, then use a regular expression to dig the JSON back out of a reply that may include markdown fences, a polite preamble, or an extra disclaimer. Through 2026, a steady run of developer guides has said the same thing: stop doing this. All of the major providers now offer native structured output, and hand-parsing is starting to look like a choice rather than a necessity19.

The argument is mostly about reliability in production. One engineer compares a fence-stripping regex to a time bomb. It works about 95% of the time in development, then fails on the slice of real traffic where wording changes, a model version updates, or the model explains in prose that it couldn't find a field2. Another write-up says the classic regex-plus-json.loads pattern produces a steady 5% to 15% error rate at scale8. These figures are practitioner estimates, not controlled measurements. But the failures they list are concrete and come up again and again across write-ups: greedy patterns that grab too much nested JSON, prices returned as the string "$142.50" instead of a number, and wrapper formats that change from one model release to the next23.

Three tiers of control

The most useful way to frame the shift is as a ladder. At the bottom is prompt-only JSON, where the system prompt asks for certain fields. One widely shared 2026 guide estimates this works 80–95% of the time and offers no type guarantees. Function calling or tool use sits in the middle and is more reliable, but the schema only guides the model rather than binding it. At the top is native structured output, which uses constrained decoding to guarantee schema-valid results1.

The top tier works differently from the others. A model produces text one token at a time. Constrained decoding steps in before each token is chosen and blocks any option that would break the target grammar2. In practice, the schema is compiled into a grammar or finite-state machine, and invalid tokens are masked before sampling19. One developer says plain JSON mode matched his schemas about 80% of the time, while strict structured outputs matched every time, which removed his retry loop entirely16.

Developers have to learn one distinction early. OpenAI's older JSON mode only guarantees that the output is valid JSON syntax. Structured Outputs with a schema also enforces required properties and enum values4. One 2026 practitioner guide calls JSON mode legacy and says strict mode is now the default path11.

Where the coverage disagrees: Is Claude guaranteed?

The clearest disagreement in the coverage is about Anthropic. Early-2026 guides described Claude as getting structured output through a forced tool call. They rated it "partial" for constrained decoding with roughly 99%+ validity, and recommended OpenAI or Gemini when compliance had to be guaranteed14. A March comparison said the same: Claude relied on instruction-following and was "technically not 100% guaranteed"17. An August piece still described Anthropic's tool use as strong but not guaranteed16.

The more recent and more detailed reporting points the other way, and I think it is right. Several guides describe a native output_config.format parameter that is generally available across Claude 4.5, 4.6 and 4.7 models, with strict: true also available on tool definitions1118. One explainer says Anthropic now ships native structured outputs rather than a tool-call workaround. It notes that strict tool use compiles the schema into a grammar that constrains sampling, which is the same family of techniques OpenAI and Gemini use19. An August analysis also lists Anthropic among the three vendors that now enforce JSON Schema at the sampling level13. The fair conclusion is that the "Claude is only 99%" claim was true for an older integration pattern. Developers repeating it now are working from an outdated picture.

The new problem: Schema subsets

If every major vendor guarantees valid output, the hard problem moves somewhere else. One analysis puts it this way: the question is no longer whether the model will produce valid JSON, but which parts of JSON Schema each vendor actually supports13. That matters for teams that keep one Zod or Pydantic schema and send it to several providers.

The differences are real. Anthropic reportedly doesn't support numeric minimum/maximum or string-length constraints, only allows minItems values of 0 or 1, and rejects recursive schemas. OpenAI requires every field to be listed as required and rejects a root-level anyOf. Gemini publishes the vaguest limits13. Another comparison adds that OpenAI requires additionalProperties: false, and that Anthropic's SDKs turn unsupported numeric constraints into description text instead of failing18. On size limits, OpenAI's are reported as 5,000 properties, 10 nesting levels and 1,000 enum values1318. An earlier guide gave a five-level depth limit14. That difference suggests either the limits have changed or some early-2026 guides were simply wrong.

Not every source sees portability as a serious risk. One explainer argues that because all three providers accept JSON Schema, a single schema can move between them with only the request code changed19. Another says OpenAI supports minLength and the full set of keywords20, which conflicts with the more detailed support matrices. The safer reading follows the detailed matrices: the format is portable, but the supported features aren't.

Chain of thought collides with the schema

For prompt engineers, the most important nuance is that guaranteed structure doesn't guarantee good reasoning. Research has warned about this for a while. The 2024 study "Let Me Speak Freely?" found that reasoning got worse under format restrictions, and that stricter constraints usually hurt more. It also found that this wasn't caused by parse failures, because several models had almost none23. The same study found that constrained JSON helped on classification tasks, where narrowing the answer space is useful23.

The cause can be as simple as field order. In that research, JSON-mode responses kept putting the "answer" key before the "reason" key, which skipped the chain of thought entirely26. Practitioner guides have turned this into a rule: put reasoning fields before answer fields, because the model otherwise commits to an answer before it has reasoned. The result is valid JSON with a wrong answer, and no validator will catch it29. Earlier benchmarks from the Instructor library showed how large the effect can be. Adding a chain-of-thought field to the response model raised GSM8K accuracy from about 33% to above 90%28.

A June 2026 arXiv paper adds an important qualification. It argues that the cost of formatting depends on how much spare capacity a model has. Claude Sonnet 4.6 scored about 89% on MATH-Hard with JSON, prose or chain of thought alike. Haiku 4.5 fell from 88.7% to 52.5% under heavy JSON, and GPT-4o-mini fell by 28 points21. Even the frontier models weren't immune: Opus 4.7 dropped from 96.2% to 91.0% on AIME when forced into JSON21. The authors trace the problem to "premature serialization," meaning the model is made to write schema-compliant tokens before it has finished thinking. Accuracy recovered whenever free reasoning came first and structured output came second21.

The patterns that actually hold up

Taken together, the coverage points to a fairly consistent set of patterns:

• Treat schema design as prompt design. One guide argues that schema design matters more than prompt wording. It recommends flat structures, enums for any known set of values, and one-line field descriptions that include units and bounds, because descriptions shape what the model generates9. Field descriptions steer the model's choices as well as documenting the schema4.
• Validate on both sides of the call. Even with enforcement at generation time, teams still run Zod or Pydantic afterward to check business rules that JSON Schema can't express414.
• Check the edge signals. Confirm that finish_reason is "stop", because a truncated payload won't parse8. Handle refusals according to each vendor's behavior. Anthropic reportedly returns stop_reason: "refusal" with an HTTP 200 and bills for the tokens18.
• Separate reasoning from formatting on hard tasks. Let the model reason freely first, or use extended thinking, and serialize the result afterward21.
• Add a confidence field and a schema version so that low-confidence results can go to human review4.

The bottom line

The shift is real, and regex parsing of model output is now hard to defend in production code. But the guarantee constrained decoding provides is narrower than the marketing suggests. It removes syntax errors and wrong types. It doesn't stop the model from inventing values or reasoning badly26. As one guide puts it, retry loops on an underspecified task just turn bad output into better-formatted bad output9. The teams doing this well treat the schema as an API contract and the field order as part of the prompt, and they keep a separate step for reasoning on tasks where accuracy matters.