Structured LLM Output: SLOT Formatter Hits 99.5% Schema Accuracy

A formatting model that runs after the main LLM

For most of the time large language models have been in production, getting usable data out of them has meant prompt engineering: a system prompt that asks for "valid JSON only," some few-shot examples, and a retry loop for when the output still breaks. SLOT, short for Structured LLM Output Transformer, takes a different route. The work comes from a team at Amazon Web Services. It does not try to make the main model follow a schema more closely. Instead, it adds a small fine-tuned model that takes the main model's free-form answer and rewrites it as JSON that fits a given schema.16

The paper first appeared on arXiv on 6 May 2025 as 2505.04016, with Darren Yow-Bang Wang, Zhengyuan Shen, Soumya Smruti Mishra, Zhichao Xu, Yifei Teng and Haibo Ding as authors.2 It was later published in the Industry Track of EMNLP 2025 in Suzhou, where it fills pages 472–491 of the proceedings.7 The order of the first two names differs between versions. arXiv lists Wang first, while the ACL Anthology entry lists Shen first. Both versions mark the two as co-first authors.378

The main argument is about where structure should be enforced. The authors present SLOT as an alternative to two common approaches: post-training a model to emit structured output, and constrained decoding during generation. Because SLOT only post-processes text, it can sit behind any model and any task.1 They describe the problem as "text-to-structure": the input is an LLM's response, not the user's question, and the job is to map that response onto a JSON schema.1

What the numbers show

The headline results are consistent across the abstract, the conference version and outside summaries. A fine-tuned Mistral-7B, combined with constrained decoding, reaches 99.5% schema accuracy and 94.0% content similarity. That beats Claude-3.5-Sonnet by about 25 and 20 percentage points.67 Smaller models also do well. With the XGrammar constrained-decoding engine attached, a 1B-parameter Llama-3.2 reaches 96.2% schema accuracy and 89.6% content similarity.46 Even with fine-tuning alone, the 1B model's 88.9% schema accuracy is roughly equal to Claude-3.5-Haiku, and its 81.7% content similarity is higher than Sonnet's 73.9%.1

The training data does a lot of the work. The authors built a synthetic dataset of about 126,000 examples covering different text lengths, writing styles, industries and levels of JSON complexity.451 Their own ablation shows how much this matters. Trained only on public data, the 1B Llama scored 63.1% on schema accuracy and 68.3% on content similarity.1 The conference version gives the curated set a name, SLOTBench. It also says the benchmark will only be released "upon legal approval," a common caveat for industry papers that makes independent replication harder for now.7

The two metrics measure different things, and that distinction matters. Schema accuracy checks that keys and value types match the schema exactly. Content similarity is a Sentence-BERT-based score of whether the meaning of the source text made it into the JSON.4 The second metric targets a problem that practitioner guides keep raising: output can be perfectly shaped and still wrong.1920

Where the sources disagree

Some summaries of the paper don't match its baselines. A machine-generated summary on alphaXiv reports a 98.2% schema accuracy for fine-tuned Mistral-7B and compares it with GPT-4-Turbo at 71.2% and Claude-3-Sonnet at 80.3%.6 Those comparisons don't line up with the Claude-3.5-Haiku and Claude-3.5-Sonnet baselines the paper itself names.1 At least one later paper also cites the work with the wrong arXiv number and year.10 The figures to rely on are the ones in the abstract and the ACL proceedings.

The biggest disagreement is about how fair the comparison is. An automated review on Pith calls SLOT a useful post-processor with credible internal results. But it argues that the large margins over Claude come from a baseline prompt that pushes the model into schema violations.4 The paper says it ran the Claude models "with standard prompting."1 That is the key question for practitioners. In 2026, the Claude family supports native JSON-schema enforcement and strict: true tool definitions.15 A prompt-only Claude baseline doesn't reflect how a careful team would use those models today.

In my reading, the critique weakens the claim that small models beat proprietary ones, but it doesn't touch the main design idea. The 25-point gap is best read as the gap between a fine-tuned formatter and plain prompting, not a fair contest against a properly configured frontier API. The reviewer seems to agree. The same review says the core idea of a separate formatter is "genuinely useful" and that the results for open-weight models largely hold up.4

Splitting reasoning from formatting

SLOT matters for system prompt design because it supports a pattern that has spread through practitioner guidance: do the reasoning in one step and the formatting in another. A 2026 analysis of constrained decoding argues that forcing a schema can hurt reasoning, but mostly because of the prompt, not the decoder. Telling a model to answer immediately as a JSON object skips the intermediate tokens it would use to work things out.11 That analysis suggests constraining the wire format but leaving the model free to think.11 Another guide gives the same advice: make one unconstrained call to reason in prose, then a cheap constrained call to extract the result, which can go to a small model.18

SLOT is essentially that second step, trained for the job. One summary describes it as separating the model's "brain" work from its "clerical" work.5 For system prompts, this means the main model's instructions no longer need long formatting rules, schema text or reminders not to add a preamble. Practitioner guides note that putting a schema in the prompt costs tokens. One estimates about 2,000 tokens for a deeply nested schema.13 With a separate formatter, the main prompt can focus on the task, and the schema only goes to the formatter.

SLOT also works alongside constrained decoding rather than replacing it. Its best results use both: fine-tuning handles how the content is organized semantically, and grammar-constrained decoding with XGrammar guarantees the structure.45 XGrammar reports less than 40 microseconds of overhead per token on JSON-schema grammars, so the guarantee adds little cost.11

How SLOT fits the 2026 structured output landscape

The environment has changed since May 2025. Practitioner overviews now describe constrained decoding as standard across the major providers. Strict schema modes are available from OpenAI, Anthropic, Google and Mistral.1215 One analysis calls structured output "a settled syntactic problem and an unsettled semantic one."11 Several guides put the same idea differently: the grammar guarantees shape, but it can't tell whether a field holds the right value.1920

That framing narrows SLOT's case. If you only need valid JSON from a frontier API, a strict mode already does that. SLOT's lasting value is in three areas:

• Open-weight or self-hosted models that lack a robust native strict mode.
• Models whose output can't be constrained at generation time, for example a hosted reasoning model whose prose you want to keep intact.
• Content fidelity, a problem that providers' grammar engines don't address at all.

The content-similarity results matter most here, because they target the semantic gap that schema enforcement leaves open.14

Support for JSON Schema also varies across tools. The JSONSchemaBench study found that the best constrained-decoding framework handled about twice as many real-world schemas as the worst.14 Even Claude's strict mode is reported to reject recursive schemas and numeric bounds.15 A learned formatter doesn't depend on a grammar compiler accepting the schema, although SLOT's best results still use one.

Limitations and what to watch

The weaknesses are clear. One summary says the model still struggles with nesting deeper than about four levels and with ambiguous cases where it is unclear which key a piece of content belongs to.5 Adding a second model also adds latency and another component to deploy, a cost the headline numbers don't show. The authors plan to explore preference tuning and reinforcement learning, along with better data synthesis.1

SLOT doesn't make native structured outputs obsolete, and its comparison with Claude overstates its lead. What it does well is show that formatting can be a separate, cheap, trainable step. That supports a practical rule for prompt design: let the main model reason in plain language, and let a separate component, whether SLOT or a strict API mode, handle the formatting. Teams building agents or extraction pipelines on open-weight models are the ones most likely to benefit, especially once SLOTBench is released.