Structured Output LLM Benchmarks Show Widespread Label Errors

Wrong answer keys

Any model leaderboard depends on the answer key being right. For structured outputs, the work of turning contracts, financial news and PII-laden documents into typed JSON, there is growing evidence that the answer keys often are not.

Cleanlab started the argument with a blunt claim. While benchmarking leading LLMs on structured outputs, the company found that many apparent model "errors" were really mistakes in the benchmarks' ground-truth outputs. It ran the public datasets through its automated label-error detection and reported that every benchmark it examined had erroneous and inconsistent ground truth. The error rates were high enough that Cleanlab said it would put little trust in accuracy figures built on them.11 It named three datasets: a Financial Relation Extraction set (FIRE), a Lease Contract Review set, and a PII Entity Recognition set. It said it had checked many others and found them too noisy to use as well.11

The failures differed by dataset. In FIRE, conceptually equivalent examples got inconsistent treatment, with an entity extracted in one place and left out in another, and many instructions were too ambiguous to say whether an extraction was correct.11 The lease dataset was described as full of annotation errors and inconsistencies. In the PII set, many labels were plainly wrong.11 Cleanlab's own takeaway is that producing accurate structured outputs is hard for human annotators too.11

The replacement benchmarks

Cleanlab didn't just criticize the old datasets. It released four new benchmarks: Data Table Analysis, Financial Entities Extraction, PII Extraction and Insurance Claims Extraction. It says their ground truth was checked through error analysis with frontier models and its own tooling.11 Each example comes with an LLM prompt, an input document, a JSON schema with typed fields, and a reference output.11 The code is public on GitHub, where the project calls itself a benchmark "whose 'ground-truth' is actually right."13

The way Cleanlab rebuilt FIRE matters for prompt and schema design. It did not just correct labels. It removed fields it judged ambiguous or overlapping (Action, BusinessUnit, Designation, FinancialEntity and Sector) and merged GeopoliticalEntity and Location into one Location field, because geographic mentions had been labeled inconsistently.11 In other words, the fix was to the task specification. If careful annotators can't tell a geopolitical entity from a location under a given instruction, a model can't be expected to either. A benchmark that scores it on that distinction is mostly measuring noise.

What the cleaner numbers show

Cleanlab scored models two ways. Field Accuracy is the share of individual fields extracted correctly. Output Accuracy is the share of examples where every field is correct.11 The gap between the two is large. On PII Extraction, GPT-4.1-mini, GPT-5, Gemini 2.5 Pro, Gemini 2.5 Flash and Gemini 3 Pro all scored between 0.966 and 0.979 on Field Accuracy, but only 0.26 to 0.46 on Output Accuracy.11 Every model gets nearly every field right, and still fewer than half of its documents come out fully correct.

On the model comparison, Cleanlab concluded that Gemini 3 is not a big step past OpenAI's frontier for these tasks. GPT-5 led on Financial Entities Extraction and was roughly level with Gemini 3 Pro on PII and Insurance Claims, while Gemini 3 Pro led on Data Table Analysis.11 Cost mattered as much as accuracy. Gemini 3 Pro cost about 1.5 times as much as GPT-5 to run because it used more thinking tokens. GPT-4.1-mini cost about a fifth as much as Gemini 2.5 Flash in its default thinking mode and had lower median latency.11 Cleanlab's overall advice was to start with OpenAI models for structured-output work because they are cheaper, not because they are more accurate, and to test several models whenever accuracy is critical.11 A week later, Cleanlab used the same suite to argue that its trust-scoring method catches structured-output errors with 25% better precision and recall than LLM-as-a-judge or token log-probabilities.12 That follow-up is also a product pitch, and readers should weigh it that way.

Other benchmarks point the same way

The finding fits a wider pattern. In early 2025, MIT researchers behind the "platinum benchmarks" project relabeled subsets of fifteen supposedly saturated benchmarks. They found that for most of them, more than half of model failures came from label noise, with about 75% of errors on the original SVAMP traced to bad or mislabeled questions.17 They also cautioned that declaring benchmarks saturated too early can hide real reliability gaps.15 Cleanlab's results suggest structured-output evaluation has the same problem, possibly worse, because each example contains many fields that can each be mislabeled.

Research since then has moved toward measuring value correctness instead of format compliance. The Structured Output Benchmark (SOB), posted to arXiv in April 2026, notes that earlier efforts such as JSONSchemaBench and StructEval mostly stop at schema compliance or cover only one input type.14 Across 21 models, SOB found JSON pass rates 15 to 25 points higher than value accuracy. GPT-5.4 produced valid JSON 99.97% of the time but got every field right on only 0.486 of responses.1 Best-in-class value accuracy fell from 83.0% on text to 67.2% on images and 23.7% on audio.1 The authors describe the most dangerous failure as a "structured hallucination": a value with the right type and a plausible look that simply isn't supported by the source.

The sources disagree on one point, and it matters. Cleanlab's main worry is that the reference answers are wrong. SOB's main worry is that the model's answers are wrong while looking correct. The two concerns interact. SOB builds its ground truth through human review with a cross-check by Gemini 2.5 Flash1, which is close to the model-assisted cleaning Cleanlab recommends. SOB also admits that exact-match scoring penalizes equivalent answers such as "USA" versus "United States."14 So the newer benchmarks have their own measurement error, of a different kind. A reviewer of StructEval made a similar point, arguing that its new metrics hadn't been validated enough to make its reported gaps reliable.16 In this field, anyone who builds a benchmark should expect it to be audited too.

Lessons for prompt and schema design

For people writing system prompts and schemas, these findings go beyond choosing a model. Several practical patterns come out of the reporting.

Remove ambiguity before blaming the model. Cleanlab's FIRE rebuild is the clearest example. If a field definition doesn't let an informed reader decide what belongs in it, rewrite the definition, merge overlapping categories, or drop the field.11 LLMStructBench, which tested 22 models with five prompting strategies, found that prompting strategy mattered more than model size. It also found that semantic errors remained even when the output's structure was valid.14

Say what to return when the information is missing. BenchLM's September 2026 tests built their fixture so that a fact absent from the input must come back as unknown or null, and any filled-in value counts as a failure.2 Their example: in a ticket with no stated urgency, a model returned "urgent." It was a legal enum value and still false.2 Prompts that never allow an "unknown" answer push models to fill in a plausible guess.

Field order is part of the prompt. One practitioner guide points out that if answer fields come before reasoning fields in a schema, chain-of-thought models commit to an answer before they've reasoned through it. Its advice is to put reasoning fields first.7 Related research cited in the SOB paper found that naive grammar-constrained decoding can skew outputs toward valid syntax at the expense of meaning, and that forcing JSON, XML or YAML can weaken reasoning.1

Split up large schemas. Developers describe sending sub-schemas grouped by domain as separate, focused prompts and merging the results. They report that this shortens outputs, nearly eliminates truncation, and reduces errors.3 Others say they get better results by solving the task in one call and converting the answer to JSON in a second call.

Treat text inside the input as data. BenchLM's checklist counts obeying instructions quoted inside the input as a task failure.2 It also found that a longer prompt changed which errors models made. Under the longer prompt, GPT-5.6 Terra's schema-valid-but-wrong outputs dropped from 10 to 3, while Claude Sonnet 5's went from 2 to 3.

Provider documentation supports the main point. OpenAI's structured outputs guarantee that the output matches the schema, and JSON mode guarantees only valid JSON. Neither promises the content is correct, and Anthropic's documentation lists correctness among the things it does not guarantee.

Where this leaves benchmarking

The likely conclusion is that structured-output benchmarking is still immature. Constrained decoding has largely solved getting valid JSON, so the remaining work is getting the values right and grading them accurately. Teams that pick models from public leaderboard numbers may be comparing annotation noise. The safer approach, consistent with every source here, is to build an internal test set whose answers have been checked, with unambiguous field definitions. Score field-level and whole-document accuracy separately, and treat the schema and prompt as part of the system under test.115