OpenAI's Astra Benchmark Edits Muddy Its Claim Over Anthropic

What happened

OpenAI launched GPT-6 Astra on September 3 with a blog post pitched as evidence the model had overtaken rivals, including Anthropic's newly released Claude Fable 5.1, and might even qualify as a step toward artificial general intelligence 3. Within days, reporting from Fortune showed that several of the benchmark figures in that launch post had quietly changed — some making Astra look stronger, others making Anthropic's comparison numbers look weaker, before a number of those edits were partially reversed 1112. NewsBytes and explainx.ai picked up the same pattern, framing it as OpenAI tweaking Astra's benchmarks after the fact in ways that tilted the comparison against Anthropic 11713.

The most-cited example is Astra's internal hallucination rate. It was published at 4.2%, versus 12.2% for predecessor GPT-5.6 Sol. An archived version of the page later showed those figures cut to 2% and 9.4% respectively, before OpenAI restored them close to the original numbers 1113. A similar back-and-forth hit FrontierMath Tier 4 scores attributed to Sol and to Anthropic's Fable 5.1: Fable's listed score dropped from 87.8% to 78% before settling back around 83%, temporarily making Astra's math lead look considerably larger than it ultimately was 111217. OpenAI also briefly listed a Sol cybersecurity score of 11.5% that turned out to reflect a reasoning tier not commercially available, versus a more representative 5.5% 1113. Not every change favored OpenAI, though — two Anthropic scores on HealthBench Professional actually moved upward across revisions 11.

OpenAI's own explanation, given to Fortune, was that evaluation results carry "noise within a few percentage points" depending on checkpoint, scaffold and evaluation run, and that the company made "fixes" to reflect its best estimate of performance 11. The company also gave shifting explanations for why the post was pulled and republished in the first place — first citing a content-management bug, then an internet outage — while maintaining the delay had nothing to do with the benchmark numbers 11.

The Anthropic backdrop

The timing sharpened the story: Anthropic had released Claude Fable 5.1 and Claude Mythos 5.1 just two days before Astra's launch, positioning Fable 5.1 as a major jump over Fable 5 in coding, knowledge work and long-running agentic tasks, with case studies of it diagnosing bugs that had stumped engineering teams for years 152. OpenAI's launch table placed Astra ahead of Fable 5.1 on several selected evaluations, including FrontierMath, GPQA Diamond and Terminal-Bench Science 14. But Fable 5.1 scored higher than Astra on Humanity's Last Exam with tools, 65.0% to 57.2% 1415, and Anthropic's own materials flag a standard error of roughly ±3.5 to 4.5 points on Terminal-Bench Science, a caveat that undercuts confidence in small gaps either way 15.

Independent testing complicates OpenAI's framing further. Artificial Analysis's Intelligence Index put Astra at 61, tied with its own predecessor Sol and five points behind Fable 5.1's 66, with Meta's Muse Spark 1.3 also ahead of Astra 16. On Artificial Analysis's Coding Agent Index, Astra scored 67 against Fable 5.1's 70 16. That outlet did credit Astra with a large improvement on hallucination as measured by AA-Omniscience, where the rate fell from 92% to 51% without the usual accuracy trade-off 16.

Astra's most eye-catching number, a near-perfect 99.9% on ARC-AGI-3, comes with a documented asterisk. The ARC Prize Foundation's own published results show that figure was achieved using OpenAI's "Provider Adapter" harness, which preserves reasoning state between requests and compacts long contexts; under a standard, provider-neutral harness, Astra scored 62.7% 20. The-Independent quoted ARC Prize's Greg Kamradt describing Astra as reaching human parity on the benchmark 18, while thenewstack.io and explainx.ai both stressed that the harness-dependent gap between 62.7% and 99.9% is the more important story than the headline score itself 1913.

Where the reporting agrees

Every outlet that dug into the launch-day numbers — Fortune, NewsBytes, and explainx.ai — agrees on the core sequence of events: OpenAI published benchmark figures for Astra, several of those figures changed within days without a clear changelog, and the changes moved in a direction that generally flattered Astra relative to Anthropic's models, at least initially 111211713. There is also agreement that the hallucination-rate figure specifically moved from 4.2% down to roughly 2% and then back toward 4.2%, and that Fable 5.1's and Sol's FrontierMath scores were temporarily lowered before being restored 111213. Multiple sources also converge on the point that OpenAI's cybersecurity comparison briefly relied on a reasoning tier unavailable to paying customers 1113. Separately, independent benchmarking coverage and Anthropic's own materials agree that Fable 5.1 leads Astra on at least one general-intelligence composite score and one coding-agent index, even as OpenAI's own comparison table shows Astra ahead on other individual evaluations 161415. On the ARC-AGI-3 result, ARC Prize's published data and thenewstack.io's analysis agree that the 99.9% figure depended on a specific OpenAI harness rather than reflecting the model in isolation 2019.

Where it doesn't

The clearest divergence is in framing and emphasis rather than in the raw numbers themselves. The Financial Times reported OpenAI's own claim at face value — that Astra had overtaken Anthropic and might be considered a step toward AGI — without dwelling on the subsequent revisions 3, and The Independent similarly led with OpenAI's framing that Astra beat all rivals, including Claude and Gemini 18. NewsBytes and explainx.ai, by contrast, led with the benchmark instability itself, treating the changing numbers as the story rather than a footnote to it 11713. Trendingtopics.eu's coverage of Artificial Analysis's independent testing goes further in the opposite direction, arguing Astra actually trails Anthropic and Meta on the two headline composite indices, a conclusion that sits uneasily next to OpenAI's own launch-table comparisons showing Astra ahead on several named evaluations 1614. There's also a live discrepancy over whether the changes were, on net, a one-way inflation of Astra's standing or something closer to ordinary methodological noise: OpenAI insists the edits were fixes reflecting evaluation noise across checkpoints and scaffolds 11, while explainx.ai's own accounting notes the hallucination figure moved in both directions rather than staying inflated, calling that pattern more consistent with genuine uncertainty than deliberate manipulation 13. No source claims OpenAI fabricated results outright; the disagreement is over how much weight to put on the timing and direction of the edits.

Which account holds up

The documented, snapshot-by-snapshot evidence from Fortune's tracking of the archived blog post is the most concrete material here, and it supports a narrower conclusion than either OpenAI's launch framing or the more alarmed "OpenAI cooked the numbers" framing found in some secondary coverage. The figures did move in Astra's favor immediately after launch and were later partially reversed, which is consistent with real methodological churn rather than a single clean act of deception — but it is also true that OpenAI never clearly disclosed the changes as they happened, and a spokesperson's shifting explanations for the post's initial retraction don't inspire confidence in the process. Meanwhile, the independent Artificial Analysis and ARC Prize data make clear that Astra's advantage is real but uneven: strong in computer use, cybersecurity, and specific math and science evaluations, while trailing Fable 5.1 on broader intelligence and coding-agent composites. The fairest reading of the coverage as a whole is that Astra represents a genuine capability advance for OpenAI, but the "decisive win over Anthropic" narrative depends heavily on which benchmarks, harnesses and snapshot-in-time figures one chooses to cite — which is exactly the vulnerability this episode exposed.