Inherent's Faraday Agent Beats GPT-5.5 at Replicating Research Papers

A small model with a large claim

Inherent, a London lab founded by Google DeepMind alumni, says its AI agent Faraday did better than two frontier systems, Anthropic's Claude Opus 4.8 and OpenAI's GPT-5.5, at one specific job: reproducing the findings of published scientific papers without being told the answers first.12 What makes the claim stand out is the size of the model underneath. Faraday runs on Qwen 3.6, an open model with 27 billion parameters, which is far smaller than the systems it was compared against.12 The work came out of a team of about a dozen people working in King's Cross, and it arrived only weeks after the company came out of stealth with a $50 million seed round.12

The headline is easy to repeat: a startup beat GPT-5.5. The research behind it says more than that. Inherent did not just put a smaller model up against larger ones. It trained a small model to plan and judge the work, and gave the actual coding to one of the large models it was being compared with. The real question is whether a thin "scientific judgment" layer on top of existing coding agents can beat those same agents working on their own. Inherent's own data says it can, by a modest margin, on a benchmark Inherent built itself.

What the paper actually reports

The technical details are in an arXiv paper titled "Training AI Scientists to Replicate Research." It describes Faraday as a post-trained version of Qwen3.6-27B. Training used a variant of GRPO, a reinforcement learning method, adjusted so that credit is assigned turn by turn, and the model was fitted with LoRA adapters.27 The authors present Faraday as a layer of scientific intelligence that sits above coding agents. In their framing, a 27-billion-parameter model directs a coding system they estimate at around 5 trillion parameters, and the pairing does better than the larger model alone.29

The testbed is a new benchmark called Replica. It has 310 figure-replication tasks taken from 100 machine learning and AI-for-science papers published between 1990 and 2026. In each task a figure is hidden, and the agent has to recreate it from the paper's description of its methods.30 The tasks are split into 242 for training and 68 held out for testing, and they cover areas including NLP, materials science, weather forecasting and structural biology.30

The results are useful mostly because they are specific. Faraday beat both Claude and Codex on 73% of in-distribution ML tasks and 60% of held-out AI-for-science tasks, according to Inherent's rubric-based judge. On the test split its average improvement was 6% over Claude and 8% over Codex.27 Another way to read the same data is the share of held-out tasks scoring above a threshold. On that measure Faraday reached 0.791, compared with 0.748 for Claude, 0.729 for Codex, and 0.554 for the untrained Qwen base.27 The authors also report that tuning Codex's prompt narrowed the gap only slightly.27

Inherent also brought in human evaluators. Across 41 rankings from 11 participants, people preferred Faraday's work to Claude's 80% of the time and to Codex's 88% of the time. The paper says both results are statistically significant.27 In a small scaled-up test on eight held-out tasks, each with up to eight hours and eight B300 GPUs, Faraday scored 0.843 to Claude's 0.800 and won five of the eight tasks. The authors admit they did not check the rubric judge against human ratings at that scale.27

Where the coverage agrees, and where it doesn't

Almost every outlet that covered the announcement pointed out that the benchmark was designed, run and scored by Inherent, and that no outside group has validated it.2325 One detailed write-up added that the rubrics were generated using one competitor's model and the judging was done with the other's. The paper discloses this openly, but it means the result has not been independently reproduced.30 Another outlet said the claim will stay unconfirmed until a third party runs Replica, or until Faraday is tested on a public benchmark such as MLE-Bench. It also asked how many of the 100 source papers might already be in Qwen's training data.21

The reporting splits on how much Inherent actually disclosed. One summary said the company published no numerical scores, no test papers and no methodology, and that the claim was therefore hard to assess.22 That doesn't match the arXiv paper, which includes win rates, threshold curves, training details and a human-preference study.27 The most likely explanation is that some early coverage was based on the TechCrunch interview, which focused on Hughes's account of how the system was built, before anyone had read the paper.12 The better-sourced reading is that Inherent disclosed a lot about its method. What's missing is independent verification. Even the funding details vary a little: one account says Index Ventures led the seed round with Radical Ventures participating, while another calls it a co-lead.2630

One point every outlet agrees on is that this is a replication result, not a discovery result. Faraday was asked to recreate known findings, not produce new ones, and Inherent does not claim any new science.2230

Why the architecture matters more than the leaderboard

Co-founder and chief scientist Edward Hughes himself downplayed the head-to-head win. He told TechCrunch that the more interesting part was how the team built the system.12 The goal was what Inherent calls "research taste": a sense of which experiments are worth running and how to design them. The company chose reinforcement learning over training agents mainly on descriptions of how science is done, betting that this would generalize better.12

The paper's breakdown of results backs up that framing. According to the authors, Faraday's advantage comes mostly from experimental depth and claim reproduction. On implementation fidelity and scientific integrity, it roughly matches Claude.27 So Faraday is not a better coder. It decides better what to test and how thoroughly to test it. That fits Inherent's decision to use GPT-5.5 Codex for coding rather than build its own tool, which the company compares to scientists using existing software.12 Faraday also seems to adapt to new tools. It was trained with GPT-5.4-mini as its coding agent and switched to the stronger GPT-5.5 Codex at test time without retraining.23

This needs to be read alongside the commercial reality: Faraday's win depends on OpenAI's model doing the coding. Several outlets noted that the comparison measures a whole system, including the base model, the training recipe and the tools, rather than Qwen 3.6 by itself.22 I'd say that makes the result more interesting, not less. If small, specialized planners can reliably get more out of frontier coding agents, then the value of research automation may shift from whoever owns the biggest model to whoever trains the best judgment on top of it.

The wider research context

Faraday is part of a broader trend toward agent designs that do better than raw model scale on scientific work. In May, Google DeepMind's AI Co-Mathematician scored 47.9% on FrontierMath Tier 4, beating GPT-5.5 Pro's 39.6%. It did this with a multi-agent setup coordinated by Gemini 3.1 Pro, not a new foundation model.16 More recently, researchers turned published papers into agents that answer questions using the papers' code. Those agents scored 91.2% on 300 questions, compared with 80.3% for Claude working directly from the code.18

Earlier benchmarks show how hard this problem has been. ResearchGym tested a GPT-5-based agent on real AI research tasks. It beat the provided baselines in only one of 15 runs and completed about a quarter of sub-tasks on average. The authors pointed to recurring problems such as impatience and overconfidence in weak hypotheses.19 Those are the same weaknesses that "research taste" training is supposed to fix. That's a good reason to take Inherent's approach seriously while the numbers wait for outside confirmation.

The frontier model Faraday beat has also already been replaced. OpenAI's GPT-5.6 Sol came out in July. It is ahead of GPT-5.5 on every like-for-like category on the BenchLM comparison and well ahead on FrontierMath.8 Faraday's design, which treats the coding model as a swappable tool, could in principle benefit from that upgrade rather than be overtaken by it. That hasn't been shown yet.

The talent story behind the lab

Inherent's founders include Tantum Collins, Edward Hughes and Louis Kirsch, all formerly of DeepMind, along with infrastructure lead Kaloyan Aleksiev. The company is set up as a public benefit corporation.30 It is one of a growing number of DeepMind spinouts. Investors are reported to be competing for departing researchers after Demis Hassabis stepped back from running Google's AI operations in August, and after Google moved more of the division's center of gravity to California.9 Inherent plans to grow to about 20 to 25 people by the end of the year, and TechCrunch suggests that could appeal to unsettled DeepMind staff.12

The verdict

Read together, the coverage supports a narrow conclusion. On a benchmark Inherent designed itself, Faraday has shown a consistent but modest advantage in replicating papers. The most important finding is that judgment, not raw coding ability, accounts for the gap. The next step for the field is clear: an independent run of Replica, or results on a public benchmark. Until then, Faraday is good evidence that how an agent is designed can matter as much as how big its model is, but not yet proof that it does.