Scaling Laws Get Cheaper as Psychometric Method Cuts Queries 99%

A cheaper way to see the future of a model

AI labs spend heavily to guess how a model will behave before they commit to training it at full scale, and that habit has made scaling-law prediction part of how frontier models get built. A team at Stanford now says much of that prediction work can be done for a small fraction of the usual cost. Their framework, Item Response Scaling Laws (IRSL), takes ideas from psychometrics and educational testing and uses them to forecast how large language models will scale. According to the researchers, it cuts the compute needed for that forecasting by as much as 99 percent.11

The study was accepted at the International Conference on Machine Learning. Sanmi Koyejo, an assistant professor of computer science, is the senior author, and a co-author identified as Truong worked with him.11 The method itself is easy to describe. The questions it raises about where scaling research is going are harder, and they become clearer once IRSL is set beside a wave of recent work on reasoning models, test-time compute and new LLM architectures.

How IRSL works

The usual workflow has stayed the same for years. Developers train many small models under different settings, fit power laws to how those models perform, and extrapolate to a far larger model they have not yet trained.18 That approach traces back to the 2020 OpenAI scaling-law work and to DeepMind's 2022 Chinchilla paper, which found that most large models of the time had seen too little training data for their size.14

The Stanford group points out that this extrapolation step is costly in its own right. A traditional scaling study can ask 10,000 or more benchmark questions. Once that is multiplied across every small model and every repeated sample, the total can reach about 10 trillion queries.11 IRSL handles this the way adaptive standardized tests such as the SAT handle students. It treats each model as a test-taker, presents harder items after correct answers, and estimates ability from far fewer responses. The researchers say it matches or beats the predictive accuracy of the standard approach with as few as 50 questions.11

The team also argues that the leaner method can be the better one. Truong says that in some cases less computation actually improves the predictions, and Koyejo calls the result a counterintuitive mix of a stronger signal for less work.11 Koyejo expects the largest effect in universities, where training costs often rule out large experiments, though he says well-funded private labs could gain too.11

These efficiency figures come from the institution's own announcement. Until outside groups reproduce them across different model families, they are best read as a strong early claim and not yet a settled result.

Part of a broader push to rethink prediction

IRSL is one of several efforts to update the measurement side of scaling research. A paper called Neural Neural Scaling Laws (NeuNeu) targets a well-known weakness: pretraining loss follows smooth power laws, but scores on individual downstream tasks do not. Some tasks improve steadily, some level off early, and some get worse as models grow.19 NeuNeu does not impose a fixed formula. It uses a transformer that reads full token-level loss distributions and reported a 44 percent reduction in downstream prediction error across 66 tasks, with mean absolute error falling from 3.56 percent to 1.99 percent compared with logistic scaling laws.19 It also predicted the behavior of a non-transformer hybrid architecture even though it had been trained only on transformer runs.19

IRSL and NeuNeu take nearly opposite routes. IRSL gets more from fewer, better-chosen evaluation items. NeuNeu gets more by keeping detail that averaged loss throws away, at the price of a black-box predictor whose authors concede it is hard to interpret.19 Both start from the same diagnosis: the classic recipe of fitting a curve to aggregate numbers is too blunt and too costly for what developers now need from it.

This is not a new complaint. Back in 2024, Gary Marcus argued that the word "scaling" had lost its meaning. In his view it had shifted from a precise prediction to a loose hope that turning some dial would make things better.20 The current research can be read as an attempt to restore that precision with better statistics, not to give up on it.

Why reasoning makes better prediction more urgent

Scaling has also become harder to predict because it now runs along several axes. Practitioners and commentators commonly describe three regimes: pretraining, post-training methods such as reinforcement learning and distillation, and test-time compute, where a model works through a problem at length before answering.1510 The test-time axis behaves like a scaling law of its own, with roughly log-linear gains as inference compute rises.9 The finding repeated most often is that a smaller model given more thinking time can match one about 14 times its size that answers immediately.29

The newest work on reasoning paints a messier picture, and this is where sources disagree most. Popular explainers tend to call test-time compute the field's main new lever.1310 The research literature is more cautious. A Carnegie Mellon study of general-purpose LLM agents found that neither longer interaction nor parallel trajectory sampling reliably improved results in realistic settings. The authors attribute this to a "scaling plateau" and a "verification gap."8 Other work reports that models given large reasoning budgets can "overthink" and drop answers that were already correct.2 One 2026 study found reasoning models used about 6,780 tokens on basic math problems while scoring worse than standard models that used about 378.10 A new benchmark, SELF-POT, charges every model call in dollars, including the calls used for selection and critique. Its authors find that whether cheap models plus extra inference can stand in for a stronger model depends on the domain, the selection rule and how failures are handled.6

Our reading is that this is exactly the situation in which a method like IRSL matters most. When capability depends on parameters, data, post-training and inference budget together, the number of configurations worth testing grows quickly. Any technique that cuts the cost of each prediction makes it possible to test more of them, and the uneven, domain-specific results from reasoning research mean those tests are needed.

Architecture brings its own scaling laws

The same trend is showing up in model design. On September 30, Meta AI researchers released scaling laws for Looped Mixture-of-Experts models. These models combine weight sharing, where a block is run repeatedly, with expert-routing sparsity.1 They report that sparsity provides about three times the active-parameter efficiency, that looping provides about twice the total-parameter efficiency on reasoning tasks, and that combining the two pushes the frontier further.4 In a trillion-token test, a looped MoE with 0.3 billion active and 1.3 billion total parameters matched the reasoning performance of a non-looped MoE with 0.6 billion active and 2.9 billion total parameters at the same training compute. Part of the price is extra inference compute.3

The looping result also connects back to reasoning. Because the loop count is set at inference time, one trained checkpoint can be run with more loops for hard queries and fewer for easy ones.1 Raising recurrence from one to five passes lifted overall benchmark performance from 36.6 to 46.9.3 The framework also reduces to standard dense and MoE scaling laws as special cases.4 Coverage says looping gains are real but limited. Returns level off after a few loops, and one production looped model reportedly found two loops compute-optimal at its scale.1 The Meta team also notes that the trillion-token validation relied on proprietary data, which makes direct comparison harder.1

The Meta paper's main contribution, much like IRSL's, is a way to choose settings before training begins: how many loops to run and how sparse to make the model.1 Scaling laws are increasingly being used as design tools for specific architecture choices, not only as broad statements about bigger models.

What it adds up to

The sources mostly agree that scaling has diversified rather than ended. One explainer says the original curve is flattening along its first axis and the field has responded by finding new axes to climb.16 Others say the gains per unit of compute are shrinking at the frontier, and that some abilities, such as analogical reasoning, do not improve reliably with size.14 The two views fit together. Raw pretraining scale is giving less, and capability now comes from a mix of training choices, sparse and looped architectures and inference-time reasoning.

In that setting, the work that matters most may be the less visible measurement work. Each new axis is useful only if developers can predict its payoff before paying for it. IRSL's promise of comparable forecasts from a tiny share of the queries, NeuNeu's more precise task-level predictions and Meta's loop-and-sparsity rules all address that need. If IRSL's numbers hold up beyond its own experiments, its biggest effect could be letting academic labs, which cannot afford to run thousands of models, take part in scaling research again.11