Scaling Laws Hold in 4M-Parameter Models When Tuned Hard Enough

What happened

For years, a common belief in language-model research has been that the smallest models tell you little about the largest ones. A paper titled "Small-Scale Experiments: Are We There Yet?" disputes that belief. Its authors report that scaling laws, the power-law curves that predict how pretraining loss falls as models and data grow, show up in models with as few as 4 million parameters. The catch is that those models need much more hyperparameter tuning than they usually get.18

The paper first went up on arXiv in mid-August. Coverage attributes it to researchers at FAIR at MSL Meta and New York University, with Nicholas Lourie listed as lead author.36 A revised version drew new attention in late October. That version tells researchers to spend more tuning runs on cheap small models and fewer on larger ones. It also warns that even a well-tuned prediction gets weaker the further it is pushed past the range of models that were actually tested.18

Co-author Sanae Lotfi stated the practical point plainly when announcing the work. Small-model fits have often looked unreliable, she said, which convinced practitioners that experiments on fairly large models were unavoidable. With enough tuning, the team got scaling laws down to 4M-parameter models that train in under an hour on a single GPU.1

The core finding: the law was there, the tuning wasn't

The paper's main claim is that hyperparameters are the hidden variable. In the authors' account, small models are highly sensitive to settings like learning rate and batch size, and that sensitivity fades as models grow. Because a clean scaling law only appears on the "fully tuned frontier," a typical small hyperparameter grid simply never reveals it.28

The clearest evidence is an experiment that varies only the tuning budget. With 4 configurations per scale, no scaling law appears at all. With 16, the law is still too noisy to estimate. At 64, the shape is clearly visible but extrapolates poorly. Only at 256 configurations per scale does the fitted law become accurate.2 The authors argue that tuning matters more than any other choice in the scaling-law recipe. By their account, details such as how parameters are counted or whether two exponents are tied barely change the result. Per-scale tuning and learning-rate decay, by contrast, sharply reduce prediction error: tuning cut test error by 50%, and decaying the learning rate cut it by 98%.2

The setup is designed to be a real test of extrapolation. The team fit its laws on models between 4M and 34M effective parameters, used 67M to 134M models for validation, and kept 268M models as a held-out test set.2 It used the familiar Hoffmann-style (Chinchilla) formula, which models loss as an irreducible floor plus separate power-law terms for parameters and data.14 One summary describes the training effort as thousands of models spanning roughly 4M to 268M effective parameters.3

Why small models are hard to tune

The paper also offers an explanation, and this is where it matters most for architecture research. The authors argue that as parameters and data grow together, good hyperparameter settings fill a larger share of the search space. The loss surface over hyperparameters also becomes lower-dimensional as scale increases.214 One independent review puts numbers on this: a measure of effective dimensionality falls from about 2 to 3 at 4M parameters to roughly 1 at larger sizes. In practice, that means tuning large models comes down to finding one main knob.4 Distribution plots in the paper show the same pattern. Results at 4M parameters are widely spread, while results at 268M cluster tightly around the optimum.3

If this holds, it explains a common experience in labs. A small model looks erratic, so researchers distrust it and move to a bigger one. The paper's reading is that the small model was never erratic. It was under-tuned, in a regime where under-tuning is costly.10

The architecture test: pre-norm versus post-norm

To show the method is useful and not just tidy, the authors revisit a settled transformer design question: whether layer normalization should go before or after each sublayer. Using only small-scale runs, they reach the known large-scale answer that pre-norm beats post-norm.4 Their diagnostics show post-norm is more sensitive to hyperparameters and has a more complicated loss surface. Fitted scaling laws also indicate pre-norm scales more efficiently.3 The paper calls pre-norm's law excellent and post-norm's merely reasonable, which itself reflects post-norm's touchier tuning.2

This is the most important part of the story for anyone comparing LLM designs. The authors present their method as a way to compare architectures or training methods with each other, not as a tool for forecasting the exact loss of a trillion-parameter model.3 They also give a practical diagnostic, called the noisy quadratic limit, for checking whether a hyperparameter search has actually reached the frontier. When the expected pattern fails to appear, the search probably missed the optimum.34

Where the coverage agrees, and where it doesn't

All the coverage agrees on the headline result and on the 256-configuration threshold.13410 The difference is in tone. The alphaXiv summary is enthusiastic. It says the best-tuned results fit the power law "almost perfectly" and that improvements found at 4M to 34M parameters accurately predicted the larger validation models.3

The Pith review is more careful, and its objections deserve attention. It notes that the tuning budget was uneven across scales. The smaller training scales got roughly 826 to 934 configurations each, while the held-out scales got only 64 to 128. The review calls this a real confound for the claim that 256 configurations are enough, though it does not regard the result as circular.4 It also questions the geometric explanation. The fitting window behind the dimensionality estimate widens from 7% of runs at 4M parameters to 50% at 134M. If the apparent drop in effective hyperparameters comes from that changing window rather than from the actual loss surface, the explanation falls apart.4 Even so, the reviewer calls the main result real and practical, and credits the authors for being candid about the limits of extrapolation.4

The authors make the same point themselves. When they compared independent scaling-law fits, each built from 128 runs at 4M to 34M parameters, the fits agreed near the data but diverged when extended far out. The divergence comes from estimates of irreducible error that vary widely from sample to sample. Their summary: the law exists, but the statistical power to extrapolate it far does not.2

Not entirely new, but newly practical

The central idea has precedents. A 2022 EMNLP Findings paper by Ivgi, Carmon and Berant studied models as small as 10K parameters. It also concluded that revealing scaling laws requires careful hyperparameter tuning and repeated runs, and that this overhead eats into the compute savings.9 The new paper's contribution is to put numbers on how much tuning is needed, explain why it shrinks with scale, and argue that the cost is affordable when each run takes under an hour on one GPU.2

Not everyone agrees on the cost. One commenter observed that at 4M parameters, the cost of the hyperparameter sweep itself can dominate and distort the scaling signal.10 This analysis holds that the authors still have the better argument. Hundreds of hour-long single-GPU runs cost far less than a single serious run at hundreds of millions or billions of parameters, and the Pith review makes a similar point.4 The real change is in how costs are accounted for: search effort should be budgeted and reported as a cost in its own right, not hidden in a footnote.4

The reasoning caveat: loss isn't capability

What these laws predict is pretraining loss, and the authors are explicit about that. A perfect loss law still does not predict capabilities, which have proved much less predictable. The authors argue that loss fully captures capability only when the pretraining data is held fixed.2 That condition matters for anyone hoping small models can forecast reasoning ability. The method can tell you which architecture reaches lower loss on the same data. It cannot tell you when a particular reasoning skill will appear.

Separate work on tiny models makes the same point from another direction. A 2026 study of vision models ranging from 22K to 4.7M parameters found that error rates followed approximate power laws. But the identity of the errors changed a lot: only about 35% of errors overlapped between the smallest and largest models.15 Smaller models in that study concentrated on easy classes and largely gave up on hard ones, so the authors argued that validation has to happen at the size you plan to deploy.1512 It is a different domain and a different metric, but it supports the same conclusion. A smooth aggregate curve can hide large changes in what a model actually gets right.

Why it matters

The most useful takeaway here is not that 4M-parameter models can stand in for frontier systems. It is that a lot of past evidence that small-scale experiments "don't transfer" was probably evidence of under-tuning. That matters for who can do credible architecture research. If small-scale comparisons hold up when tuned rigorously, academic labs with a few GPUs can test design choices like normalization placement in ways that predict behavior at scale, rather than deferring to groups with large clusters.23

The limits are just as clear. Extrapolation across many orders of magnitude remains statistically fragile. The 256-configuration figure carries a sampling confound, and the geometric explanation still needs independent replication.24 The fair reading is that this paper makes small models much more useful for ranking architectures and training methods. It does not make them reliable for predicting the exact loss of a frontier model, and it says little about reasoning ability.