Looped MoE Scaling Laws Show Half-Size Models Match Reasoning Scores

What happened

Three Meta researchers, Yanbei Chen, Anirudh Goyal and Raghuraman Krishnamoorthi, have published a paper titled "Scaling Laws for Looped Mixture of Experts" on arXiv.10 It describes "Loop Scaling Laws," which the authors call the first scaling law to model recurrence and sparsity together with model size and training data.10 Recurrence here means running the same layers more than once. The central result is easy to quote. When training compute is held equal, a looped MoE whose loop count comes from the law performs about as well on reasoning benchmarks as a non-looped MoE roughly twice its size. The looped model can also spend extra loops at inference time to scale its test-time compute.1015

The paper was posted September 30.11 Coverage since then has mostly treated the two-to-one size ratio as the headline.1117 That figure holds up, but read closely it is a narrower result than the shorthand implies. What it really shows is a saving in stored parameters, not a saving in compute.

Two efficiency tricks, finally modeled together

Looping and MoE sparsity attack the cost of large language models from different angles. A looped model adds computational depth without adding parameters. An MoE model adds total capacity while keeping the number of parameters active for each token fixed.10 Until now, scaling laws dealt with each technique separately.10 Tech Times put the practical gap plainly: an engineer combining the two had no principled way to choose a loop count or sparsity level, or to predict the resulting loss, before training began.11

The paper's core tool is what the authors call a "bounded, sparsity-conditional recurrence mapping." It estimates how many extra effective parameters looping is worth, and how that gain changes with sparsity.10 "Bounded" means each added loop helps less than the one before, until returns flatten. "Sparsity-conditional" means the size of the payoff depends on how sparse the model is. An alphaXiv summary of the authors' experiments says loss drops as passes are added and then levels off, and that dense models level off sooner than models with more experts.12

The authors make three claims for the laws. They predict held-out loss more accurately than earlier alternatives, and they reduce to the standard dense and MoE scaling laws as special cases.10 They can also guide model design when both compute and memory are limited.10 On downstream tasks, the paper credits sparsity with roughly 3x efficiency in active parameters and looping with roughly 2x efficiency in total parameters on reasoning tasks. Combining the two pushes the frontier further.19

Reading the headline number carefully

Coverage agrees on the topline but differs in how much detail it gives. Tech Times illustrated the result with a hypothetical: a 10-billion-parameter looped MoE might match a 20-billion-parameter conventional MoE.11 One practitioner blog went further, calling it half the parameter footprint for the same compute and the same reasoning performance.17

The actual trillion-token experiment is smaller in scale and closer in outcome. The alphaXiv breakdown describes the looped model as five loops, 0.3 billion active parameters and 1.3 billion total, trained on about 3 trillion tokens. Its reasoning average was 36.5%.12 The baseline was a non-looped MoE with 0.6 billion active and 2.9 billion total parameters, trained on about 6 trillion tokens, which averaged 36.35%.12 Both runs used about 1.5×10²² training FLOPs.12 The same summary notes that the averages are computed from BBH and GSM8K scores, and that the looped model does not beat the baseline on both tasks individually.12

Three points follow from this, as a matter of interpretation. First, "matches" is accurate, but the margin is a fraction of a point on a two-benchmark average. This is parity, not a decisive win. Second, the models are small: billions of total parameters, not frontier scale. The 10B-versus-20B framing is an extrapolation and was not tested. Third, matched compute meant the looped model saw only half as many training tokens. Every extra pass through the shared block costs FLOPs, so the saving shows up in stored weights, not in arithmetic.

That third point is the key to judging what the result is worth. SMELT, a separate paper on looped MoE transformers, spells out the cost: a 12-layer model looped to 24 executed layers stores half the weights but spends roughly a 24-layer model's per-token FLOPs and needs its full KV cache.13 A looped MoE is therefore cheaper to hold in memory, not necessarily cheaper to run. For deployment on memory-limited hardware, that is still a real advantage.

Why sparsity changes the looping math

The more interesting scientific claim is that looping pays off more when it is combined with sparsity. Earlier work on dense looped models was discouraging. Researchers at the Technical University of Munich ran 116 pretraining runs at fixed depth. They found that a block looped r times is worth only about r^0.46 unique blocks, so each extra loop raises validation loss at matched training compute.2 In one of their examples, a 410M looped model at four loops performs like a 580M non-looped model but costs as much to train as a 1B model.4

The Meta paper's sparsity-conditional term is partly an argument that MoE routing gets around that penalty. Tech Times explains the mechanism as routing divergence. On a second pass, the router sees a representation that the first pass has already transformed, so it can pick different experts, and the second pass does work the first did not.11 The outlet acknowledges that the Meta paper leaves this mechanism mostly implicit and draws on outside research to explain it.11 SMELT provides some direct evidence. Its analysis found that on the second pass the router keeps a core set of experts while varying the others, and that the second pass writes a larger update into the residual stream than the first.18

The two findings fit together. Looping a dense model gives back capacity weakly and predictably. Looping a sparse model appears to give back more, because the shared block does not behave identically on each pass.

A crowded, converging field

The Meta paper is part of a burst of 2026 research on looped models. A citation tracker for the Ouro looped-model project lists dozens of related papers this year on stability, scaling laws and looped models at scale, including the Meta paper.9 SMELT was submitted September 1.18 Its recipe loops the middle half of the layers twice while matching the baseline on FLOPs, parameters and KV cache. Across four scales and four sparsity levels it saved 6.8% to 18% of training compute at the frontier.13 It also found two loops better than three or four.13

Other groups are working on the instabilities that come with repeated passes. DeepLoop argues that looped transformers need residual-scaling rules that account for repeated visits to the same weights, not just the nominal layer count.8 The STARS framework addresses a common failure in which performance peaks at some loop depth and then collapses as more loops are added.3 Parcae treats looping as a dynamical system. It reports perplexity gains of up to 6.3% from constraining the injection parameters.9

The overall picture is consistent. Looping gains are real, bounded and easily lost to poor training. SMELT's finding that two loops is optimal and Meta's law-selected five loops at small scale are not necessarily in conflict, because the Meta law makes the optimal loop count depend on sparsity and budget.1213 Whether the two results agree once their setups are aligned is something the field has yet to check.

Caveats worth weighing

Several limits apply. The reasoning gains concentrate on tasks that involve iterative, multi-step work, and Tech Times notes that gains on plain language modeling are more modest.11 The outlet also reports that the trillion-token validation relied on proprietary data and architectural details, which makes outside comparison harder.11 Like most scaling laws, these are empirical fits and may need recalibrating when routing schemes, normalization or hardware change.11 The TUM work points the same way: a training choice such as truncated backpropagation lowered its recurrence exponent to 0.38, while hyperconnections raised it to 0.65.4 Fitted coefficients for looped models clearly depend on recipe.

The verdict

Tech Times compared the work to Chinchilla, the scaling study that set compute-optimal training for dense transformers.11 That comparison is premature, but the direction is right. The paper's lasting contribution is less the two-to-one headline than its framework: a model that treats loop count as a design variable whose payoff depends on sparsity, and that reduces to familiar laws at the extremes.10

What the paper has shown so far is reasoning parity on BBH and GSM8K at the low-billions scale, with half the stored parameters and the same training compute.12 That is a useful result for anyone limited by memory, and it is a credible sign that looping and sparsity work better together than either does alone. It does not yet show that a 10-billion-parameter model can replace a 20-billion-parameter one in production. That needs independent replication at larger scale, and a breakdown of inference cost that counts every extra pass.