Looped MoE Scaling Laws: What the 2x Efficiency Claim Hides

Two papers, one architecture bet

Within a single month, two research efforts tried to answer the same question. Can language models get more out of their parameters by running some layers more than once, and does that still help when the model already uses Mixture-of-Experts (MoE) sparsity?

The more prominent effort comes from Meta AI. Yanbei Chen, Anirudh Goyal and Raghuraman Krishnamoorthi posted a paper to arXiv on September 30 proposing scaling laws for "looped" MoE models 41. The paper models loss as a joint function of model size, training data, the number of recurrent passes, and expert sparsity 1.

About a month earlier, on September 1, a separate preprint called SMELT examined looping on MoE Transformers with an explicit focus on fair comparisons 2.

Read together, the two papers make a stronger case for looping than either does alone. They also show that the headline "2x" figure needs careful reading.

What Meta claims

The Meta paper uses a bounded recurrence formulation. The authors say it predicts held-out loss for looped models more accurately than earlier alternatives 1. They also say it reduces to the familiar dense and MoE scaling laws as special cases 1.

In downstream tests, the authors attribute distinct benefits to each axis [1]:

• Sparsity delivers roughly 3x efficiency in active parameters.
• Recurrence delivers about 2x efficiency in total parameters on reasoning tasks.
• Combining the two pushes the performance frontier further.

The practical claim is that these gains hold at trillion-token scale. At matched training compute, a looped MoE with a law-derived number of loops matched a non-looped MoE about twice its size on reasoning benchmarks 1. Tech Times called this the first mathematical framework to govern weight-sharing loops and expert routing together. It framed the result as a recipe that lets engineers choose loop counts before training begins, along with adjustable test-time scaling 4.

The fine print on "2x"

The comparison is specific. Meta's 2x figure refers to total-parameter efficiency at matched training compute 1.

That is not the same as saying the looped model is twice as cheap to run. A looped model reuses its weights, so it needs less memory for parameters. But each additional pass through a shared block is extra computation for every token.

Our reading is that the savings mainly come from parameter count and memory. Per-token inference compute probably rises roughly in line with the number of loops. Meta's own mention of test-time scaling points the same way: the loop count works as a dial that trades compute for quality 14. Neither the abstract nor the Tech Times coverage gives a per-token inference cost comparison. Teams that are limited by serving FLOPs rather than memory should treat "half-size" carefully.

SMELT's stricter test

SMELT goes after exactly this confound. Its authors argue that most looped-Transformer evaluations compare models of fixed size. That setup mixes any real architectural advantage with the extra FLOPs that looping adds 2.

Their fix is to match three budgets at once against the unlooped baseline [2]:

• per-token FLOPs
• total non-embedding parameters
• KV cache

The recipe they settled on after ablations loops the middle half of the layers twice. That is where the name comes from: Sparse MoE Transformer, middle layers Loop Twice 23. They scaled it across four sizes, up to 54B non-embedding parameters, and fit a separate Chinchilla-style law for each architecture 3.

The gains under these stricter conditions are smaller but still real:

• Training savings: SMELT's loss falls faster with compute, saving 6.8–18.0% of training FLOPs along the compute-optimal frontier 23.
• Downstream results: the advantage on benchmarks exceeds what validation loss would predict. It is largest on code and grows with sample length and the number of in-context examples 2.
• Mechanism: the second pass weakens the attention sink and shifts attention toward content-relevant tokens. The authors present this as an inductive bias rather than just added capacity 23.

Where they agree and where they diverge

Both papers conclude that looping and MoE sparsity complement each other rather than overlap. Both also find that the benefit shows up most clearly on reasoning-heavy or structured tasks: reasoning benchmarks for Meta 1, and code and long-context tasks for SMELT 2.

The difference is in what each holds constant:

	Meta	SMELT
Held fixed	Training compute	Per-token FLOPs, parameters, KV cache
Result	Large parameter efficiency gain	Modest compute efficiency gain

These are not contradictory findings. They describe different trade-offs.

Why it matters

For anyone designing models under memory limits, Meta's laws offer something new: a principled way to pick loop counts and sparsity levels before spending a large training budget 14. SMELT works as a useful check on that promise. Once compute is held equal per token, looping still pays off, but the gain is measured in percentages rather than multiples 2.

The takeaway is that looped MoE appears to be a real improvement, not an accounting trick. The size of the improvement depends on which resource is scarce: memory and parameters, or inference compute.