Looped Mixture-of-Experts: Scaling Laws, Gains, and Limits

What happened

Two recent papers put a spotlight on an architecture that combines two efficiency techniques: looped Transformers and mixture-of-experts (MoE) sparsity. Looped models run the same Transformer blocks repeatedly. That adds effective depth without adding parameters. MoE models route each token to a subset of experts, so only a fraction of the network is active at any time.

The first paper, "Scaling Laws for Looped Mixture of Experts," was submitted on September 30, 2026. It proposes a single scaling law that jointly models how model size, training data, the number of recurrent passes, and MoE sparsity affect language-model loss 1. Tech Times attributes the work to Meta AI researchers Yanbei Chen, Anirudh Goyal, and Raghuraman Krishnamoorthi. The outlet calls it the first mathematical framework to govern both weight sharing through loops and expert routing at the same time 3.

The second paper, "Looping Beyond Twice" (arXiv 2610.01153), focuses on a narrower problem. Looping tends to stop helping after about two iterations, and the paper proposes a training recipe called LOOM to push past that limit 2.

The headline result: half the size, similar reasoning

The scaling-law paper's main practical claim concerns matched training compute. Under that condition, a looped MoE whose recurrence depth is chosen using the fitted law matches a non-looped MoE roughly twice its size on reasoning benchmarks. The authors say this holds at trillion-token scale 1. Tech Times highlights this result. It frames the work as a way for engineers to choose optimal loop counts before training begins, rather than finding them by trial and error 3.

The paper also separates the contributions of the two techniques:

• Sparsity delivers about 3x efficiency in active parameters 1.
• Recurrence delivers about 2x efficiency in total parameters on reasoning tasks 1.
• Scaling both together pushes the performance frontier further 1.

The authors also report two technical results. Their bounded-recurrence formulation predicts held-out loss more accurately than earlier alternatives. It also reduces to the standard dense and MoE scaling laws as special cases, which suggests it extends existing theory rather than replacing it 1. Both the paper and Tech Times note that the approach supports test-time scaling: a model can run more loops at inference time when a problem needs more computation 13.

The catch: loops cost compute, and gains fade

The second paper explains why these results come with conditions. Every extra loop is another full pass through the shared blocks, so it costs real FLOPs even though it adds no parameters. According to the LOOM authors, the benefits of looping in large MoE models are unclear once comparisons are FLOPs-matched. Additional iterations show quickly diminishing returns and can even make results worse. As a result, earlier work has usually stopped at two loops 2.

The paper identifies two causes:

• Amplified "curse of depth." Hidden-state variance grows with each iteration as residual updates pile up. This destabilizes deep recurrence and causes representations to drift 2.
• Expert selection collapse. Routers keep choosing the same experts on every loop, so extra iterations add compute without adding much new capacity 2.

LOOM is built around one principle: each loop should contribute new computation while the recurrent state stays stable 2.

How the two papers fit together

The two papers do not contradict each other, but they emphasize different things. The scaling-law work presents looping as a predictable, tunable dimension of model design. It uses "law-derived recurrence" to pick how many passes to run 1. The LOOM paper argues that, in practice, this dimension has been capped by instability and router behavior. On its account, simply adding loops wastes compute 2.

One reasonable interpretation is that the scaling law tells designers where the best loop count lies under current conditions, while work like LOOM tries to move that optimum higher. The available evidence does not show whether the 2x size-matching result depends on staying near the two-loop range. Readers should treat that link as an open question, not an established fact.

The framing of compute also deserves attention. The headline comparison is made at matched training compute 1. Looping saves memory and parameter count, but each forward pass at deployment runs the shared blocks several times. Teams constrained by memory may find that trade-off attractive. Teams constrained by inference latency or serving cost may not. The LOOM paper's focus on FLOPs-matched comparisons makes the same point from another direction: loops are not free 2.

Why it matters

Scaling laws have mostly guided choices about how big to make a model and how much data to train it on. Adding recurrence and sparsity as jointly modeled variables gives architects more options, especially when memory is tight and parameter budgets are fixed 13.

The overall picture is promising but limited. Looped MoE models appear to offer real parameter efficiency on reasoning tasks 1. However, the gains depend on choosing the loop count carefully and on fixing the stability and routing problems that have kept models at two loops 2. Practitioners should weigh the smaller model against the extra compute each loop spends at inference time.