Encoder-Free MLLM Scaling Laws Predict Crossover Near 10^22 FLOPs

What happened

Most multimodal large language models today connect a pretrained vision encoder to a language decoder. A new scaling-laws study suggests that this encoder may stop being worth its cost once a training run is large enough. The preprint, titled How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining, appeared on arXiv on 28 September 2026. Its authors are Lin Chen, Bolin Ni, Qi Yang, Lan Jiang, Kun Ding, Xiaoran Fan, Hower Yang, Ying Wang and Shiming Xiang11. Summaries credit the work to researchers at CASIA, UCAS and Tencent5. One outlet goes further and calls it a Tencent Hunyuan study9. A separate summary says the work was done during internships at Tencent8.

The main claim fits in one line. Encoder-free models fall behind on the multimodal objective at small scales, but the authors' fitted curves predict they catch up at roughly 10^22 FLOPs, a budget the authors call well within practical pretraining ranges11. On the text objective, the two designs follow loss–compute frontiers that almost overlap11.

How the comparison was built

The study is useful mainly because its comparison is tightly controlled. Encoder-free and encoder-based models were trained on the same ladder of 11 sparse mixture-of-experts decoders, ranging from 1.1B to 44B total parameters and 71M to 2.4B active non-embedding parameters. Both families used the same data mixture, optimization setup and visual-token granularity13. In the encoder-free version, projected image patches go straight into the decoder, which has to learn visual representations from raw pixels.

The authors fit separate scaling laws for text loss and multimodal loss using IsoFLOP profiles. For each compute budget, they varied model size per token, fit validation loss as a quadratic in log model size, and took the minimum as the compute-optimal point5. They used six log-spaced budgets per objective: 2×10^19 to 2×10^20 FLOPs for text and 1×10^20 to 1×10^21 FLOPs for multimodal1.

The numbers behind the crossover

The result comes from the slopes of the curves. On the multimodal objective, the fitted loss–compute exponent is about 0.378 for encoder-free models and about 0.300 for encoder-based ones1. Encoder-free models start with higher loss, but their loss falls faster as compute grows, so the gap narrows13. On text, the exponents almost match: 0.0973 versus 0.09791.

Under compute-optimal allocation, the crossover point estimate is 6.1×10^21 FLOPs, with a conditional bootstrap 80% interval of 4.2×10^21 to 1.0×10^221. Many deployed models are deliberately overtrained, so the authors also modelled a 5× overtraining regime. There the estimate moves to 1.2×10^22 FLOPs, with an interval of 8.4×10^21 to 2.0×10^227. Every bootstrap replicate produced a finite crossover between 10^21 and 10^23 FLOPs. Leave-one-budget-out refits put the compute-optimal crossover between 4.3×10^21 and 9.8×10^21 FLOPs1.

The authors compare this with frontier training budgets. They estimate that a recent flagship such as Kimi K2.5 used about 10^25 FLOPs of pretraining compute, so the projected crossover sits about three orders of magnitude below current frontier runs1.

Removing the encoder changes how compute should be spent

The first finding gets less attention than the crossover, but it may matter more in practice. Without a visual encoder, the compute-optimal model-allocation exponent for the multimodal objective rises from 0.464 to 0.570. That pushes the optimum toward larger decoders, while text allocation stays essentially unchanged. The authors read this as the decoder needing extra capacity to handle visual representation learning on top of language modelling. AI Weekly's summary puts it plainly: an encoder-free run should spend its budget on parameters rather than tokens2.

This has a direct effect on how training runs get planned. A team that drops the encoder but keeps the token-heavy recipe tuned for encoder-based systems would, by this analysis, be training at the wrong point on the curve.

How the decoder replaces the encoder

The paper also looks inside the decoder to see how it makes up for the missing encoder. Three adaptations get stronger with scale[11]:

• Bidirectional attention among visual tokens helps more as training compute grows.
• Visual processing moves toward earlier layers.
• Expert routing for visual tokens becomes more concentrated.

The conclusion adds that encoder-free scaling delays the decoder's reliance on visual input7. Put simply, the lower layers of a sufficiently large decoder start doing the encoder's job. One Chinese-language analysis concludes that encoder-free models may need decoder architectures built for native visual representation learning, not designs carried over unchanged from language models8. That is an inference from the findings, but a reasonable one. If bidirectional attention over image tokens keeps getting more useful, a strictly causal decoder looks like the wrong default.

The crossover depends on the task

The single 10^22 figure hides a lot of variation. Within the measured range, encoder-free models were less compute-efficient on every topic. Extrapolated, they catch up first on STEM, which was already close to parity at the largest fitted budget, then on Charts. GUI, OCR and Captioning catch up much later13. The authors explain this by noting that STEM and chart inputs often reduce to symbolic content, after which prediction depends mostly on language and reasoning. Captioning and similar tasks rely more on pretrained visual representations.

This is the part of the paper most likely to affect real deployments. A model aimed at screen understanding or document OCR could need far more than 10^22 FLOPs before dropping the encoder pays off. A math-and-diagrams assistant could get there much sooner.

Where coverage agrees and where it is cautious

The coverage agrees on the main facts: the near-overlapping text frontiers, the multimodal gap, the ~10^22 projection and the three internal adaptations25910. The differences are in emphasis. AI Weekly stresses that the 10^22 number is a projection, not a completed training run, and notes that the authors present encoder-free designs as "a promising direction" rather than a settled winner2. Its editor's note focuses on the engineering benefit: removing a whole pretrained component and spending the savings on parameters2. CCTest's headline asks whether visual encoders will disappear, while its text uses the conditional "could" for convergence9.

The more critical commentary comes from community paper notes. One review lists three weaknesses. The measured range covers about 10^19 to 10^21 FLOPs, so the crossover rests entirely on power-law extrapolation. Dataset details are thin. And the encoder-based baseline keeps the ViT at a fixed size, which may understate what the encoder approach can do at scale8. The authors acknowledge the extrapolation assumption themselves: the forecast assumes the fitted laws continue beyond the measured range, with a fixed encoder size and an irreducible loss set only by the data7. The same notes say the paper gives no code release or project page, which limits reproducibility. Another research note predicts that any catch-up will show up in pretraining loss before it shows up in fine-grained perception. Given the topic-level results, that prediction looks sound.

The authors do offer some evidence that the curves can be trusted outside the fitted range. In held-out tests, the multimodal fit forecast the 10^21-FLOP optimum, 2.5× past its largest fitted budget, with signed errors of −0.01% for encoder-free models and +2.19% for encoder-based ones7. That is encouraging. It is still a 2.5× extrapolation, while the headline claim requires roughly 10×.

The accounting detail that favors encoder-free designs

One appendix result points the other way from the main caveats and receives little coverage. The main analysis counts only decoder compute. When the cost of the vision front end is included, the encoder-based model's multimodal exponent rises to 0.362 and its optimal data share grows, but the encoder-free exponent stays at 0.37813. The authors conclude that under full accounting the crossover comes no later than 6.1×10^21 FLOPs, and that at the decoder-only crossover encoder-free models already need less total compute1. In that respect, the headline number is conservative.

Assessment

This paper is best read as a careful forecast, not a verdict. The fixed-encoder baseline and the extrapolation gap are real limitations, and the GUI and OCR results show that perception-heavy tasks will not shift soon. Still, the main pattern holds up under bootstrap and leave-one-out checks. The encoder's advantage is a head start that shrinks with scale, and the compute where it disappears is small compared with what frontier labs already spend. For anyone designing multimodal architectures, the practical lessons are concrete: an encoder-free run needs larger decoders, attention patterns that suit visual tokens, and evaluations that measure perception, not just aggregate loss. The open question is whether someone will train at 10^22 FLOPs and test the projection directly.