AI Web Text Hurts LLM Pretraining Past 20 Tokens per Parameter

A token that can be worth less than nothing

Scaling laws for large language models have mostly rested on one simple assumption: within a given quality band, a token is a token. A new arXiv paper argues that this assumption no longer holds for the web as it exists today. The study, titled "How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text," was written by Jenna Russell, Ben Glickenhaus, Katherine Thai, John Wieting, Mohit Iyyer, Max Spero and Bradley Emi. It concludes that machine-written web text can help some models and harm others, and that which one happens depends mainly on how much human data a model already has.511 The authors are affiliated with the University of Maryland and with Pangram Labs, a company that sells AI-text detection.3

The team pretrained 800 language models, ranging from 19.9 million to 973 million parameters. They varied how many AI-generated tokens were added on top of a fixed human corpus, going as high as 64 AI tokens per human token, and then fit scaling laws to held-out loss on human-written and AI-written text separately.1 The result is a sign change. Models starved of human data do better at first when AI text is added, but that benefit levels off and then turns negative. Models trained on generous human budgets get worse almost as soon as AI tokens appear, while adding the same number of fresh human tokens keeps improving them.8

How much of the web is now machine-written

The paper's starting point is a measurement, and it is the measurement that makes the scaling result urgent. The authors ran Common Crawl snapshots through FineWeb's quality filter and then labeled the surviving documents. By Pangram's labels, 27.5% of filtered tokens in June 2026 were AI-generated, compared with 10.1% two years earlier, and the figure reached 31.1% by August 2026.1 The authors project that just over half of FineWeb tokens, about 50.7%, will be AI-generated by the end of 2028.1

A second finding gets less attention than it should: standard quality filters make the problem worse. In the authors' audit, FineWeb's pipeline kept AI-labeled documents 2.3 times as often as human-labeled ones, and DCLM's pipeline kept them 9.8 times as often.49 This means the filters that open pretraining pipelines depend on are selecting in favor of exactly the text the paper says large models should avoid. A plausible explanation is that fluent, tidy machine prose resembles what quality classifiers were trained to reward, though that is an interpretation and not something the paper states as a mechanism.

The authors also separate this "wild" AI text from two cases the field has already studied: synthetic data produced on purpose for training, and model-collapse setups where a model learns from its own outputs. Wild text comes from many different models, is written for human readers, and enters crawls without labels and mixed in with human writing.59

Why Chinchilla breaks, and what replaces it

This is where the work matters for architecture and scaling. The paper says existing laws fail in one of three ways. They treat AI tokens as if they were human tokens, as the Chinchilla formulation does. They never allow a token's value to fall below zero, as most repetition laws do. Or they model harm without separating it from benefit, as most mixture laws do.12 The authors set five criteria for a better law, including that a token's value must be able to vary with model size, human budget and AI ratio, and that the benefit and harm terms must be interpretable on their own.12

Their proposed law pairs a benefit that saturates with a harm term that grows logarithmically, and it reduces exactly to Chinchilla when no AI text is present.1 One detail stands out. The marginal harm of an extra AI token peaks when AI and human tokens are present in equal amounts.12 The authors also report that the harm slows down rather than speeds up as more AI text is added. That sets it apart from earlier work modeling repeated data, where harm accelerated over epochs.12

The practical threshold is easy to state. AI text lowers human-text loss only below roughly 10 human tokens per parameter. At the Chinchilla-optimal 20 tokens per parameter the benefit is gone, and above that point AI tokens raise loss.12 Most production models today are trained well past 20 tokens per parameter, and one analysis notes the paper cites an open model trained at 1,100. That puts frontier-style training firmly in the range where harm occurs.14

When the law was fit on smaller models, it predicted held-out human-text loss for models up to 3.6 times larger with 41% lower error than the best existing law across all AI ratios.1 One coverage summary describes this as beating Chinchilla by 41%.4 That wording overstates things a little: the paper compares against the strongest of eleven existing laws it tested, not only Chinchilla.1

The compute bill

The headline number from the paper is about cost. Under the new law, training on unfiltered web text at August 2026's 31.1% AI share takes 1.6 times the compute needed to train on only the human portion at 20 tokens per parameter. The gap grows as the human budget grows.1 At the forecast shares of 42% and 51% for the ends of 2027 and 2028, the multiplier rises to 2.1 and then 3.0.1

The authors make three recommendations. First, filter AI text out when the goal is to model human text. Second, repeat human text before adding AI-generated web text. Third, report validation loss on human and AI text separately.5 The data behind the second recommendation is clear. On C4, one added epoch of repeated human data gave loss 2.3% lower than adding the same amount of fresh AI text, and the gap widened to roughly 6.3 to 6.4% after eight added epochs.12 The third recommendation may matter most. AI text is easier to predict, so a mixed validation set at a 22.3% AI share hid the harm in 95.5% of the runs where harm actually occurred.1 Put simply, labs that track one blended loss curve could see improvement while their models get worse on human text.

The paper does not argue that AI text is always bad. When the target is AI text itself, such as Cosmopedia, the loss-minimizing training mix is more than 90% AI.1 The authors' framing is that human and AI text are separate domains and should be handled that way in both training and evaluation.1

Where the coverage agrees, and where it diverges

The coverage agrees on the core finding. Every write-up repeats the help-then-harm curve, the 800-model experiment and the 1.6x compute figure.234 The differences are in tone and caution. One alert service's editor's note says any lab still training on bulk web scrapes without classifier-filtered human sources is "burning compute."11 A more careful analysis points out that the result comes from one paper that has not yet been peer reviewed, and that the AI labels come from a detector sold by one of the authors' employers. That analysis also notes that the filtered share runs higher than the raw share because of how filters behave.14 Another commentator calls the 31% figure a detector-labeled estimate and not ground truth, and describes the real risk as gradual degradation, not sudden collapse.2

The paper's own limitations section supports the cautious reading more than the alarmed one. The law was fit on models up to 268 million parameters and tested up to 973 million, which is far below frontier scale.12 The authors measure next-token loss, not capability. At their model sizes, AI text raised CORE benchmark scores about as much as fresh human text did, although the authors note the models are too small for those scores to mean much.1 The study covers only pretraining on English web text labeled by Pangram.1 The authors also acknowledge a separate line of work, Mohri et al., which finds that with enough compute it is best not to filter pretraining data at all.1

The scaling direction could go either way. The authors note that larger models memorize more and can be hurt more by model-generated data, so they may absorb weaker models' habits more easily.12 If that holds, the 973-million-parameter ceiling understates the harm and does not overstate it.

What it means for reasoning and post-training

The biggest open question is what this means for reasoning. Labs depend heavily on synthetic data after pretraining, including generated math reasoning traces. Researcher Bhuwan Dhingra raised exactly this point in response to the paper's announcement.9 The paper does not address it. Its scope is wild web text in pretraining, which it explicitly separates from synthetic data built for training.5 So it says nothing directly about curated chain-of-thought data. It does warn that a pretraining base made worse by unlabeled machine text is the foundation that post-training then builds on.

This work also connects to a wider body of research showing that what goes into pretraining persists. Earlier studies found that poisoning just 0.1% of pretraining data can survive alignment. More recent work shows that third-party web comments can get through curation pipelines into training corpora. Wild AI text is not adversarial, but the lesson is the same: pretraining data behaves more like infrastructure than raw material.

My reading is that the exact compute multipliers should be treated as provisional, because they come from a single detector and sub-billion-parameter models. The structural point is likely to hold up anyway: token value can turn negative, and blended validation sets can hide that. The authors have released WildAI, an 83-billion-token labeled corpus, along with all 800 models and the code.5 That makes the claims unusually easy to test, and replication at larger scale is the next thing to watch for.