Blocking AI Crawlers: Why News Publishers Lost More Traffic

The defensive move that didn't work

When large language models began changing how people find and read news, many publishers reached for the most obvious tool they had: blocking AI crawlers through their robots.txt files. The logic seemed sound. If AI companies were absorbing journalism to answer questions that once sent readers to news sites, shutting the door should protect both content and traffic.

New research suggests that logic was wrong. A study from Rutgers Business School and The Wharton School, published December 31, 2025, found that publishers who blocked AI bots fared worse than those who did not 1. The researchers place the shift in news consumption patterns as beginning in August 2024, when LLMs started to meaningfully reshape how audiences reached news 1.

How big was the damage?

The 23% figure in many headlines needs context. According to Digital Content Next's reading of the Wharton/Rutgers work, blocking reduced weekly traffic by about 7% within six weeks on average 2. The steeper 23% decline applied to the largest publishers 2. Coverage of the study varies in how it presents that number. Some framing suggests a broad 23% loss across news publishers 1, while the more detailed account treats it as the upper bound for major outlets 2.

That distinction matters. A 7% weekly hit is serious for a sector with thin margins. A 23% drop at the biggest brands, which often lead industry strategy, is the kind of result that should prompt executives to rethink their policies.

Why blocking backfires

Digital Content Next points to two mechanisms 2. First, blocking removes a publisher's brand from AI-generated answers. When a chatbot can't cite or surface a outlet, readers lose one of the remaining ways they might discover it. Second, blocking does not actually stop scraping. An estimated 30–40% of bots ignore robots.txt directives entirely 2.

Taken together, the result is lopsided. Compliant AI systems drop the publisher's name, while non-compliant scrapers keep taking the content anyway. The publisher pays the visibility cost and gets little of the protection it wanted.

The referral gap that makes this worse

A reasonable objection is that AI referrals are minor, so losing them shouldn't matter much. Digital Content Next's analysis of Chartbeat data shows how small those referrals are 2. ChatGPT accounted for just 0.02% of publisher referrals and Perplexity for 0.002%. Google Search, by comparison, delivered 7.3%, roughly 500 times more traffic than ChatGPT 2.

That comparison makes the traffic loss from blocking harder to explain. If AI tools send almost no direct clicks, a 7% to 23% drop can't come only from lost AI referrals. Our reading is that the effect is indirect. AI answers may work more like brand awareness than a referral pipeline, shaping which outlets people later search for, remember, or visit directly. Removing a publication from that layer of discovery may reduce demand in ways that show up across other channels. The sources don't fully map this mechanism, so it should be treated as interpretation, but it fits the brand-exposure explanation the research offers 2.

A trap with no easy exit

Digital Content Next describes the situation as a trap 2. Publishers can't replace declining search-era traffic with AI referrals, because those referrals barely register. Yet blocking AI systems, the natural defensive response, leaves them worse off 2.

This puts news organizations in an uncomfortable position. Leaving the door open means their work feeds products that send back very little traffic. Closing it, at least through robots.txt, appears to cost them visibility without stopping the extraction.

What this means going forward

In our view, the findings argue against treating robots.txt as a strategy. It is a voluntary signal, and a large share of crawlers ignore it 2. Publishers worried about AI use of their content will likely need tools that work in practice: licensing agreements, technical enforcement against non-compliant bots, or collective bargaining through industry groups.

The research also suggests a change in how publishers measure the value of AI platforms. Counting referral clicks alone understates what is at stake, since presence in AI answers appears tied to broader traffic outcomes 12. The question for publishers may be less about whether to let AI in and more about the terms of that access.

The broader lesson is that instinctive defenses can be costly in a shifting distribution landscape. The publishers who blocked AI crawlers were trying to protect their audiences. By the study's measure, many ended up with smaller ones 12.