Cloudflare AI Crawler Block Targets Mixed-Use Bots by Sept 15

What Cloudflare announced

Cloudflare has set a deadline for the AI industry. Starting September 15, 2026, its default settings will block "mixed-use" crawlers from pages that carry advertising. These are bots that combine traditional search indexing with AI training or agent activity. Site owners can still change the setting themselves. 13 The company presents the change as a choice for AI firms: separate search crawlers from training and agent crawlers, or lose default access to a large share of ad-supported publishing. 1

The new defaults apply to new Cloudflare customers, to new sites that existing customers set up, and to all existing free-tier customers. 1 Cloudflare tied the announcement to a second "Content Independence Day." That marks one year since it introduced a one-click "Block AI Bots" option and a Pay-Per-Crawl marketplace. 23 It is also adding finer controls, so every customer can now manage Search, Agent and Training bots separately and apply extra protection to ad-monetized pages. 2 Where several rules overlap, the most restrictive one wins. 2

From pay-per-crawl to pay-per-use

The second shift concerns how publishers get paid. Pay-per-crawl, which lets site owners charge AI crawlers a fee for access, is still in closed beta with publishers across a range of content types. 4 Cloudflare now says its default position is moving to "pay-per-use," where compensation would depend on how content is actually used. The company admits this model is still at a very early stage. 43

Stephanie Cohen, Cloudflare's chief strategy officer, describes the logic as creating "reliable scarcity." Her argument is that once scraping is blocked by default, AI companies have a reason to sign licensing deals, and Cloudflare says that is already producing more licensing revenue for publishers. 4 Cloudflare does not publish independent figures to back that up, so it is best treated as the company's own account of how well its strategy is working.

The sources describe the history slightly differently. Press Gazette says Cloudflare already began blocking AI crawlers by default last year unless customers opted in. 4 TechCrunch frames the September change as a new default aimed at mixed-use crawlers on ad-carrying pages. 1 Both can be true. The earlier default targeted AI crawlers in general, while the new rule closes a gap: bots that bundle search with training, which publishers could not easily turn away without also losing search visibility.

Why the crawl ratio matters

The economic case rests on a lopsided exchange. Cloudflare's own framing is that the 30-year bargain of "we crawl you, you get referrals" has broken down, with AI "taking everything and sending back nothing." 2 Its Radar data puts numbers on this. As of mid-2026, Anthropic's ClaudeBot crawled roughly 1,900 to 5,100 pages for every referral it sent. OpenAI's ratio was around 250 to 870 to one. Google's has held steady at about five to one. 5 Those AI ratios are reportedly improving, but they remain orders of magnitude worse than search. 5

The referral picture is disputed. Chartbeat's 2026 Publisher Playbook finds AI chatbot referrals at a negligible 0.01% of publisher pageviews, and slipping through 2026. Contentsquare, by contrast, reports 632% year-over-year growth. 5 These figures can coexist, since rapid growth from a tiny base can still leave AI referrals close to zero as a share of traffic. Even so, the gap shows how much the story depends on which dataset you trust. For publishers deciding whether to invest in optimizing for AI answer engines, the more pressing problem may be that so few visits arrive at all, not how well those visits convert. 5

Reading the move

The significant change is that the crawl ratio stops being a grievance and becomes a bargaining position. Robots.txt has always been a polite request that crawlers could ignore. 3 Cloudflare sits in front of about 20% of websites, so it can enforce the request at the network level. 4 Setting the block as the default is what shifts the leverage. Free-tier customers and new sites, many of which would never adjust settings themselves, will be shielded automatically. 1

The pressure falls hardest on companies that run a single crawler for several purposes. Separating search bots from training bots would let AI firms keep indexing access while negotiating separately for training and agent use. That is the kind of unbundling publishers have wanted.

Several questions remain open. The headline payment mechanisms are still in beta or early design. 4 Publishers can override the defaults. And whether AI companies actually split their crawlers by mid-September is up to them. Still, Cloudflare has moved from documenting the imbalance to setting terms, and that changes the negotiation even before any payment system is fully working.