Crawl-to-Refer Ratios Expose AI's Lopsided Deal With Publishers

The exchange rate of the open web

For two decades, the bargain between search engines and publishers was simple. Crawlers could index your pages, and in return they would send readers back. A growing body of measurement suggests that bargain is fraying. Some of the pressure comes from AI companies that take far more than they return. Some comes from Google itself.

The clearest lens is Cloudflare Radar's crawl-to-refer ratio, which counts how many pages an operator crawls for every single referral it sends back to a site 3. One monthly crawler report calls it the most honest gauge of whether an AI company is a fair trading partner or simply an extractor 3. The latest figures, covering July 2026, show how wide the gap remains.

Who takes, who gives back

Anthropic still sits at the extreme end, crawling roughly 1,917 pages per referral in July 2026 2. That is a dramatic improvement from about 38,744:1 a year earlier, roughly a twentyfold gain, and a 43% month-over-month improvement 2. OpenAI also moved sharply, falling from 1,104:1 to 251:1 over the year and improving 61% in a single month 2. Perplexity went the other way, worsening from 195:1 to 289:1 2. That reversal pushed OpenAI ahead of Perplexity, and on July numbers both now sit in a middle band that had not been occupied before 2.

Traditional search engines look far more balanced by comparison. Google crawled about 4.7 pages per referral, Microsoft about 36, and Yandex about 27 2. Some smaller players deteriorated badly from near-parity a year ago. Baidu moved from 0.85:1 to 12:1, ByteDance from 0.90:1 to 9.6:1, and DuckDuckGo from 0.32:1 to 2.4:1 2.

These numbers deserve some caution. Radar revises figures after publication. One tracker notes that June's Anthropic ratio now reads about 2,947:1 rather than the roughly 2,978:1 first published, and OpenAI's about 513:1 rather than 530:1 3. Those numbers also differ from the June values of 3,386:1 and 647:1 cited in a separate analysis 2. That points to differing measurement windows or snapshot dates rather than a settled record. The metric can also break entirely. When referrals approach zero, the ratio can return unbounded or infinite values, a caveat flagged for Mistral 3. The direction of travel is consistent across trackers. The exact figures are not.

Publishers respond at the robots.txt layer

Publishers are not waiting for the ratios to improve. An analysis of robots.txt files across Cloudflare's network finds sites increasingly blocking bots that gather training data while allowing bots that fetch content to answer live queries 2. The logic is defensible. A model trained on your archive may never cite you. An answering bot at least has a chance of linking out. It amounts to a crude attempt to price access according to the likelihood of getting something back.

The bigger leak is closer to home

The AI crawler debate can obscure a larger problem. Chartbeat's 2026 Publisher Playbook found Google Search referrals to news and media sites fell 40.2% between July 2025 and July 2026 1. Across its network, Google Search's share of pageviews dropped from 9% in July 2024 to 5% two years later, a 46% relative decline 1. The report attributes the drop mainly to AI-generated overviews that answer queries directly on the results page 1.

Other research points the same way. Ahrefs measured a 58% click-through drop for top-ranking pages when an AI Overview appears 4. Seer Interactive, tracking 5.47 million queries across 53 brands, found that pages cited inside an Overview still earn less than half the clicks of a result with no Overview, even after a partial recovery in 2026 4. Google's comparatively healthy crawl ratio therefore tells only part of the story. It still sends traffic, but less of it per search.

Small share, better visitors

There is a counterweight. AI referrals remain tiny, about 0.2% of visits in Contentsquare's 99-billion-session benchmark, though that represents 632% year-over-year growth 4. Adobe found AI-referred retail visitors converting 54% better than non-AI traffic in May 2026, a reversal from converting at roughly half the rate a year earlier 4. Publishers, meanwhile, are leaning more on direct visits, loyal readers and private "dark social" sharing to sustain audiences 1.

The reading

Taken together, the data describes a content tax. AI systems consume publisher output at rates of hundreds to thousands of pages per click. Google's own interface is absorbing a growing share of the clicks that once justified open crawling. The improving ratios at Anthropic and OpenAI are real and worth acknowledging, but they start from an extremely lopsided base. High-converting AI visitors are encouraging for retailers. For news publishers, a premium on 0.2% of traffic does not replace a 40% decline in search referrals. The more durable response is the one already visible: selective blocking, and audiences that do not depend on someone else's results page.