llms.txt Server Logs Show Coding Agents, Not Crawlers, Read It

The file everyone writes, almost no crawler reads

Few files in AI search have drawn more attention than llms.txt, the markdown map that sites publish to guide large language models through their content. Server logs now show how rarely the major AI crawlers actually request it.

EZY Research watched 83 websites with live llms.txt files over 12 weeks, from late April to mid-July 2026, using server-side records of verified crawler traffic. 1 OpenAI's crawler fetched llms.txt seven times across all 83 sites. In the same period it fetched robots.txt 3,990 times. 1 Anthropic did slightly better with nine llms.txt fetches. Perplexity requested the file zero times while pulling robots.txt 775 times. 1

The gap is the main finding. These crawlers were active on the sites and checking access rules constantly. They were not looking at the file written for them.

Independent studies point the same way

The EZY numbers match earlier research. One server-log study across roughly 900 domains logged 1,227 requests for llms.txt-family files. 3 The largest single requester was a commercial data aggregator, with 794 requests. The second-largest source was ordinary browser traffic, which the study attributes to people checking whether the file existed. 3 None of the requests came from a verified frontier-lab crawler. 3 A separate analysis of about 300,000 domains found no measurable link between having the file and how often a site was cited by AI. 3

Ahrefs looked at a larger sample of 137,000 domains and found that 97% of llms.txt files received no requests at all in May 2026. 4 The studies differ in method and size. All of them find that search-oriented AI crawlers largely ignore the file.

The sources do diverge on Perplexity. EZY recorded no Perplexity fetches. 1 Another guide says Perplexity has been observed surfacing llms.txt content outside its standard retrieval pipeline. 2 Both can be true. Content can reach a model by routes other than a crawler fetching the file directly. Still, the anecdote is thin compared with the log data.

Adoption keeps climbing anyway

Publishing has not slowed. Originality.ai counted 36,120 llms.txt files across the roughly 3 million sites it scans, an 8.8-fold increase in a year. 1 Large publishers such as TIME are building agent-readable versions of their content. 1 Adoption is rising quickly while measurable consumption by search crawlers stays near zero.

Google contradicts itself

Part of the confusion comes from Google. Its Search Central guidance from May 2026 says llms.txt, content chunking and special markup are not needed for visibility in AI features. 4 Google has said in two separate contexts that the file does nothing for Search. 3 Around the same time, Chrome's Lighthouse 13.3 added an "Agentic Browsing" audit category that checks whether a site provides the file. 24

Google's John Mueller has described llms.txt as a "temporary crutch" for coding agents rather than a search signal. 4 That description likely explains both positions. Search ignores the file. Agentic tooling, which navigates documentation and performs tasks, has some use for it.

Where llms.txt does matter

The agent use case has real backing. Anthropic recommends llms.txt in its Writing for Agents guidance. 2 OpenAI maintains llms.txt files for its Agents SDK and its Agentic Commerce Protocol. 2 For developer documentation, API references, and sites where Claude or OpenAI agents send meaningful traffic, the calculation is different from that of a typical marketing site. 2

This may also explain why the log studies come up mostly empty. They mainly track identifiable crawlers and their user agents. An AI coding assistant fetching documentation during a developer's session would not necessarily appear as OpenAI's or Anthropic's verified crawler. It might show up as generic traffic, or not be counted at all. That is an inference rather than a measured result. It fits Mueller's framing and the vendors' own documentation-focused uses of the file.

The takeaway

The evidence supports a narrow conclusion. llms.txt is not an AI-search ranking lever, and anyone selling it as one has the server logs against them. 134 It is a reasonable, low-cost addition for sites whose audience includes software agents working through technical documentation. 2

For most publishers, the file is cheap to add and unlikely to cause harm. Expecting it to change AI search visibility is not supported by the data. The crawlers that matter for search are still reading robots.txt thousands of times and llms.txt only a handful.