llms.txt: Google Says Search Ignores It, Yet Uses It Itself

Google has spent the past several weeks telling site owners that llms.txt files do nothing for its search results. Its own engineering teams keep publishing them anyway. The mixed signal has become a useful test case for the emerging practice of AI search optimization, or AEO. Fresh traffic data and wildly inconsistent market forecasts suggest the discipline is less settled than its marketing implies.

What Google actually said

In mid-May 2026, Google Search Central published a documentation page on optimizing websites for generative AI features in Search 1. A section framed as "mythbusting" told site owners that machine-readable files such as llms.txt are not needed to appear in generative AI search 34.

The accounts differ slightly on timing. One describes the guide arriving May 15 1. Another places Google's statements in late May 3. A third points to an update dated June 15 that hardened the language into a flat statement: Google Search does not use new machine-readable files, AI text files, special markup or Markdown, including for its generative AI capabilities 2. Google said that update was added to address questions from the community 2. The most likely reading is an initial guide followed by a clarifying revision as confusion persisted.

The contradiction SEOs noticed

The friction came from Google's own properties. Its developer documentation sites have carried llms.txt files, and several Google properties publish Markdown versions of their pages 1. On May 19, Amsive's Lily Ray asked John Mueller on Bluesky why Google uses these files itself while telling everyone else they are unnecessary 1.

Mueller said llms.txt is "not done for search." He called it a "temporary crutch, perhaps to save some tokens" for AI coding tools that parse developer documentation, and said it is not something non-developer sites need to worry about 3. He added that site owners who check their logs will find very little AI agent traffic 3. He had made a similar point on Reddit earlier, comparing llms.txt to the old keywords meta tag. That tag was a site's own claim about its content, which Google learned not to trust 2.

The internal picture is still messy. The Chrome team reportedly audits for llms.txt even as Search says it can be skipped 3. That is less a contradiction about ranking than evidence that Google's product groups treat the file differently depending on the use case.

What the data shows

Ahrefs' analysis of 137,000 domains found that 28% publish an llms.txt file, but 97% of those files received zero requests 3. Another account cites the same 97% figure across "hundreds of thousands of domains." It adds that statistical modeling found no effect on citations 2. The scope differs, but the conclusion is the same: most of these files are never read.

That matches the support picture. No major LLM provider has committed to using llms.txt 4. OpenAI's GPTBot honors robots.txt but does not officially use the format 4. Anthropic publishes its own llms.txt without saying its crawlers rely on the standard 4. Google manages AI crawling through robots.txt via the Google-Extended user agent 4. Google did include llms.txt in its Agent2Agent protocol in April 2025. One analysis describes that as one proposed protocol nested inside another, with no commitment to crawl it 4.

Adoption among developer-focused companies such as Stripe, Vercel and Anthropic continues anyway 2. That supports Mueller's framing: the file's real audience is coding assistants reading documentation, not search engines ranking pages.

The AEO market can't agree on its own size

The llms.txt episode sits within a broader uncertainty about AEO as a category. Vendor estimates for the 2026 market vary widely [5]:

• Dimension Market Research: $160.9 million
• Coherent: $1.25 billion
• QY Research: $9.04 billion by 2031

Google has also publicly rejected several practices agencies sell as AEO essentials, including llms.txt, structured data requirements and content "chunking" 5. Its position is that optimizing for AI search is still, fundamentally, SEO 5. That makes it hard for any vendor to claim a definitive framework for the field.

The takeaway

Google's teams are less split than the headlines suggest. Search ignores llms.txt. Developer documentation teams use it as a token-saving convenience for AI coding tools. The Chrome team's auditing is a separate signal that does not change ranking behavior.

For most publishers, the practical answer is that llms.txt is cheap to add and unlikely to matter. Sites with technical documentation aimed at developers have a reasonable case for it. Everyone else should treat it as optional housekeeping, not a visibility strategy.

The bigger lesson is about AEO itself. A market whose forecasts span two orders of magnitude, and whose core tactics the dominant search engine says it ignores, is still defining itself. Until AI providers commit to specific standards, the safest strategy looks much like traditional SEO: crawlable, well-organized content that answers real questions.