- [ Home ](/) / - [ Blog ](/blog/) / - Who fetches llms.txt? Agents, not AI search crawlers # Who fetches llms.txt? Agents, not AI search crawlers Many llms.txt files are published in the hope of appearing in ChatGPT or Perplexity answers. The measured requests point elsewhere, and that changes what belongs in the file. Last updated: August 12, 2026 ## The measured breakdown Ahrefs classified every user agent that requested `/llms.txt` across 137,210 domains during May 2026, sorting them into twelve categories. Only 3% of the roughly 38,000 accepted files received any request at all, about 1,100 files in this technical panel. The breakdown below describes those requests, not the full population and not what a model ultimately used. Four categories sit above every individual AI category: Category Share of requests SEO audit tools 21.7% Other and unidentified 14.9% General web crawlers 13.1% Tech profiling tools 11.6% Combined, named AI bots account for **19.5%**. Split by what they are for, the picture sharpens considerably: AI bot type Share AI agents and agentic infrastructure 10.5% AI training crawlers 5.3% AI assistants 2.5% AI retrieval bots 1.1% A further 12% comes from an industry studying the convention rather than consuming it: GEO and AEO scoring tools at 5.8%, dedicated llms.txt scanners and directories at 3.6%, research crawlers at 2.7%. One comparison makes the scale legible: **Slackbot, a link-preview bot, fetched llms.txt files more often than PerplexityBot did.** ## Agents lead among the measured AI fetches The bottom row of that second table is the one to sit with. AI retrieval bots, the crawlers that fetch pages to answer a live user query in an AI search product, such as OAI-SearchBot and PerplexityBot, made **1.1% of all measured requests**. Ahrefs report that OAI-SearchBot, PerplexityBot and Claude’s search crawler made only a couple of hundred fetches combined. Meanwhile AI agents and the infrastructure serving them made 10.5%, and Claude-Code out-fetched every retrieval bot, every assistant and every training crawler in the dataset. This matches, rather than contradicts, what Google has said. As we cover on [does Google use llms.txt](/does-google-use-llms-txt/), John Mueller’s public position is that no AI service has said it uses the file, and [he compared it to the keywords meta tag](https://www.searchenginejournal.com/google-says-llms-txt-comparable-to-keywords-meta-tag/544804/): a claim by a site owner about itself, which a system can verify more cheaply by reading the site directly. Training crawlers are the second AI category at 5.3%, fetching llms.txt nearly five times more often than retrieval bots do. If the file influences anything, the data points upstream, at corpus construction, rather than at the moment an answer is assembled. ## What that means for what you write If an intended consumer is a coding agent working on behalf of a developer, several of the usual instincts are wrong. **Marketing pages are the wrong entries.** An agent asked “how do I authenticate against this API” needs the reference page, the error codes and a working example. Your pricing page does not help it. Our [best practices](/best-practices/) make the general case; the readership data makes it concrete. **Structure beats breadth.** The [specification](https://llmstxt.org/) is a curated index, not a sitemap. A file that lists forty pages saves an agent nothing, because it still has to choose. Twelve well-described entries do the work. **Descriptions carry the signal.** Each link’s description is the only thing an agent sees before deciding whether to spend a fetch. “API reference” is close to useless. “REST endpoints, auth headers, rate limits and error codes” is a routing decision. **Consider llms-full.txt if your docs are the product.** Where the whole point is to hand an agent the content rather than a map, the companion file exists for that; see [what llms-full.txt is](/llms-full-txt/). And, per the finding we covered in [no AI agent goes looking for your llms.txt](/blog/agents-do-not-look-for-llms-txt/), none of this matters until something tells the agent the file exists. ## Every number here is a ceiling Ahrefs are explicit on this, and it should be repeated rather than buried: a request in a log proves that a client retrieved the bytes. It proves nothing about whether the model read them, used them, or discarded them. Their 19.5% AI share is therefore the most generous possible reading of AI consumption, not an estimate of it. Their population also skews technical, being drawn from their own analytics customers, which is why they present their 28% adoption figure as an upper bound. Our own [adoption panel](/llms-txt-adoption/) measures a different population entirely and is not comparable. ## Sources - [ Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read (June 2026) ](https://ahrefs.com/blog/llmstxt-study/) - [ Search Engine Journal, Google Says LLMs.txt Comparable To Keywords Meta Tag ](https://www.searchenginejournal.com/google-says-llms-txt-comparable-to-keywords-meta-tag/544804/) - [ llmstxt.org, the specification ](https://llmstxt.org/) On this page - [ The measured breakdown ](#the-breakdown) - [ Agents lead among the measured AI fetches ](#agents-lead) - [ What that means for what you write ](#write-for-agents) - [ Every number here is a ceiling ](#ceiling)