- [ Home ](/) / - AI crawlers # Which clients fetch llms.txt? Request logs can identify fetches and client labels. They cannot prove that a model read the file, used it, or cited it in an answer. Last updated: August 12, 2026 ## The short answer There is no verified complete list of products that automatically discover, parse and use `llms.txt`. The strongest public evidence is request-log data: it shows that named clients fetched files in a defined sample, not what a model did with the bytes afterward. V2 allows a map at the origin root or a more specific path and defines `rel="describedby"` for discovery. A page may also advertise a Markdown representation with `rel="alternate" type="text/markdown"`. Clients must implement those relations for them to have an effect. ## What the request logs measured Ahrefs analysed requests to `/llms.txt` across 137,210 domains during May 2026. Roughly 38,000 domains served a response that Ahrefs accepted, but only 3% of those files received any request in the month. The panel is technical and SEO-oriented, so its rates are not Web-wide estimates. Named AI bots accounted for 19.5% of the measured requests. SEO audit tools, general crawlers, profiling tools and unidentified clients together accounted for much more. Slackbot fetched the file more often than PerplexityBot in the sample. ## Which client categories fetched the file Named AI category Share of all measured requests Agents and agentic infrastructure 10.5% Training crawlers 5.3% AI assistants 2.5% Retrieval bots for live AI search 1.1% Agentic clients formed the largest named AI category, and Claude-Code generated more requests than any retrieval bot, assistant or training crawler in the dataset. That makes developer-initiated workflows a plausible audience, but it does not establish automatic discovery or product-wide support. ## Search, training and user visits are different OpenAI's crawler documentation separates `OAI-SearchBot` for ChatGPT Search, `GPTBot` for content that may be used to train foundation models, and `ChatGPT-User` for certain actions initiated by a user. OpenAI states that `ChatGPT-User` is not used for automatic web crawling and is not used to decide whether content appears in Search. This distinction is useful when classifying logs. It does not establish that any of those user agents automatically discover, parse or give special treatment to `llms.txt`: the public documentation does not make that claim. ## What a request cannot prove - A request proves that a client retrieved a response, not that a model read or retained it. - A user-agent string is a declared label and can be spoofed unless its origin is verified. - A fetch does not show that the file influenced an answer, citation, ranking or training corpus. - A manual request to an explicit URL is different from automatic discovery. - One technical panel and one month cannot establish behavior across the entire Web. ⚠ Use fetch, read and use as different words Server logs can support the verb fetch . Use read , use or cite only when a separate observation proves that stronger claim. ## How to measure your own file - Log requests to every root and path-level `llms.txt` URL. - Keep status, timestamp, requested URL, final URL and user-agent string. - Verify known bots through the provider's documented method when attribution matters. - Separate human, audit, general crawler, agentic, training, assistant and retrieval traffic. - Report requests separately from referrals, citations and downstream conversions. The [monitoring method](/blog/how-to-monitor-llms-txt/) explains denominators and response states. For the measured category breakdown, read [who fetches llms.txt](/blog/who-actually-reads-llms-txt/). ## FAQ ### Does ChatGPT read llms.txt? No OpenAI documentation establishes automatic `llms.txt` discovery or special treatment. OpenAI documents `OAI-SearchBot` for Search and `ChatGPT-User` for certain user-initiated visits. Those roles should not be conflated, and a fetch to an explicit URL is not proof of product-wide support. ### Does Claude read llms.txt? No public provider documentation establishes automatic `llms.txt` discovery across Claude. A log entry can show that a client fetched a response; it cannot show that the file was parsed, retained or used in an answer. ### What is the llms.txt crawler user agent? There is no single user agent for `llms.txt` crawlers. Each AI system uses its own crawler identity (PerplexityBot, GPTBot, ClaudeBot, etc.). `llms.txt` is a passive file, crawlers must explicitly fetch it. ## Related pages - [Who uses llms.txt](/llms-txt-adoption/), real-world adoption evidence. - [llms.txt format reference](/llms-txt-format/), spec details. - [How to create llms.txt](/how-to-create/), step-by-step for any stack. - [Validator](/validator/) · [Generator](/generator/). ## Sources - [ llmstxt.org, official spec ](https://llmstxt.org/) - [ Ahrefs, 137K-site llms.txt request-log study (June 2026) ](https://ahrefs.com/blog/llmstxt-study/) - [ Answer.AI, original proposal ](https://www.answer.ai/posts/2024-09-03-llmstxt.html) - [ Chrome for Developers, Lighthouse llms.txt audit ](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt) - [ OpenAI, overview of crawlers and user agents ](https://developers.openai.com/api/docs/bots) On this page - [ The short answer ](#overview) - [ What the request logs measured ](#evidence) - [ Which client categories fetched the file ](#categories) - [ Search, training and user visits are different ](#intent) - [ What a request cannot prove ](#limits) - [ How to measure your own file ](#measure) - [ FAQ ](#faq)