- [ Home ](/) / - [ Blog ](/blog/) / - AI bots did not probe missing llms.txt files in one study # AI bots did not probe missing llms.txt files in one study Ahrefs' May 2026 logs contain a useful negative result: AI bots did not request missing llms.txt files in that sample. If you publish one, you still need a discovery path. Last updated: August 12, 2026 ## The finding Ahrefs analysed the server logs of 137,210 domains that received traffic in May 2026, then classified every user agent that requested a `/llms.txt` path. One result stands out from the rest, and it is not the headline one. They looked separately at requests that returned **404**, meaning a client asked for an `llms.txt` that was not there. In that May 2026 sample, the share of those requests coming from identified AI bots was **zero**. Missing files drew 98% human traffic, while accepted files drew 96% bot traffic. The logs do not establish why each human requested the URL. In other words, the study observed no AI bot probing a missing root file. That is evidence about this panel and month, not proof that no system ever performs discovery. That reframes the headline finding from the same study, which is that **97% of the roughly 38,000 valid llms.txt files they found received no requests at all** during the month, from any client, bot or human. ## Why this changes the job The implicit model most people have is the `robots.txt` model: put the file at the root, and crawlers that apply the Robots Exclusion Protocol know its standard location. That protocol is formalised in [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), but it is still a crawler convention rather than access control. It is not a safe assumption for `llms.txt`. As we set out on [what llms.txt actually does](/does-llms-txt-work/), the convention was [proposed by Jeremy Howard at Answer.AI in September 2024](https://www.answer.ai/posts/2024-09-03-llmstxt.html). The [v2 specification](https://llmstxt.org/) defines discovery relations and path-scoped files, but it does not oblige every client to fetch them. Ahrefs observed no speculative AI requests for missing root files in its sample. So publishing the file is roughly half the work, and it is the half that does not by itself produce a single fetch. The other half is making something tell an agent the file exists. ## How to route agents to the file Start with the discovery mechanisms defined by v2, then add human-visible routes where they help: **Declare the relation.** On an HTML page, add ` ` for the applicable file and, where a Markdown counterpart exists, ` `. The same relations can be sent through HTTP `Link` headers. A file on a subpath can describe that section, with the most specific path applying. **Link it from your HTML.** A plain ` ` in a footer or docs navigation places the URL in the ordinary link graph and gives people a visible fallback. **Reference it in your documentation.** If your product has a docs site, an “for AI assistants” or “for agents” page that names the URL explicitly gives a coding agent something to act on when a developer asks it about your API. Ahrefs found AI agents and agentic infrastructure to be the largest AI category fetching these files at 10.5% of requests, well ahead of AI retrieval bots at 1.1%. Claude-Code out-fetched every retrieval bot, every assistant and every training crawler in their data. Those are tools a developer points at a URL. **Put it where instructions live.** Anywhere a user or an integration hands an agent context about your site, such as a README, an onboarding page or an MCP server description, naming the `llms.txt` URL can turn publication into a discoverable route. None of these require touching the file itself. If you have not written one yet, [how to create llms.txt](/how-to-create/) covers the format and the per-stack deployment, and the [validator](/validator/) checks it against the spec. ## What not to conclude Two honest caveats, both stated by Ahrefs themselves. Their population skews technical and SEO-aware, because it is drawn from their own analytics customers. They flag the 28% adoption figure as an **upper bound**, not a web-wide rate. Our own [adoption panel](/llms-txt-adoption/), which tracks a fixed list of notable hosts, is a different measurement of a different population and should not be compared directly. And “fetched” is not “read”. A request in a server log proves a client retrieved the bytes. It proves nothing about whether the model then used them. Every number in that study is therefore a ceiling on real consumption, not an estimate of it. The practical takeaway survives both caveats. If you publish an `llms.txt` without a declared relation, a link or another route to it, Ahrefs’ sample gives no reason to assume an agent will discover it on its own. Adding a standards-based discovery signal is a small, testable fix. ## Sources - [ Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read (June 2026) ](https://ahrefs.com/blog/llmstxt-study/) - [ Chrome for Developers, Lighthouse agentic browsing audits: llms.txt ](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt) - [ llmstxt.org, the specification ](https://llmstxt.org/) On this page - [ The finding ](#the-finding) - [ Why this changes the job ](#why-it-matters) - [ How to route agents to the file ](#how-to-route) - [ What not to conclude ](#what-not-to-conclude)