AI bots did not probe missing llms.txt files in one study

The finding

Ahrefs analysed the server logs of 137,210 domains that received traffic in May 2026, then classified every user agent that requested a /llms.txt path. One result stands out from the rest, and it is not the headline one.

They looked separately at requests that returned 404, meaning a client asked for an llms.txt that was not there. In that May 2026 sample, the share of those requests coming from identified AI bots was zero. Missing files drew 98% human traffic, while accepted files drew 96% bot traffic. The logs do not establish why each human requested the URL.

In other words, the study observed no AI bot probing a missing root file. That is evidence about this panel and month, not proof that no system ever performs discovery.

That reframes the headline finding from the same study, which is that 97% of the roughly 38,000 valid llms.txt files they found received no requests at all during the month, from any client, bot or human.

Why this changes the job

The implicit model most people have is the robots.txt model: put the file at the root, and crawlers that apply the Robots Exclusion Protocol know its standard location. That protocol is formalised in RFC 9309, but it is still a crawler convention rather than access control.

It is not a safe assumption for llms.txt. As we set out on what llms.txt actually does, the convention was proposed by Jeremy Howard at Answer.AI in September 2024. The v2 specification defines discovery relations and path-scoped files, but it does not oblige every client to fetch them. Ahrefs observed no speculative AI requests for missing root files in its sample.

So publishing the file is roughly half the work, and it is the half that does not by itself produce a single fetch. The other half is making something tell an agent the file exists.

How to route agents to the file

Start with the discovery mechanisms defined by v2, then add human-visible routes where they help:

Declare the relation. On an HTML page, add <link rel="describedby" href="/llms.txt"> for the applicable file and, where a Markdown counterpart exists, <link rel="alternate" type="text/markdown" href="...">. The same relations can be sent through HTTP Link headers. A file on a subpath can describe that section, with the most specific path applying.

Link it from your HTML. A plain <a href="/llms.txt"> in a footer or docs navigation places the URL in the ordinary link graph and gives people a visible fallback.

Reference it in your documentation. If your product has a docs site, an “for AI assistants” or “for agents” page that names the URL explicitly gives a coding agent something to act on when a developer asks it about your API. Ahrefs found AI agents and agentic infrastructure to be the largest AI category fetching these files at 10.5% of requests, well ahead of AI retrieval bots at 1.1%. Claude-Code out-fetched every retrieval bot, every assistant and every training crawler in their data. Those are tools a developer points at a URL.

Put it where instructions live. Anywhere a user or an integration hands an agent context about your site, such as a README, an onboarding page or an MCP server description, naming the llms.txt URL can turn publication into a discoverable route.

None of these require touching the file itself. If you have not written one yet, how to create llms.txt covers the format and the per-stack deployment, and the validator checks it against the spec.

What not to conclude

Two honest caveats, both stated by Ahrefs themselves.

Their population skews technical and SEO-aware, because it is drawn from their own analytics customers. They flag the 28% adoption figure as an upper bound, not a web-wide rate. Our own adoption panel, which tracks a fixed list of notable hosts, is a different measurement of a different population and should not be compared directly.

And “fetched” is not “read”. A request in a server log proves a client retrieved the bytes. It proves nothing about whether the model then used them. Every number in that study is therefore a ceiling on real consumption, not an estimate of it.

The practical takeaway survives both caveats. If you publish an llms.txt without a declared relation, a link or another route to it, Ahrefs’ sample gives no reason to assume an agent will discover it on its own. Adding a standards-based discovery signal is a small, testable fix.