How agents discover llms.txt v2 resources
The v2 proposal defines two link relations for discovery. Our baseline measures how often they appeared across pages linked from H1-first root responses.
Last updated:
alternate points from an HTML page to its clean Markdown representation. describedby points from that page to the most specific applicable llms.txt. Together they make v2 resources discoverable without guessing paths, but only clients that implement the proposal will act on them.
Key takeaways
- 31 of 286 reachable sampled pages advertised Markdown.
- Four advertised
rel="describedby"in HTML or an HTTP Link header.- Discovery does not imply use, citation, or a Google ranking effect.
What are the two discovery signals?
An HTML page can add <link rel="alternate" type="text/markdown" href="..."> for a representation containing the same essential information in Markdown. It can add <link rel="describedby" href=".../llms.txt"> for the map covering that page.
The proposal also permits equivalent HTTP Link headers. Headers are useful for non-HTML resources, while HTML is easier to inspect and generate from a shared layout. Use absolute or correctly resolved URLs, keep both resources reachable, and do not advertise a stale conversion.
What did 294 pages expose?
Our 12 August benchmark sampled up to three linked pages from each of 113 root responses meeting the H1-first inclusion rule. It requested 294 pages and reached 286. Thirty-one reachable pages advertised a Markdown alternate, and four advertised rel="describedby". The detector inspected both HTML and HTTP Link headers.
The result establishes a dated baseline for this specific sample. It does not show that the remaining pages lack Markdown at an unadvertised URL, and it does not cover every page on each host. The aggregate and methodology are published in v2-adoption-stats.json.
How to implement the relations safely
Generate Markdown from the same source as HTML. For a trailing-slash route, v2 suggests an index.md or index.html.md representation. Keep the HTML page canonical for Search and treat Markdown as an alternate representation rather than a competing editorial page.
Resolve describedby by scope. An English product page may use /llms.txt; a documentation page may use /docs/llms.txt; a localized section may use its own path-specific file. Test the final built HTML, not only template variables.
Add CI checks for missing alternates, non-200 resources, wrong MIME types, and relations pointing outside their intended scope. The validator checks file syntax, while the v2 migration checklist covers rollout order.
What these relations cannot do
They cannot force a crawler to fetch a resource, override robots controls, make private information public safely, or improve Search rankings. Google explicitly says Search does not use special AI text files for its generative features.
The practical outcome is narrower: a compatible agent can discover a cleaner representation and a curated map using standard Web links. Measure requests in logs before claiming more.
For context, start with what llms.txt is and compare the relations with real production examples.
Is rel="describedby" required?
No. It is part of the v2 proposal’s discovery mechanism, not the minimum file syntax.
Can I use HTTP headers instead of HTML?
Yes. The proposal describes both. Choose the layer you can generate and test reliably.
Should Markdown self-canonicalize?
For Search, keep the HTML page as the canonical public document. The Markdown resource should clearly identify its source and remain content-equivalent.