llms.txt subpaths: a 113-site benchmark
Path-level files are central to llms.txt v2. A reproducible crawl shows they already exist, but the result needs careful interpretation.
Last updated:
A path-level llms.txt can give an agent a smaller, more relevant map than a single origin-wide file. Our first benchmark found at least one distinct file on a tested common subpath for 21 of 113 hosts that already served a verified root file.
Key takeaways
- 21 hosts returned a distinct H1 file on at least one tested path.
- The test covered four conventional paths, not every possible location.
- A subpath is useful only when it represents a real content boundary.
What did the benchmark find?
On 12 August 2026, 21 of the 113 verified root-file hosts also returned a distinct text file beginning with an H1 at one or more of these locations: /docs/llms.txt, /documentation/llms.txt, /help/llms.txt, or /api/llms.txt. The test excluded redirects, HTML fallback pages, responses without an H1 and bodies identical to the root file.
This is an observed count, not a global adoption rate. The denominator deliberately includes only hosts already present in the verified directory, and the numerator only includes the four tested paths. A host may publish at another location and remain invisible to this test.
How the test was run
The script started from the fixed directory snapshot, not a search-engine sample. For each origin, it requested four paths with an explicit text-oriented Accept header, followed redirects, and counted a result only when the response succeeded and contained a Markdown H1.
That H1 check reduces false positives from custom 404 pages and single-page-app fallbacks, but does not certify the full file. Use the validator for structure and inspect the response manually before relying on it.
The raw aggregate is published in v2-adoption-stats.json. The fixed input panel and the directory methodology remain public so later runs can use the same denominator.
When a subpath file is useful
A separate file makes sense when the content below a path has its own audience, vocabulary, or release cycle. Documentation, API references, support centers, and course areas are common candidates. A compatible agent working inside /docs/ can select /docs/llms.txt rather than load an unrelated company-wide map.
Do not create path files merely because the proposal permits them. Duplicate files increase maintenance cost and create conflicting descriptions. Define ownership, generate links from the same content source, and keep the root file broad enough to route an agent toward the specialist map.
The v2 migration checklist explains a safe rollout. The best-practice guide covers curation once the scope is established.
Limits and implementation checklist
The benchmark cannot tell whether an agent discovered or used any file. It does not test every path, authenticated areas, subdomains beyond their listed origin, or the semantic quality of the returned map. It also cannot attribute Search performance to path-level files.
For implementation:
- Choose a path with a stable information boundary.
- Keep the file within that path and link only relevant resources.
- Add
rel="describedby"on covered pages when practical. - Test the most specific-file rule against nested paths.
- Monitor requests in logs instead of assuming adoption.
Can I publish only /docs/llms.txt?
Yes. The v2 proposal supports files below the origin root. Whether clients discover it depends on their implementation and your discovery signals.
Should subpath files repeat the root list?
Usually no. Repetition removes the contextual advantage. Keep the root as a router and make the path file specific.
Does a subpath file help Google index that section?
Google says Search does not use llms.txt as a special signal. Use sitemaps, internal links, crawlability, and useful content for Search.