Home / Blog # llms.txt blog In-depth articles on the llms.txt spec, AI crawlers, generative engine optimization, and practical implementation guides for developers and site owners. llms.txt 01 Featured reading 2026-09-29 ## [Google ignores llms.txt. Shopify still ships it.](/blog/google-ignores-llms-txt-shopify-ships-it/) Google does not use llms.txt, while Shopify publishes it for compatibility. The difference is consumers, data layers, and evidence. Read article → Study with Firecrawl ## [Detecting llms.txt files: what an analysis of 2,000 domains shows](/observatory/firecrawl-llms-txt-2000-domains/) 192 files recognized across 2,000 domains. Study with Firecrawl: charts, method, limits and practical checks for your site. Read the study → 192 / 2 000 - 2026-09-21 ## [llms.txt v2 adoption: the first public baseline](/blog/llms-txt-v2-adoption-baseline/) A dated baseline across 113 verified files and 294 linked pages measures subpath files, Markdown alternates, describedby and link reachability. [ Read article → ](/blog/llms-txt-v2-adoption-baseline/) - 2026-09-17 ## [How to monitor llms.txt without misleading metrics](/blog/how-to-monitor-llms-txt/) Build a fixed-panel llms.txt monitor with explicit denominators, reachability states, snapshots and reproducible reports that separate publication from use. [ Read article → ](/blog/how-to-monitor-llms-txt/) - 2026-09-13 ## [llms.txt adoption by sector: fixed-panel data](/blog/llms-txt-adoption-by-sector/) A fixed panel of 218 reachable hosts shows large sector differences in root llms.txt publication. Read the exact rates, method and sampling limits. [ Read article → ](/blog/llms-txt-adoption-by-sector/) - 2026-09-09 ## [llms.txt redirects: what 113 files reveal](/blog/llms-txt-redirects-benchmark/) Among 113 H1-first root responses, 33 redirected and 31 crossed origins. Learn why final URLs matter and how to monitor migrations without overstating validity. [ Read article → ](/blog/llms-txt-redirects-benchmark/) - 2026-09-07 ## [Search Console AI report: what it actually measures](/blog/search-console-ai-report/) Read the Search Console AI report without confusing impressions, clicks and missing data. A tracking template and the limits of llms.txt. [ Read article → ](/blog/search-console-ai-report/) - 2026-09-05 ## [llms.txt validator errors: reject or warn?](/blog/llms-txt-validator-errors/) A v2-based guide to validator severity: what makes a file invalid, what deserves a warning, and what 113 production files reveal about optional conventions. [ Read article → ](/blog/llms-txt-validator-errors/) - 2026-09-01 ## [How large is a real llms.txt file?](/blog/llms-txt-size-benchmark/) A 113-file benchmark measures real llms.txt size, sections and links: 14,347 median bytes, 7 sections and 76 links, with wide variation. [ Read article → ](/blog/llms-txt-size-benchmark/) - 2026-08-31 ## [AI crawlers explained: intent, evidence, and llms.txt](/blog/ai-crawlers-explained/) Distinguish search, training, agent, transaction, SEO, and ads crawlers, then learn what a request to llms.txt can actually prove. [ Read article → ](/blog/ai-crawlers-explained/) - 2026-08-28 ## [llms.txt link health: 294 URLs tested](/blog/llms-txt-link-health-benchmark/) A reproducible health check of 294 URLs linked from verified llms.txt files found 286 reachable pages. Learn what to test and what the result cannot prove. [ Read article → ](/blog/llms-txt-link-health-benchmark/) - 2026-08-24 ## [How agents discover llms.txt v2 resources](/blog/llms-txt-discovery-relations/) A benchmark of Markdown alternates and rel=describedby across 294 pages, including HTML and HTTP Link headers, with deployment advice and limits. [ Read article → ](/blog/llms-txt-discovery-relations/) - 2026-08-20 ## [llms.txt subpaths: a 113-site benchmark](/blog/llms-txt-subpaths-benchmark/) We tested four common llms.txt subpaths across 113 verified hosts. After excluding redirects, HTML fallbacks and root duplicates, 21 hosts served a distinct path file. [ Read article → ](/blog/llms-txt-subpaths-benchmark/) - 2026-08-16 ## [llms.txt v2 migration: a practical checklist](/blog/llms-txt-v2-migration/) Migrate to the August 2026 llms.txt v2 proposal: path-specific files, Markdown page alternates, discovery relations, validation and safe rollout. [ Read article → ](/blog/llms-txt-v2-migration/) - 2026-08-10 ## [Adoption is not usage: reading the llms.txt studies properly](/blog/llms-txt-adoption-is-not-usage/) Why llms.txt adoption measures range from 10.13% to 51.8%: each study samples a different population, and publication is not consumption. [ Read article → ](/blog/llms-txt-adoption-is-not-usage/) - 2026-08-07 ## [Threat-modeling llms.txt prompt injection](/blog/llms-txt-prompt-injection/) Ahrefs observed a research user agent named prompt-injection-survey. The label does not prove an attack, but external agent context deserves supply-chain controls. [ Read article → ](/blog/llms-txt-prompt-injection/) - 2026-08-04 ## [Who fetches llms.txt? Agents, not AI search crawlers](/blog/who-actually-reads-llms-txt/) In Ahrefs logs for 137,210 domains, AI retrieval bots made 1.1% of llms.txt requests and agentic tools 10.5%. Fetching still does not prove use. [ Read article → ](/blog/who-actually-reads-llms-txt/) - 2026-08-01 ## [What Chrome's Lighthouse llms.txt audit actually checks](/blog/lighthouse-llms-txt-audit/) Lighthouse treats a 4xx response as Not Applicable. For a successful response it also checks minimum length, one H1 and one Markdown link, but not full v2 conformance. [ Read article → ](/blog/lighthouse-llms-txt-audit/) - 2026-07-29 ## [AI bots did not probe missing llms.txt files in one study](/blog/agents-do-not-look-for-llms-txt/) In Ahrefs' May 2026 sample, no AI bot requested a missing /llms.txt. The result is not universal proof, but it shows why publishing alone is not discovery. [ Read article → ](/blog/agents-do-not-look-for-llms-txt/) - 2026-06-10 ## [State of llms.txt adoption in 2026: measured weekly](/blog/state-of-llms-txt-2026/) We fetch and parse /llms.txt on a fixed panel of 219 notable hosts every week. Around half serve the file, with sharp differences by sector. Raw data published. [ Read article → ](/blog/state-of-llms-txt-2026/) - 2026-04-22 ## [llms.txt vs robots.txt, what's the difference?](/blog/llms-txt-vs-robots-txt/) robots.txt publishes crawl rules; llms.txt offers compatible agents a curated map. Learn how the files differ in format, purpose and enforcement. [ Read article → ](/blog/llms-txt-vs-robots-txt/) - 2026-04-22 ## [GEO: Generative Engine Optimization, what it is and why it matters](/blog/what-is-geo/) Generative Engine Optimization (GEO) is the practice of optimizing content to appear in AI-generated answers. Learn how it differs from traditional SEO and where llms.txt fits in. [ Read article → ](/blog/what-is-geo/) - 2026-04-22 ## [llms.txt best practices: write a clear, maintainable file](/blog/llms-txt-best-practices-2025/) A practical guide to llms.txt: absolute URLs, factual descriptions, clear sections and regular updates. Publishing a file does not establish use by an AI system. [ Read article → ](/blog/llms-txt-best-practices-2025/) - 2026-04-22 ## [llms.txt for API-first products](/blog/llms-txt-for-api-products/) Developer tools and API products benefit especially from llms.txt because developers ask AI assistants about APIs. A guide with a full example template. [ Read article → ](/blog/llms-txt-for-api-products/) - 2026-04-22 ## [llms-full.txt, what it is, when to use it, and how to create one](/blog/llms-full-txt-guide/) llms-full.txt is the companion file that inlines your full page content. Learn when it makes sense, how it differs from llms.txt, and who uses it. [ Read article → ](/blog/llms-full-txt-guide/) - 2026-04-22 ## [How to measure the impact of your llms.txt file](/blog/measure-llms-txt-impact/) No direct analytics dashboard exists yet, but you can triangulate. Track AI referral traffic, monitor bot fetch logs, and test citation accuracy manually. [ Read article → ](/blog/measure-llms-txt-impact/) - 2026-04-22 ## [llms.txt for documentation sites](/blog/llms-txt-documentation-sites/) Docs sites are the ideal use case for llms.txt. Learn what to include, what to exclude, and how to structure your file so AI assistants give accurate answers. [ Read article → ](/blog/llms-txt-documentation-sites/) Start here Understand llms.txt Create a file Check its structure Sources and method Methodology Observatory Corrections ## Practical guides 4 October 2026 · EN ### [Test HTML and Markdown delivery through your cache](/blog/markdown-cache-representation-testing/) Check whether your CDN serves the intended HTML and Markdown representations. A practical test matrix for explicit URLs, negotiation and cache validation. 4 October 2026 · EN ### [Versioned docs: keep language and API release separate](/blog/versioned-documentation-language-routing/) Design llms.txt links for multilingual, versioned documentation without sending readers to the wrong API release or treating unrelated pages as translations.