Featured reading
Google ignores llms.txt. Shopify still ships it.
Google does not use llms.txt, while Shopify publishes it for compatibility. The difference is consumers, data layers, and evidence.
Read articlellms.txt blog
In-depth articles on the llms.txt spec, AI crawlers, generative engine optimization, and practical implementation guides for developers and site owners.
Study with Firecrawl
Detecting llms.txt files: what an analysis of 2,000 domains shows
192 files recognized across 2,000 domains. Study with Firecrawl: charts, method, limits and practical checks for your site.
Read the study-
llms.txt v2 adoption: the first public baseline
A dated baseline across 113 verified files and 294 linked pages measures subpath files, Markdown alternates, describedby and link reachability.
Read article -
How to monitor llms.txt without misleading metrics
Build a fixed-panel llms.txt monitor with explicit denominators, reachability states, snapshots and reproducible reports that separate publication from use.
Read article -
llms.txt adoption by sector: fixed-panel data
A fixed panel of 218 reachable hosts shows large sector differences in root llms.txt publication. Read the exact rates, method and sampling limits.
Read article -
llms.txt redirects: what 113 files reveal
Among 113 H1-first root responses, 33 redirected and 31 crossed origins. Learn why final URLs matter and how to monitor migrations without overstating validity.
Read article -
Search Console AI report: what it actually measures
Read the Search Console AI report without confusing impressions, clicks and missing data. A tracking template and the limits of llms.txt.
Read article -
llms.txt validator errors: reject or warn?
A v2-based guide to validator severity: what makes a file invalid, what deserves a warning, and what 113 production files reveal about optional conventions.
Read article -
How large is a real llms.txt file?
A 113-file benchmark measures real llms.txt size, sections and links: 14,347 median bytes, 7 sections and 76 links, with wide variation.
Read article -
AI crawlers explained: intent, evidence, and llms.txt
Distinguish search, training, agent, transaction, SEO, and ads crawlers, then learn what a request to llms.txt can actually prove.
Read article -
llms.txt link health: 294 URLs tested
A reproducible health check of 294 URLs linked from verified llms.txt files found 286 reachable pages. Learn what to test and what the result cannot prove.
Read article -
How agents discover llms.txt v2 resources
A benchmark of Markdown alternates and rel=describedby across 294 pages, including HTML and HTTP Link headers, with deployment advice and limits.
Read article -
llms.txt subpaths: a 113-site benchmark
We tested four common llms.txt subpaths across 113 verified hosts. After excluding redirects, HTML fallbacks and root duplicates, 21 hosts served a distinct path file.
Read article -
llms.txt v2 migration: a practical checklist
Migrate to the August 2026 llms.txt v2 proposal: path-specific files, Markdown page alternates, discovery relations, validation and safe rollout.
Read article -
Adoption is not usage: reading the llms.txt studies properly
Why llms.txt adoption measures range from 10.13% to 51.8%: each study samples a different population, and publication is not consumption.
Read article -
Threat-modeling llms.txt prompt injection
Ahrefs observed a research user agent named prompt-injection-survey. The label does not prove an attack, but external agent context deserves supply-chain controls.
Read article -
Who fetches llms.txt? Agents, not AI search crawlers
In Ahrefs logs for 137,210 domains, AI retrieval bots made 1.1% of llms.txt requests and agentic tools 10.5%. Fetching still does not prove use.
Read article -
What Chrome's Lighthouse llms.txt audit actually checks
Lighthouse treats a 4xx response as Not Applicable. For a successful response it also checks minimum length, one H1 and one Markdown link, but not full v2 conformance.
Read article -
AI bots did not probe missing llms.txt files in one study
In Ahrefs' May 2026 sample, no AI bot requested a missing /llms.txt. The result is not universal proof, but it shows why publishing alone is not discovery.
Read article -
State of llms.txt adoption in 2026: measured weekly
We fetch and parse /llms.txt on a fixed panel of 219 notable hosts every week. Around half serve the file, with sharp differences by sector. Raw data published.
Read article -
llms.txt vs robots.txt, what's the difference?
robots.txt publishes crawl rules; llms.txt offers compatible agents a curated map. Learn how the files differ in format, purpose and enforcement.
Read article -
GEO: Generative Engine Optimization, what it is and why it matters
Generative Engine Optimization (GEO) is the practice of optimizing content to appear in AI-generated answers. Learn how it differs from traditional SEO and where llms.txt fits in.
Read article -
llms.txt best practices: write a clear, maintainable file
A practical guide to llms.txt: absolute URLs, factual descriptions, clear sections and regular updates. Publishing a file does not establish use by an AI system.
Read article -
llms.txt for API-first products
Developer tools and API products benefit especially from llms.txt because developers ask AI assistants about APIs. A guide with a full example template.
Read article -
llms-full.txt, what it is, when to use it, and how to create one
llms-full.txt is the companion file that inlines your full page content. Learn when it makes sense, how it differs from llms.txt, and who uses it.
Read article -
How to measure the impact of your llms.txt file
No direct analytics dashboard exists yet, but you can triangulate. Track AI referral traffic, monitor bot fetch logs, and test citation accuracy manually.
Read article -
llms.txt for documentation sites
Docs sites are the ideal use case for llms.txt. Learn what to include, what to exclude, and how to structure your file so AI assistants give accurate answers.
Read article
Practical guides
Test HTML and Markdown delivery through your cache
Check whether your CDN serves the intended HTML and Markdown representations. A practical test matrix for explicit URLs, negotiation and cache validation.
Versioned docs: keep language and API release separate
Design llms.txt links for multilingual, versioned documentation without sending readers to the wrong API release or treating unrelated pages as translations.