- [ Home ](/) / - Comparison # llms.txt vs robots.txt, sitemap.xml, and llms-full.txt Four files, four jobs. Here is exactly what each one does, and how to use them together. Last updated: August 12, 2026 ## TL;DR They are not interchangeable. **`robots.txt`** publishes crawl rules that compliant clients may apply. **`sitemap.xml`** tells search engines what exists. **`llms.txt`** offers compatible clients a curated resource map. **`llms-full.txt`** hands them the actual content. ℹ Use them together, not against each other robots.txt and sitemap.xml have established crawler and search uses. Add llms.txt only when a maintained, agent-oriented map serves a defined use case. ## Side-by-side matrix Quick reference. See sections below for the nuances. Criterion robots.txt sitemap.xml llms.txt llms-full.txt Primary purpose Crawl instructions for compliant crawlers Discovery of pages for search engines Curated map for LLM clients Inline corpus for LLM ingestion Audience Web crawlers (Googlebot, Bingbot, GPTBot…) Search engines Compatible agents and LLM clients LLM clients needing full content Format Plain text, custom REP grammar XML Markdown Markdown (concatenated) Standard? Yes, IETF RFC 9309 (2022) Yes, sitemaps.org Community proposal, llmstxt.org Community proposal, llmstxt.org Required? No, but recommended No, but recommended for large sites No No Enforces access or indexing? No (allow / disallow are crawler instructions) No (just discovery hint) No No Approach Exclusion Discovery (be exhaustive) Curation (be selective) Inlining (provide full text) File path /robots.txt /sitemap.xml (or any URL declared in robots.txt) /llms.txt or a scoped path such as /docs/llms.txt /llms-full.txt ## llms.txt vs robots.txt `robots.txt` is a **crawler-instruction file** standardized as [IETF RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html). It uses the Robots Exclusion Protocol grammar (`User-agent`, `Disallow`, `Allow`, `Sitemap`) to express which paths compliant crawlers should or should not fetch. It does not enforce access and is not an authorization layer. `llms.txt` is the **opposite intent**: a positive recommendation list. It does not block anyone, it does not grant access, and it has no effect on whether a crawler fetches anything else on your site. It just says: if you are an LLM client, here is the high-quality subset. Practical implication: continue to use `robots.txt` for crawl preferences and sitemap discovery. Use authentication or network controls for actual access restrictions. Add `llms.txt` as a complement only when it has a maintained audience and purpose. ## llms.txt vs sitemap.xml `sitemap.xml` aims for **completeness**: it lists every URL you want a search engine to know about, plus metadata (`lastmod`, `priority`, alternate languages). It is XML, machine-only, often auto-generated. `llms.txt` aims for **curation**: a small Markdown list of the pages an LLM should read first. It does not replace your sitemap. It rarely includes more than a few dozen URLs, while a sitemap on a content-heavy site can list hundreds of thousands. Think of `sitemap.xml` as a directory and `llms.txt` as a recommended-reading shelf curated by a librarian. ## llms.txt vs llms-full.txt Same family, different role: - **`llms.txt`** is the *map*: a list of titled links. - **`llms-full.txt`** is the *territory*: the actual content of those (and other) pages, concatenated as Markdown into one file. The `llms-full.txt` convention was popularized by Mintlify in collaboration with Anthropic. It can give a developer or tool one URL for a consolidated documentation corpus. That convenience must be weighed against freshness, size and prompt-injection exposure. Publish `llms.txt` when a curated map solves a real task. Add `llms-full.txt` only when bulk textual ingestion is an explicit requirement. ## llms.txt vs schema.org / JSON-LD Schema.org is a vocabulary for marking up the meaning of *individual pages* in JSON-LD or microdata. Search engines and other software use it to extract structured facts: a product’s price, a recipe’s ingredients, a FAQ’s questions and answers. `llms.txt` operates one level above: it’s a **site-wide map**, not a per-page enrichment. The two are complementary. Schema.org tells an LLM what a page *is*; `llms.txt` tells it which pages to look at first. ## How to combine them - Publish **both** `robots.txt` and `sitemap.xml` as you already do for SEO. - Add a root or path-scoped **`llms.txt`** for a defined compatible client. - Optionally add **`llms-full.txt`** if your site is documentation- or knowledge-heavy. - In `robots.txt`, leave `/llms.txt` and `/llms-full.txt` accessible (do not `Disallow` them). - Keep **schema.org** markup on individual pages where it makes sense (Product, FAQ, Article…). ⚠ Common mistake Some teams try to replace their sitemap with llms.txt . Don’t. Search major search engines document sitemap support, not llms.txt as a replacement, and sitemap protocols provide metadata such as lastmod that llms.txt intentionally omits. ## Next - [How llms.txt works](/how-it-works/), the spec in detail. - [Best practices](/best-practices/). - [FAQ](/faq/). ## Sources - [ IETF RFC 9309, Robots Exclusion Protocol ](https://www.rfc-editor.org/rfc/rfc9309.html) - [ sitemaps.org, Sitemap protocol ](https://www.sitemaps.org/protocol.html) - [ llmstxt.org, official llms.txt spec ](https://llmstxt.org/) - [ Search Engine Land, llms.txt is a treasure map for AI ](https://searchengineland.com/llms-txt-isnt-robots-txt-its-a-treasure-map-for-ai-456586) On this page - [ TL;DR ](#tldr) - [ Side-by-side matrix ](#matrix) - [ vs robots.txt ](#vs-robots) - [ vs sitemap.xml ](#vs-sitemap) - [ vs llms-full.txt ](#vs-llms-full) - [ vs schema.org / JSON-LD ](#vs-schema) - [ How to combine them ](#combine)