- [ Home ](/) / - [ Blog ](/blog/) / - llms.txt vs robots.txt # llms.txt vs robots.txt, what's the difference? Two files, two different jobs. One publishes advisory crawl rules; the other offers a curated map. Neither is an access-control mechanism. Last updated: August 12, 2026 ## The short answer **robots.txt** publishes crawl rules for compliant clients such as Googlebot or Bingbot. It communicates which URLs they should or should not fetch, but it is not authorization or technical access control. **llms.txt** is a context layer. It tells AI language models which pages on your site are most useful for understanding what you do. It is not an access control mechanism at all, it does not grant or deny crawl permission. It is a curated reading list written in Markdown, proposed in September 2024 by Jeremy Howard at Answer.AI. Neither file replaces the other. They operate at different layers of the web stack and serve different audiences. ## What robots.txt does The `robots.txt` file lives at the root of a domain (`/robots.txt`) and uses the Robots Exclusion Protocol (REP) to communicate crawl permissions. A typical entry looks like this: ``` User-agent: Googlebot Disallow: /private/ Allow: / ``` The key characteristics of robots.txt: - **Audience:** Web crawlers of all kinds, search engine bots, AI training crawlers, link checkers, archiving bots. - **Format:** Plain text. Key-value pairs using a defined syntax (User-agent, Allow, Disallow, Crawl-delay, Sitemap). - **Purpose:** Crawl policy. It tells compliant crawlers what they should and should not fetch. - **Effect:** Compliant crawlers will not request Disallowed URLs. Non-compliant crawlers may ignore it. - **Standard status:** Widely adopted industry convention; an IETF informational RFC (RFC 9309) was published in 2022. - **What it does NOT do:** robots.txt does not tell crawlers what your content means, what your most important pages are, or what context they should use when answering questions about you. ℹ robots.txt is advisory, not enforced Compliant crawlers respect robots.txt by convention, not by technical enforcement. There is no server-side mechanism that automatically blocks disallowed bots. If you need actual access control, use authentication, IP allowlists, or server-side rules. ## What llms.txt does A `llms.txt` file can live at the domain root or a more specific path such as `/docs/llms.txt`. It is a Markdown document, not a key-value configuration file. A minimal valid file looks like: ``` # My Product > A short, factual description of what this site is about. ## Documentation - [Getting started](https://example.com/docs/start/): first steps for new users. - [API reference](https://example.com/docs/api/): complete endpoint documentation. ## Optional - [Blog](https://example.com/blog/): articles and updates. ``` The key characteristics of llms.txt: - **Audience:** Compatible agents, RAG pipelines and developer tools with explicit support for the convention. - **Format:** Markdown. A required title (H1), an optional blockquote description, and H2 sections containing Markdown link lists. - **Purpose:** Context and curation. It points compatible clients to selected resources; it does not guarantee accurate answers or downstream use. - **Effect:** Tools that read llms.txt use it as a starting point for fetching your content. It does not grant or restrict access. - **Standard status:** A community proposal. Not an IETF or W3C standard. Maintained at llmstxt.org by Jeremy Howard. - **What it does NOT do:** llms.txt does not control who can crawl your site. It does not improve your Google rankings. It is not a robots.txt replacement. ## Side-by-side comparison Attribute robots.txt llms.txt Location /robots.txt /llms.txt or a scoped path Format Plain text, key-value pairs Markdown Primary audience All web crawlers AI/LLM clients and agent frameworks Function Advisory crawl rules Context and curation (what to read) Enforces crawl access? No No Affects Google ranking? Indirectly (blocking crawl prevents indexing) No Formal standard? IETF RFC 9309 Community proposal (llmstxt.org) Year introduced 1994 2024 Required sections User-agent + Disallow/Allow H1 title (everything else optional) ## Why they are complementary The crawl-policy layer and the optional resource-map layer can coexist without conflict. A careful deployment looks like this: - **Use robots.txt to express crawl preferences.** If you want compatible AI systems to read your content, make sure you have not accidentally blocked AI crawler user-agents in robots.txt. Many sites added blanket bot blocks during the 2023–2024 AI training data controversy; review your rules to ensure legitimate retrieval crawlers (as opposed to training crawlers, if you wish to distinguish) can access your public pages. - **Use llms.txt for a defined compatible client.** The map can offer a curated shortcut, but publication does not prove the client discovers, reads or follows it. - **Make sure llms.txt itself is not blocked.** If your robots.txt disallows bots from your root, they may not be able to fetch `/llms.txt` either. Verify that your robots.txt does not prevent access to the file you want AI systems to read. ✓ Quick check Run a validation on your llms.txt using the llmtxt.info validator , then check your domain's llms.txt status to confirm it is reachable. Cross-reference with your robots.txt to confirm AI crawlers are not blocked. ## Frequently asked questions ### Does llms.txt replace robots.txt? No. robots.txt publishes advisory crawl rules for compliant clients. llms.txt offers a curated map to compatible agents. Use either file only for the problem it actually solves. ### Can I use llms.txt to block AI crawlers? No. llms.txt has no access-control function. To block specific AI crawlers, add their user-agent strings to your robots.txt Disallow rules. For example, to block GPTBot: ``` User-agent: GPTBot Disallow: / ``` llms.txt is not read as a permission document. Publishing it does not grant crawlers any additional access they did not already have. ### Do AI crawlers respect robots.txt? Providers publish crawler-specific robots.txt guidance, but REP remains advisory. A `Disallow` rule communicates intent to compliant clients. If content must not be fetched, protect it with authentication, network policy or server-side authorization. ## Continue reading - [llms.txt vs robots.txt vs sitemap.xml](/comparison/), full three-way comparison. - [AI crawlers explained](/blog/ai-crawlers-explained/), how GPTBot, ClaudeBot, and PerplexityBot work. - [How to create llms.txt](/how-to-create/), step-by-step with templates. - [Validator](/validator/), check your existing llms.txt file. ## Sources - [ llmstxt.org, official spec ](https://llmstxt.org/) - [ Google, robots.txt reference ](https://developers.google.com/search/docs/crawling-indexing/robots/intro) - [ Jeremy Howard, llms.txt proposal (Answer.AI) ](https://answer.ai/posts/2024-09-03-llmstxt.html) On this page - [ The short answer ](#overview) - [ What robots.txt does ](#robots-txt) - [ What llms.txt does ](#llms-txt) - [ Side-by-side comparison ](#comparison) - [ Why they are complementary ](#complementary) - [ Frequently asked questions ](#faq)