/ llmtxt.info
Français

llms.txt vs robots.txt — what's the difference?

Two files, two completely different jobs. One controls access; the other provides context. Here is how they compare and why you need both.

Last updated:

The short answer

robots.txt is a permission layer. It tells web crawlers — Googlebot, Bingbot, AI crawlers — which URLs on your site they are allowed to fetch. It is an access control mechanism that has been part of the web since 1994.

llms.txt is a context layer. It tells AI language models which pages on your site are most useful for understanding what you do. It is not an access control mechanism at all — it does not grant or deny crawl permission. It is a curated reading list written in Markdown, proposed in September 2024 by Jeremy Howard at Answer.AI.

Neither file replaces the other. They operate at different layers of the web stack and serve different audiences.

What robots.txt does

The robots.txt file lives at the root of a domain (/robots.txt) and uses the Robots Exclusion Protocol (REP) to communicate crawl permissions. A typical entry looks like this: 

User-agent: Googlebot
Disallow: /private/
Allow: /

The key characteristics of robots.txt:

  • Audience: Web crawlers of all kinds — search engine bots, AI training crawlers, link checkers, archiving bots.
  • Format: Plain text. Key-value pairs using a defined syntax (User-agent, Allow, Disallow, Crawl-delay, Sitemap).
  • Purpose: Access control. It tells crawlers what they may and may not fetch.
  • Effect: Compliant crawlers will not request Disallowed URLs. Non-compliant crawlers may ignore it.
  • Standard status: Widely adopted industry convention; an IETF informational RFC (RFC 9309) was published in 2022.
  • What it does NOT do: robots.txt does not tell crawlers what your content means, what your most important pages are, or what context they should use when answering questions about you.

What llms.txt does

The llms.txt file also lives at the domain root (/llms.txt), but it is a Markdown document — not a key-value configuration file. A minimal valid file looks like: 

# My Product

> A short, factual description of what this site is about.

## Documentation

- [Getting started](https://example.com/docs/start/): first steps for new users.
- [API reference](https://example.com/docs/api/): complete endpoint documentation.

## Optional

- [Blog](https://example.com/blog/): articles and updates.

The key characteristics of llms.txt:

  • Audience: AI language models, agent frameworks, RAG pipelines, and developer tools (Cursor, Windsurf) that want to understand your site.
  • Format: Markdown. A required title (H1), an optional blockquote description, and H2 sections containing Markdown link lists.
  • Purpose: Context and curation. It points AI systems to the pages that best represent your content so they can give accurate answers about you.
  • Effect: Tools that read llms.txt use it as a starting point for fetching your content. It does not grant or restrict access.
  • Standard status: A community proposal. Not an IETF or W3C standard. Maintained at llmstxt.org by Jeremy Howard.
  • What it does NOT do: llms.txt does not control who can crawl your site. It does not improve your Google rankings. It is not a robots.txt replacement.

Side-by-side comparison

Attribute robots.txt llms.txt
Location /robots.txt /llms.txt
Format Plain text, key-value pairs Markdown
Primary audience All web crawlers AI/LLM clients and agent frameworks
Function Access control (allow/deny crawling) Context and curation (what to read)
Controls crawl access? Yes No
Affects Google ranking? Indirectly (blocking crawl prevents indexing) No
Formal standard? IETF RFC 9309 Community proposal (llmstxt.org)
Year introduced 1994 2024
Required sections User-agent + Disallow/Allow H1 title (everything else optional)

Why they are complementary

The permission layer (robots.txt) and the context layer (llms.txt) work together without conflict. Here is what a thoughtful deployment looks like:

  1. Use robots.txt to grant AI crawlers access. If you want AI systems to read your content, make sure you have not accidentally blocked AI crawler user-agents in robots.txt. Many sites added blanket bot blocks during the 2023–2024 AI training data controversy; review your rules to ensure legitimate retrieval crawlers (as opposed to training crawlers, if you wish to distinguish) can access your public pages.
  2. Use llms.txt to tell AI systems which pages matter. Once an AI client can access your site, llms.txt gives it a curated shortcut. Instead of crawling hundreds of pages to find your most important content, the client reads llms.txt and loads the five to fifteen pages you have identified as the most authoritative.
  3. Make sure llms.txt itself is not blocked. If your robots.txt disallows bots from your root, they may not be able to fetch /llms.txt either. Verify that your robots.txt does not prevent access to the file you want AI systems to read.

Frequently asked questions

Does llms.txt replace robots.txt?

No. robots.txt and llms.txt serve entirely different purposes. robots.txt is a permission layer that tells crawlers which URLs they may or may not access. llms.txt is a context layer that tells AI systems which pages best represent your site. You need both.

Can I use llms.txt to block AI crawlers?

No. llms.txt has no access-control function. To block specific AI crawlers, add their user-agent strings to your robots.txt Disallow rules. For example, to block GPTBot: 

User-agent: GPTBot
Disallow: /

llms.txt is not read as a permission document. Publishing it does not grant crawlers any additional access they did not already have.

Do AI crawlers respect robots.txt?

The major AI crawlers — GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot — have publicly stated they respect robots.txt. This means disallowing their user-agents in robots.txt is an effective way to prevent them from crawling your content. Whether all AI crawlers do so is a separate question; some less scrupulous crawlers may not.

Continue reading

Sources