- [ Home ](/) / - [ Blog ](/blog/) / - llms.txt vs robots.txt # `llms.txt` vs robots.txt: what's the difference? robots.txt tells compliant crawlers which URLs they may fetch. llms.txt offers compatible tools a curated list of resources to read. They serve different purposes, and neither file protects private content. Last updated: September 22, 2026 On this page - [ The differences at a glance ](#comparison) - [ What robots.txt actually controls ](#robots-txt) - [ Crawl blocking is not the same as preventing indexing ](#crawling-and-indexing) - [ What llms.txt adds ](#llms-txt) - [ Does llms.txt improve Google rankings? ](#google-rankings) - [ AI search, training and user requests have different policies ](#ai-crawler-policies) - [ How to use the two files together ](#complementary) The distinction matters when deciding what to publish, what to let crawlers retrieve, and what should appear in search. A crawl restriction is not a reliable way to remove a URL from Google. A resource list is not a guarantee that an AI service will discover or use your content. ## The differences at a glance Question robots.txt llms.txt What does it communicate? Crawl rules for particular user agents. Selected resources and context for tools that support the proposal. What is the format? Plain text with directives such as User-agent and Disallow . Markdown with a title and optional supporting sections. Where does it live? /robots.txt at the root of the relevant origin. /llms.txt , or a scoped path such as /docs/llms.txt . Is it standardized? The Robots Exclusion Protocol is specified in IETF RFC 9309. It is a community proposal published at llmstxt.org. Does it enforce access restrictions? No. The crawler must respect the rules. No. It is not a permission document. For a broader comparison that includes sitemaps, see [the roles of llms.txt, robots.txt, sitemap.xml and llms-full.txt](/comparison/). ## What robots.txt actually controls robots.txt expresses crawl rules. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), published on the IETF Standards Track in September 2022, defines the protocol and explicitly separates these rules from access authorization. For example, this file asks Googlebot not to crawl URLs under `/archive/`: ``` User-agent: Googlebot Disallow: /archive/ ``` It leaves other paths unrestricted for that user agent. These are instructions to a crawler, not server rules that prevent a visitor or another client from retrieving a page. The file also has a specific scope. `https://example.com/robots.txt` does not automatically set rules for every subdomain or protocol. Google documents scope by host, protocol and port. Its parser supports `user-agent`, `allow`, `disallow` and `sitemap`; it does **not** support `crawl-delay`. Check the documentation for the crawler you want to address instead of assuming every directive works everywhere. See [Google's robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec). ### Crawl blocking is not the same as preventing indexing Google can discover a blocked URL through a link and still index that URL without crawling its content. That is why [Google's robots.txt guidance](https://developers.google.com/search/docs/crawling-indexing/robots/intro) says crawl blocking is not a dependable way to keep a page out of search results. For a public page that should not appear in Google Search, use an appropriate indexing control such as `noindex`. **Google must be able to crawl the page to see its `noindex` rule.** Blocking the same URL in robots.txt can prevent that. Google supports `noindex` through a page's HTML or an HTTP response header, not as a robots.txt directive. Removal depends on Google fetching and processing the rule; adding it does not cause an immediate update. See [Google's noindex instructions](https://developers.google.com/search/docs/crawling-indexing/block-indexing). For confidential content, use authentication or another server-enforced access control. Neither robots.txt nor `noindex` makes a publicly accessible document private. ## What llms.txt adds llms.txt is an optional resource map for tools that support it. The [llms.txt proposal](https://llmstxt.org/) describes a Markdown file with a required H1 title; the description and other sections are optional. For example, this illustrative file points to two guides on this site: ``` # llmtxt.info > Guides to llms.txt, its format and its limitations. ## Guides - [Create an llms.txt file](https://llmtxt.info/how-to-create/): a practical setup guide. - [Compare discovery files](https://llmtxt.info/comparison/): the different roles of four web files. ``` This shows the structure using existing web pages. The proposal recommends linking to content suited to agents, such as clean Markdown versions when available. The proposal's v2 text, updated on August 10, 2026, allows files at scoped paths: `/docs/llms.txt` covers pages beneath `/docs/`, with the most specific file applying. It describes intended behavior, not proof that a particular client implements it. Publishing a file does not establish that an agent found it, read the linked pages or used them in an answer. For a useful implementation, identify the intended client and test that workflow. The file itself does not grant access or override robots.txt. ### Does llms.txt improve Google rankings? Google says it does not use llms.txt for Search and that the file does not improve visibility or rankings, including in its generative AI features. See [Google's guide to optimizing for generative AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide). That statement concerns Google Search; it does not establish how every other AI tool handles the file. ## AI search, training and user requests have different policies Treat the crawler name and purpose as part of the decision. [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots) distinguishes `OAI-SearchBot` for search from `GPTBot` for potential model training. Their robots.txt settings are independent. A robots.txt rule expressing that preference for GPTBot looks like this: ``` User-agent: GPTBot Disallow: / ``` This targets GPTBot; it is not a blanket block on every AI interaction. OpenAI also says robots.txt rules may not apply to user-initiated requests made through `ChatGPT-User`. A claim that robots.txt blocks all AI access would therefore be too broad. For more context on reading these policies, see [AI crawlers explained](/blog/ai-crawlers-explained/). ## How to use the two files together Start with the outcome you need. Set crawl preferences in robots.txt, handle search exclusion with the search engine's documented indexing controls, and protect private material at the server. Add llms.txt when you have a defined consumer or workflow for it. Keep the reading list small enough to maintain, point to accurate public documentation, and verify that both the file and the linked resources are accessible to the intended client. Our [guide to creating an llms.txt file](/how-to-create/) covers preparation and deployment. After a change, check the actual responses and client behavior. A syntax check can help review a file, but cannot establish that an external service consumes it or that an answer is accurate. The [llmtxt.info validator](/validator/) is one place to start that review; discovery, retrieval and downstream use still need their own evidence. ## Sources - [ IETF, RFC 9309 ](https://www.rfc-editor.org/rfc/rfc9309.html) - [ Google, robots.txt introduction ](https://developers.google.com/search/docs/crawling-indexing/robots/intro) - [ Google, robots.txt specification ](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec) - [ Google, noindex instructions ](https://developers.google.com/search/docs/crawling-indexing/block-indexing) - [ llms.txt proposal, v2 ](https://llmstxt.org/) - [ Google, generative AI search guidance ](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) - [ OpenAI, crawler documentation ](https://developers.openai.com/api/docs/bots)