llms.txt for documentation sites

Why docs sites benefit most

Documentation sites — the kind hosted on Mintlify, GitBook, Docusaurus, or custom-built with tools like MkDocs or Nextra — are the ideal use case for llms.txt. The reason is straightforward: developers routinely use AI assistants to navigate documentation. Questions like "how do I set up authentication with Acme?", "what are the webhooks supported by Acme?", and "show me the Acme SDK for Python" are documentation questions.

When a developer asks these questions in an AI assistant, the assistant either draws from training data (which may be months old), retrieves live content from the web, or loads documentation from a tool like Cursor that has fetched llms.txt. In all three cases, the quality of the answer depends on the quality of the documentation it has access to.

llms.txt is your mechanism for telling AI systems: "here are the pages that matter most for understanding this product." For a documentation site, that is exactly the curation a developer-facing AI needs.

What to include

For documentation sites, the following page types consistently belong in llms.txt:

• Quickstart guide — the page most new users and AI assistants reach first. Should walk through the minimal path to getting something working. If you only include one page, include this one.
• Core concepts — the page or pages that explain the mental model of your product. What are the fundamental entities? How does the product's data model work? What is the key terminology?
• API reference — the complete reference for your API endpoints, parameters, and response schemas. If it is a single long page, link the top level. If it is paginated by resource type, link the resource-level pages.
• SDK and client library guides — one link per officially supported language or platform.
• Authentication and authorization — how users and apps authenticate. One of the most common documentation questions AI assistants are asked.
• FAQ or troubleshooting guide — if you have a comprehensive FAQ, it is worth linking. AI assistants can use it to answer common questions accurately.
• Changelog — helps retrieval systems distinguish current behavior from outdated training data.

What to exclude

Not every page in your documentation site belongs in llms.txt:

• Internal notes or team documentation — if your docs platform hosts both public-facing docs and internal team wikis, include only the public documentation.
• Draft or unlisted pages — pages that are not yet ready for users should not be in llms.txt. An AI that cites a draft page is citing something you have not approved for use.
• Auth-gated content — documentation behind a login wall cannot be fetched by AI crawlers or agent frameworks. Include only pages that are publicly accessible.
• Deprecated documentation — if you maintain documentation for old versions of your product, exclude those pages or mark them clearly. You do not want AI assistants teaching your users deprecated patterns.
• Very granular sub-pages — if your API reference has 200 individual endpoint pages, do not link all 200. Link the top-level reference and let the AI navigate from there. llms.txt is a curation layer, not a sitemap.
• Marketing or sales pages — pricing, case studies, and comparison pages are useful content, but they are not documentation. If you include them, put them in the ## Optional section.

Priority ordering

Within each section of your llms.txt, the order of links matters. AI clients that read your file treat earlier links as higher priority. When they operate under context length constraints, they may stop reading mid-file. Put your most essential pages first.

A suggested priority order for developer documentation sites:

1. Quickstart / Getting started
2. Core concepts / Architecture overview
3. API reference (top-level or most-used resource)
4. Authentication / Authorization
5. SDK guides (most-used language first)
6. Webhooks reference (if applicable)
7. Troubleshooting / FAQ
8. Changelog (in Optional section)
9. Migration guides (in Optional section)

Auto-generating from your navigation

For docs sites with many pages, maintaining llms.txt manually is impractical. The better approach is to auto-generate it at build time from your documentation structure. The general approach:

1. Use your navigation configuration as the source. Most documentation platforms define navigation in a configuration file (sidebar.json, mint.json, mkdocs.yml, etc.). This configuration already represents your curated, ordered view of the documentation — it is a natural input for generating llms.txt.
2. Filter to top-level and second-level items. Do not include every leaf node in your navigation tree. Pull the top-level sections and their direct children. This typically gives you 10–30 pages, which is the right range for llms.txt.
3. Map each item to an absolute URL. Your navigation config likely uses relative paths. Transform them to absolute URLs using your configured base URL.
4. Use page titles as link text and page descriptions as link descriptions. If your pages have meta descriptions, use them. If not, use the first paragraph of each page.
5. Output to your public directory. Write the generated file to wherever your static files live so it is served at /llms.txt.

Run this generation at every documentation build so llms.txt stays in sync with your navigation structure automatically.

How Mintlify handles it

Mintlify is a documentation platform widely used by developer-facing companies (many AI-era developer tools use it). Mintlify automatically generates both llms.txt and llms-full.txt for every documentation site hosted on its platform.

Conceptually, Mintlify's generation works as follows:

• The site's navigation is defined in a mint.json configuration file. This file specifies the page hierarchy, including which pages appear in which sections of the sidebar.
• At build time, Mintlify reads the navigation structure and generates a llms.txt file where each nav item becomes a link entry in the appropriate section.
• For llms-full.txt, Mintlify inlines the full Markdown content of each page below its link, giving AI retrieval systems the complete documentation corpus in one file.
• Both files are deployed alongside the documentation site and served at the standard paths (/llms.txt and /llms-full.txt).

The result is that documentation sites on Mintlify get spec-compliant llms.txt files automatically, without any manual curation. The tradeoff is that the file reflects the navigation structure, not a hand-curated priority order — which may include low-priority pages that a manual author would have placed in the Optional section or excluded entirely.

Other documentation platforms — GitBook, Docusaurus, ReadMe, and others — have varying levels of llms.txt support. Check your platform's documentation or release notes for the current state of support.

Example structure

Below is an example llms.txt for a hypothetical developer documentation site. This structure works for most product documentation sites with a REST API:

# Acme Documentation

> Acme is a platform for real-time inventory management. This documentation covers the
> REST API, SDKs for Python and Node.js, and integration guides for common e-commerce
> platforms. The API is used by developers building stock tracking, warehouse management,
> and demand forecasting applications.

## Getting started

- [Introduction](https://docs.acme.example/introduction/): what Acme is and how it works.
- [Quickstart](https://docs.acme.example/quickstart/): create your first integration in five minutes.
- [Authentication](https://docs.acme.example/authentication/): API key setup and OAuth 2.0.
- [Core concepts](https://docs.acme.example/concepts/): products, locations, stock records, and events.

## API reference

- [API overview](https://docs.acme.example/api/): base URL, versioning, and conventions.
- [Products](https://docs.acme.example/api/products/): create, read, update, and delete products.
- [Stock adjustments](https://docs.acme.example/api/stock/): record stock movements and reconcile inventory.
- [Webhooks](https://docs.acme.example/api/webhooks/): event types, payloads, and signature verification.
- [Rate limits](https://docs.acme.example/api/rate-limits/): limits by plan tier.
- [Errors](https://docs.acme.example/api/errors/): error codes and handling recommendations.

## SDKs

- [Python SDK](https://docs.acme.example/sdk/python/): official Python client library.
- [Node.js SDK](https://docs.acme.example/sdk/node/): official Node.js client library.

## Optional

- [Changelog](https://docs.acme.example/changelog/): release notes and breaking changes.
- [FAQ](https://docs.acme.example/faq/): common questions from developers.
- [Migration guide (v1 to v2)](https://docs.acme.example/migration/): upgrading from v1.

Continue reading

• llms.txt for API products — deeper guide for API-first products.
• llms-full.txt guide — the full corpus companion file for docs sites.
• How to create llms.txt — step-by-step with per-stack deployment guides.
• Generator — build a spec-compliant file from a form.