Partner study · September 2026
Detecting llms.txt files: what an analysis of 2,000 domains shows
This study with Firecrawl identified 192 recognizable files across 2,000 domains and examines why a successful HTTP response still requires a content check.
· llmtxt.info
What the study measures
A website can publish an llms.txt text file to introduce its content and point language-model tools to useful pages, although publishing the file does not guarantee that an AI system will discover or use it.
A request to /llms.txt can succeed without returning the file you expect. This study, conducted with Firecrawl, examines that distinction across 2,000 domains selected from Tranco, a domain popularity ranking designed for research. The main observations were collected on 2 September 2026.
Combining a Firecrawl extraction with a direct HTTP check identified 192 files, or 9.60% of the sample, at the address tested on each domain. This measures file detection; it does not show whether an AI system reads the files or whether they improve search visibility or answers. All study counts below are supplied by our research partner; the source and respective contributions are identified at the end of this page.
This is a separate study from the Observatory’s recurring, curated panel. Its population and recognition rules differ. The two sets of percentages should not be combined or treated as a trend.
Three outcomes across 2,000 domains
Read the chart values as a table
| Outcome | Count | Share of base |
|---|---|---|
| Recognized files | 192 | 9.60% |
| Testable addresses without a recognized file | 1,226 | 61.30% |
| Addresses that could not be tested | 582 | 29.10% |
For 192 domains, a file was recognized. Another 1,226 addresses could be examined without a file being recognized: 965 returned an HTTP 404 error, indicating a resource not found, and 261 fell into other categories.
The remaining 582 addresses could not be tested. A block or timeout does not establish that a file is absent. This is why the 1,808 domains without a recognized file cannot all be described as confirmed absences.
There are two useful ways to state the result, provided the calculation base stays explicit:
| Calculation base | Recognized files | Proportion |
|---|---|---|
| All sampled domains | 192 / 2,000 | 9.60% |
| Testable addresses only | 192 / 1,418 | 13.54% |
The second proportion is higher because its calculation excludes the 582 inaccessible addresses, which may differ systematically from accessible ones; the number of recognized files stays the same. The calculated 95% Wilson intervals are 8.39% to 10.97% for all 2,000 domains and 11.86% to 15.42% for the 1,418 testable addresses. These intervals describe statistical uncertainty under a binomial model. They do not account for classification errors or selection and access biases, and should not be read as an estimate for the entire web.
Why an HTTP 200 response is not enough
An HTTP 200 status tells you that the server returned a successful response. It does not identify the response as an llms.txt file.
Of 405 direct HTTP 200 responses, the pilot recognizes 177 files. The other 228 responses, or 56.30%, belong to four different categories. This total is specific to the direct HTTP check; the overall 192-file result combines both methods.
Read the chart values as a table
| Outcome | Count | Share of base |
|---|---|---|
| Recognized files | 177 | 43.70% |
| Redirect anomalies | 154 | 38.02% |
| Soft 404s | 60 | 14.81% |
| Generic HTML shells | 7 | 1.73% |
| Invalid formats | 7 | 1.73% |
- 154 redirect anomalies: the destination path does not meet the pilot’s rule for a
.txt,.mdor.markdownending. This classifies the address; it does not by itself prove that the destination is broken. - 60 soft 404s: error pages returned with a successful status.
- 7 generic HTML application shells: an application response rather than the expected text file.
- 7 invalid formats: a separate category of unrecognized content.
These categories require different investigations. Checking the final address, response content and structure is more informative than counting success codes alone. Percentages in the chart are rounded independently.
What comparing the two methods adds
Read the chart values as a table
| Outcome | Count | Share of base |
|---|---|---|
| Both sources | 176 | 91.67% |
| Firecrawl only | 15 | 7.81% |
| Direct HTTP only | 1 | 0.52% |
Of the 192 files, 176 are recognized by both sources, 15 by Firecrawl only and one by direct HTTP only. The chart uses 192 files as its base, unlike the first chart’s 2,000 domains or the HTTP chart’s 405 responses.
Among the 16 files recognized by only one source, 15 involve blocked or unreachable access for the other source. These are not two conflicting readings of the same content: one method lacks a response it can assess. In the remaining case, Firecrawl recognizes a file while direct HTTP classifies its response as a soft 404; the pilot includes a manual adjudication of that case.
The time limits were 30 seconds for Firecrawl and 10 seconds for direct HTTP. This asymmetry is a known limitation of the method used in the pilot. Its effect has not been established in either direction. Firecrawl states that a replication using matched timeouts is planned.
What to check on your own site
Use the findings as a reading guide for your checks. These actions are practical recommendations, not measured outcomes of the pilot.
Your address returns HTTP 200
Read the response body to check whether it contains the expected text or Markdown file, an application shell or an error message. Once you have identified the content, inspect its headings and links to assess the file itself.
Your request redirects
Inspect the final address and the content returned there. The pilot’s extension rule is a detection rule, not a universal test of whether a redirect is useful or a page works.
Your request is blocked or times out
Keep the result separate from “file absent”. Record the access failure and repeat the check when appropriate. A later response can clarify accessibility, but does not reconstruct the site’s historical state.
Your file is recognized
Check its contents and links for usefulness and accuracy. File detection alone does not demonstrate use by an agent. Any claim about citations, search visibility or answer quality needs separate measurements.
Fetch a public file to inspect its response, or validate text you already have. These site tools perform their own checks; their results are not a rerun of the historical pilot. For the wider distinction between publishing and use, read Adoption is not usage.
Method and limitations
The sample was selected deterministically from a Tranco list of one million domains. The same rules applied to the same list reproduce the selection. Each request targets the exact selected host over HTTPS at /llms.txt. No additional search covers www variants, HTTP, documentation subdomains or other paths.
Recognition targets a successful text or Markdown response, not HTML, with at least one H1 heading outside a code block. In Markdown, an H1 is a top-level heading written as # Heading; a code block is a section displayed as an example of code or literal text. It is the study’s detection criterion, not full format certification.
The original detector could mistake an H1 inside a code block for a file heading. Firecrawl re-examined the affected detector logic and confirmed that none of the 192 recognized files was affected. Firecrawl reports that 189 files could be verified from the stored data; the remaining three were fetched again, verified by checksum as byte-identical to the original 2 September responses, and reclassified without changing the results. According to Firecrawl, the separate text error-page case beginning with # Page not found also affected none of the positive results.
The pilot also records an opening H1 in 174 of the 192 files (90.63%). These structural metrics use Firecrawl’s extracted representation for 191 files and direct HTTP content for one. This is not a measure of content quality or agent usefulness.
The two sources’ different time limits do not support a conclusion that one tool is superior. The observations also do not establish a definite effect on search visibility, citations or response quality. These questions require further measurements and will provide the basis for future reports.
Sources and contributions
Results of the llms.txt pilot on 2,000 domains, September 2026, supplied by Firecrawl. Main observations collected on 2 September. Firecrawl is the research partner and supplies the counts, classifications and information on the classification checks, including its confirmation of 16 September 2026. llmtxt.info provides and remains responsible for the analysis and interpretation, including the calculated proportions. This research credit does not mean that Firecrawl endorses every statement.
This page provides the results and their calculation method. The detailed observations are retained within the partnership and are not disclosed; their classifications therefore cannot be checked individually from this publication.
- Tranco: a domain ranking designed for research. This link explains the sampling frame; it is not the pilot dataset.
For questions about these findings, use the site’s corrections page.
When citing this study: llmtxt.info, Detecting llms.txt files: what an analysis of 2,000 domains shows. Research partner: Firecrawl. Observations: 2 September 2026. Keep the calculation base and classification limits with any quoted figure.