How to monitor llms.txt without misleading metrics
A trustworthy adoption tracker needs stable inputs and visible uncertainty. This method turns scheduled fetches into an auditable time series.
Last updated:
A useful monitor measures whether a fixed set of hosts serves a parseable file at a stated URL on a stated date. It must keep network failures separate from missing files and must never rename publication as adoption by agents, citation, or SEO impact.
Key takeaways
- Keep panel composition and URL rules versioned.
- Separate missing, HTML fallback, malformed, valid, and unreachable states.
- Publish raw inputs and aggregates so readers can reproduce the denominator.
What should the monitor measure?
Define one narrow question. Our tracker asks whether each named host serves a text response beginning with a Markdown H1 at https://host/llms.txt. It does not ask whether an agent reads the file or whether the file is useful.
The public panel contains 219 hosts. In the 3 August run, 218 were reachable and 113 met the root-file condition. The denominator for the 51.8% measured rate is therefore 218, not 219.
Use explicit response states
Do not collapse every failure into “no file.” Record at least:
- valid: successful text response with an H1;
- missing: explicit 404 or 410 response;
- restricted: 401 or 403 response;
- server error: 5xx response;
- other HTTP error: any remaining unsuccessful HTTP response;
- HTML: successful response that looks like an application shell or custom error page;
- malformed: text response without the minimum structure;
- unreachable: timeout or network error.
Only reachable states belong in a presence-rate denominator. This protects the metric from temporary network conditions.
Build the scheduled pipeline
Store the panel in version control with a category and human-readable name. Fetch in bounded concurrent batches, follow redirects, set a timeout, and use a transparent user agent. Preserve requested and final URLs. If response size must be bounded, stop reading the response stream rather than slicing only after a full download.
Write dated snapshots before generating the directory and aggregate. Run the job on a schedule, commit only measured changes, then pass the normal site build and QA gates. Do not deploy through a special path that bypasses production controls.
The link-health benchmark describes retry policy. The verified directory exposes the current successful subset.
Report without overclaiming
State the panel size, reachable base, valid count, date, paths tested, parser condition, and exclusions beside every percentage. Link the aggregate to adoption-stats.json and the input to adoption-panel.json.
Never infer agent behavior from publication. If you later add server-log evidence, report it as a separate dataset with its own denominator and bot-verification method.
Can I add hosts to the panel?
Yes, but version the change and avoid comparing pre-change and post-change rates as one continuous series.
Should network errors count as absent?
No. They reveal nothing about file presence.
Is a valid H1 enough to recommend a file?
No. It is a reproducible inclusion condition, not a quality endorsement.