AI crawlers explained: intent, evidence, and llms.txt
Classifying an AI crawler requires more than reading its name. The operator's declared purpose, technical identity, and your own logs answer different questions.
Last updated:
A crawler name is not enough to establish its purpose
The useful questions are what the operator says a bot does, how the bot proves its identity, and what your own evidence shows. The label "AI bot" covers several kinds of activity.
Cloudflare's verified-bot taxonomy shows the range: Search, Agent, Training, Transact, Data Collection, Security Testing, SEO, Ads Verification, Social or Link Preview, Feed Fetching, and Monitoring or Operations. A bot may declare more than one behavior.
Those categories help organize a policy. They do not prove that one request led to a citation, trained a model, or completed a transaction.
What a request to llms.txt can prove
A successful request to /llms.txt can establish a short list of server-side facts: the
request reached the server, it carried a recorded user-agent, it targeted that URL, and the server
returned a recorded status at a recorded time.
That log entry does not establish that the client:
- parsed the file correctly;
- followed every link;
- preferred the file over HTML or structured data;
- used the content for retrieval or training;
- cited the website in an answer;
- changed the ranking of a page or product.
Treat the evidence as a ladder: fetched, parsed if independently demonstrated,
used if independently demonstrated, then cited or
converted if the destination system measures it. Skipping a step turns a technical observation
into an unsupported claim.
Why intent-level distinctions matter
OpenAI documents separate user-agents for search, model training, user-triggered actions, and
advertising review. OAI-SearchBot supports ChatGPT search. GPTBot
covers content that may be used to train foundation models. ChatGPT-User performs certain
user-requested actions and is not an automatic search crawler.
OAI-AdsBot visits only pages submitted as ads. OpenAI says that data collected by this
bot is not used to train foundation models. Advertising review is therefore a different purpose from
search visibility or training.
These roles still do not prove that any OpenAI bot relies on llms.txt. The file is
useful only where a client chooses to read it.
Where llms.txt fits
llms.txt can provide a concise, owner-selected map of useful pages. It does not replace
HTML, sitemaps, feeds, APIs, structured data, platform catalogs, or crawler directives in
robots.txt. Each surface has its own role.
Shopify is a useful example. Its storefronts can publish agent discovery files while Shopify Catalog separately carries structured product facts. The presence of either surface does not guarantee that a channel will show or rank a product.
Publish llms.txt for clients that support the convention, then verify only what your
evidence can support. File availability is not proof of adoption, and adoption is not proof of a citation.
Continue reading
- AI crawlers, complete reference, the canonical technical reference for user-agents and controls.
- llms.txt vs robots.txt, the difference between discovery guidance and crawler control.
- How to measure llms.txt impact, a method for reading crawler logs without claiming more than they show.