- [ Home ](/) / - [ Blog ](/blog/) / - AI crawlers explained # AI crawlers explained: intent, evidence, and `llms.txt` Classifying an AI crawler requires more than reading its name. The operator's declared purpose, technical identity, and your own logs answer different questions. Last updated: August 31, 2026 On this page - [ A crawler name is not enough to establish its purpose ](#purpose) - [ What a request to llms.txt can prove ](#evidence) - [ Why intent-level distinctions matter ](#openai) - [ Where llms.txt fits ](#llms-txt) - [ Continue reading ](#next) ## A crawler name is not enough to establish its purpose The useful questions are what the operator says a bot does, how the bot proves its identity, and what your own evidence shows. The label "AI bot" covers several kinds of activity. Cloudflare's verified-bot taxonomy shows the range: Search, Agent, Training, Transact, Data Collection, Security Testing, SEO, Ads Verification, Social or Link Preview, Feed Fetching, and Monitoring or Operations. A bot may declare more than one behavior. Those categories help organize a policy. They do not prove that one request led to a citation, trained a model, or completed a transaction. ## What a request to llms.txt can prove A successful request to `/llms.txt` can establish a short list of server-side facts: the request reached the server, it carried a recorded user-agent, it targeted that URL, and the server returned a recorded status at a recorded time. That log entry does not establish that the client: - parsed the file correctly; - followed every link; - preferred the file over HTML or structured data; - used the content for retrieval or training; - cited the website in an answer; - changed the ranking of a page or product. Treat the evidence as a ladder: `fetched`, `parsed` if independently demonstrated, `used` if independently demonstrated, then `cited` or `converted` if the destination system measures it. Skipping a step turns a technical observation into an unsupported claim. Note Identity, intent, and observation are separate A verified identity tells you who sent a request. A declared behavior says why the operator says it accesses content. Logs show what reached your server. None of these proves every later use of the content. ## Why intent-level distinctions matter OpenAI documents separate user-agents for search, model training, user-triggered actions, and advertising review. `OAI-SearchBot` supports ChatGPT search. `GPTBot` covers content that may be used to train foundation models. `ChatGPT-User` performs certain user-requested actions and is not an automatic search crawler. `OAI-AdsBot` visits only pages submitted as ads. OpenAI says that data collected by this bot is not used to train foundation models. Advertising review is therefore a different purpose from search visibility or training. These roles still do not prove that any OpenAI bot relies on `llms.txt`. The file is useful only where a client chooses to read it. ## Where llms.txt fits `llms.txt` can provide a concise, owner-selected map of useful pages. It does not replace HTML, sitemaps, feeds, APIs, structured data, platform catalogs, or crawler directives in `robots.txt`. Each surface has its own role. Shopify is a useful example. Its storefronts can publish agent discovery files while Shopify Catalog separately carries structured product facts. The presence of either surface does not guarantee that a channel will show or rank a product. Publish `llms.txt` for clients that support the convention, then verify only what your evidence can support. File availability is not proof of adoption, and adoption is not proof of a citation. ## Continue reading - [AI crawlers, complete reference](/llms-txt-ai-crawlers/), the canonical technical reference for user-agents and controls. - [llms.txt vs robots.txt](/blog/llms-txt-vs-robots-txt/), the difference between discovery guidance and crawler control. - [How to measure llms.txt impact](/blog/measure-llms-txt-impact/), a method for reading crawler logs without claiming more than they show. ## Sources - [ Cloudflare, Verified bots ](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) - [ OpenAI, Overview of OpenAI Crawlers ](https://developers.openai.com/api/docs/bots) - [ Shopify, agent discovery files and Catalog ](https://help.shopify.com/en/manual/online-sales-channels/agentic-storefronts/products) - [ llmstxt.org, community proposal ](https://llmstxt.org/)