Markdown for agents: what changes in October
The conversion engine changed, the limit rose to 6 MiB and token counters disappeared. What the update reveals about architecture and editorial responsibility.
Last updated:
On October 9, 2026, Cloudflare added a short entry to its developer changelog about Markdown for Agents, the feature that converts an HTML page into Markdown on the fly when a client says it prefers that format. The entry describes a new in-process engine that streams at the edge of the network: it handles content as it arrives instead of buffering the HTML response and handing it to a separate conversion service. It also announces three concrete changes. The conversion limit rises from 2 MiB to 6 MiB of HTML, measured after decompression. The x-markdown-tokens and x-original-tokens headers are no longer generated. And Content-Length is now removed from converted responses rather than recalculated, because the body is streamed.
The reference documentation, updated the same day, adds that headers describing the original body are stripped from the converted representation because they no longer match it. Among them are ETag and Last-Modified, and the stated reason is explicit: conditional requests cannot be honored for a converted response.
Read quickly, this is an infrastructure note. Read by someone responsible for a site’s editorial output, it is a chance to reopen a question many sites settled too fast. To be readable by agents, does a site need an llms.txt file, Markdown pages published alongside its HTML pages, a full export of its content, or is it enough to let an intermediary convert the existing HTML? Those options do not answer the same need. The October 9 change removes the token counters; the current documentation also states that the origin’s validators are not passed through. Together, these findings take away two props that made automatic conversion look like a strategy in its own right.
This piece is not a configuration guide. The practical procedures already exist: our French-language HTML/Markdown parity recipe covers content checks, and our guide to testing Markdown representations in the cache covers variants and purges. The aim here is to examine the architectural choice this change brings into focus, the oversimplifications it exposes and the editorial responsibility it leaves untouched. Throughout, the text separates what the vendor documents, what llmtxt.info observed in its October 19 panel, and what is our own analysis. It promises neither better citation nor better ranking: none of the sources consulted establishes either.
What the October 9 changelog actually says
The first step is to separate what is written from what is often inferred. The changelog describes a change of engine and three visible consequences. Cloudflare expects lower conversion overhead and lower memory use, without publishing figures. Nothing in the entry says the Markdown output is more faithful, more complete or better understood by models than before. This is an efficiency gain for the network operator, claimed by the operator, not a demonstrated improvement in the editorial quality of the output.
Each of the three consequences deserves a careful reading. The first is a higher ceiling: 6,291,456 bytes of decompressed HTML, up from 2,097,152. The phrase “after decompression” matters. A page that looks light once compressed on the wire can be much larger once decompressed, and it is the decompressed size that is measured. The second is the disappearance of the two headers that expressed a token count; the changelog simply says that clients relying on those values now need to calculate token counts themselves. The third, the removal of Content-Length, follows from streaming: the final size is not known when the headers go out.
The documentation adds the full list of transformations. The content type becomes text/markdown; charset=utf-8. The Vary header includes Accept, while preserving any dimensions the origin already declared, so that caches store the HTML and Markdown variants separately. Content-Length, Content-Encoding, Content-Range, Transfer-Encoding, ETag and Last-Modified are removed. Conversely, the origin’s security and caching headers are described as preserved: Strict-Transport-Security, Content-Security-Policy, X-Frame-Options, Set-Cookie, CORS headers, and Cache-Control, Expires and Age.
Two limits are stated in plain terms. Only HTML is converted, with other document types possibly supported later, and the 6 MiB ceiling applies to decompressed HTML. On availability, the page as consulted on October 21 says the feature is offered at no extra cost on the Pro, Business and Enterprise plans and to SSL for SaaS customers. Commercial terms change and should be checked again at integration time; this article assumes no purchase.
One point of method to close: the feature was introduced in February, and material from that period describes headers that no longer exist. The reference is now the documentation dated October 9. Any internal dashboard or script built on an earlier description should be reviewed against it.
Discovery, representation, corpus: three distinct needs
The most common confusion around “Markdown for AI” comes from one word being used for three different things. The first is a discovery object: a document that helps an agent learn what exists and where to start. The second is a representation: another form of a given page, meant for a client that prefers structured text to HTML. The third is a corpus: a large body of assembled content handed to a tool in one piece.
The llms.txt file belongs to the first category. The current version of the proposal published at llmstxt.org describes it as a Markdown file placed at the root of a site or at any path within it, covering the pages under that path, offering brief background, guidance and links to more detailed files. The file stays small enough to fit in context; the detail lives behind the links and is fetched only when needed. Where more than one file applies, the agent should use the most specific one. Only the H1 title is required. A blockquote summary and H2 sections holding lists of links are optional, including an “Optional” section for material an agent can skip when its context is tight.
Markdown for Agents belongs to the second category. It says nothing about how the site is organized. It answers a specific request for a specific URL and returns another form of the same resource: the client adds text/markdown to its Accept header, and the network fetches the HTML from the origin and converts it before serving it.
The llms.txt proposal also provides for representations, in a different shape: a clean Markdown version of each useful page, published at the original URL with .md appended or substituted for the extension, or index.html.md for URLs without a file name. It recommends advertising them with standard link relations, rel="alternate" type="text/markdown" for the Markdown version and rel="describedby" for the llms.txt file that covers the page, expressed either as HTML elements or as an HTTP Link header, the framework defined in RFC 8288.
The corpus, for its part, is not defined as a format by either source. It is a practice: a whole documentation set concatenated into one file, an export for a developer tool, an archive handed to a partner. The llms.txt proposal explains why it cannot replace an index: context windows, though larger than they were, are still too small for most websites in their entirety, and every wasted token costs time and money. It makes the same argument about sitemaps, which generally cover too much and include a great deal that is not needed to understand a site.
These three objects sit on different floors of the same building. An index helps an agent choose; a representation helps it read what was chosen; a corpus serves those who want everything and know what they will do with it. The mistake is asking a conversion to do the curation work of an index, or asking an index to stand in for readable pages. The Cloudflare documentation pages consulted for this article in fact combine a pointer to an llms.txt index, a “View as Markdown” link and negotiation on Accept; OpenAI’s page on its crawlers points to its llms.txt and to Markdown versions reachable by appending .md to the URL. These are isolated observations on technical documentation sites, not a web standard.
Content negotiation: an old idea that shifts responsibility
Markdown for Agents rests on a mechanism as old as HTTP/1.1. RFC 9110, which defines current HTTP semantics, distinguishes the resource, identified by a URL, from its representations, which reflect its state in a form that can be transferred. A resource can have several representations, and the content negotiation described in Section 12 selects the one best suited to a request. That selected representation supplies the data and metadata on which conditional requests rely.
Serving Markdown at the same URL as the HTML is therefore entirely orthodox, and even elegant: one address to cite, share and correct. But the elegance has a cost. With negotiation, the representation an agent receives depends on a request header the publisher does not always see in its logs, on configuration set on a third-party network, and on a conversion whose rules the publisher does not write. With a .md file published next to the page, the representation is an object the team produces, versions and can proofread.
That is the first shift of responsibility. Turning on conversion amounts to saying: the Markdown version of our pages is whatever this algorithm produces from our HTML. Publishing .md files amounts to saying: it is what we chose to write. Both positions are defensible, but they do not commit a publisher in the same way. In the first case, quality depends on the semantics of the HTML, on how well the template isolates the main content and on the converter’s choices. In the second, it depends on a publishing pipeline that has to be maintained and checked.
Negotiation also raises a caching question. A negotiated response has to declare in Vary the request headers that influenced the selection, otherwise a shared cache may serve the HTML variant to a client that wanted Markdown, or the reverse. Section 4.1 of RFC 9111 forbids reusing a stored response without revalidation when the headers nominated by Vary do not match. Cloudflare says it adds Accept to Vary on the converted response. What remains is to check, on every delivery chain, that intermediate caches honor this and that the HTML response declares the same dimension. That is the subject of our existing cache testing guide, which this article does not rewrite.
What the converter keeps and what it drops
The documentation describes output with a stable structure, designed so that AI systems can rely on it without site-specific parsing logic. It has three blocks, in this order.
The first is YAML frontmatter, emitted only if the page contains at least one supported meta tag. It has three fields. The title comes from <meta name="title">, falling back to og:title. The description comes from <meta name="description">, falling back to og:description. The image comes from og:image. The standard form always wins over Open Graph, regardless of tag order, and only fields with a value are emitted.
The second block is the body, converted from the document body. Elements treated as non-content, such as headers, footers, navigation, scripts and styles, are stripped during a pre-processing step that the documentation refers to the Workers AI conversion service’s description. The third block, emitted only if the page has any, gathers JSON-LD structured data into a single fenced json code block at the end, with multiple scripts concatenated each on its own line. JSON-LD is the only <script> content kept.
That description makes it possible to list what does not come through in structured form. The frontmatter contains no author, no publication or update date, no language, no canonical URL and no alternate language versions. That information can survive in the visible body text, if the template displays it there, or in JSON-LD, if the page publishes it. It does not appear in the documented fields. This is an inference from the published list, not an observation of a particular page, and it should be checked against one’s own templates.
Stripping headers, footers and navigation is mostly good news for an agent: the noise repeated on every page goes away. For a publisher, it is a potential loss. Many sites put things in those zones that matter for understanding a text: the section name, the breadcrumb that places an article in a series, a correction notice shown as a banner, a warning that a page documents an old version of a piece of software. Nothing guarantees that such information will reach the converted representation. The only way to know is to compare, template by template, what the HTML displays and what the Markdown contains.
The quality of an automatic conversion is therefore decided upstream, in the template. HTML that marks up its main content with real headings, real lists and real tables will plausibly yield more usable Markdown than a page built out of generic blocks. That is not a promise of results, only the ordinary consequence of a converter working with the structure it is given. The new engine does not change that dependency.
The end of token counters: measuring without making things up
The removal of x-markdown-tokens and x-original-tokens deserves particular attention. Those headers offered a ready-made figure, visible in a single request, that could be dropped into a slide to illustrate the supposed gain of Markdown over HTML.
That convenience is exactly what deserves scrutiny. A token count is not a property of a text. It is the result of segmentation by a particular tokenizer, and several models can share the same tokenizer. The count for the same document can therefore vary with the tokenizer used. A header published by a network intermediary could therefore only express an estimate under some convention: useful as an order of magnitude, misleading once turned into a performance indicator. The changelog does not justify the removal on those grounds, but the practical outcome is the same. Any numerical claim about token savings now has to state which tokenizer, which text and which method produced it.
This article therefore offers no ratio between HTML and Markdown, and the sources consulted provide none that generalizes. Cloudflare’s documentation states that Markdown’s explicit structure reduces token waste and leads to better results; that is the vendor’s position, with no measurement in support on the page. The llms.txt proposal makes the same general point. Neither quantifies an effect on answer quality, still less on the likelihood of being cited.
So what can be measured without inventing anything? The size in bytes of the decompressed HTML response and of the Markdown response for a given page is an objective measure that does not depend on any model. The number of requests received with text/markdown in Accept is measurable if logs record that header, which is not the default everywhere. The share of converted responses, status codes, the proportion served from cache and the load on the origin are ordinary operational metrics. Content parity, finally, is measured by comparison: is this paragraph, this table, this date present in both representations?
These are measures of delivery, not of effect. They tell you what the site serves and to whom, not what the agent does with it afterwards. A volume of Markdown requests does not prove use by a given product, and a lighter file is not necessarily better understood. Whatever effect a choice has on visibility inside an assistant cannot be read off response headers.
Validators removed: freshness, corrections and caching
Dropping ETag and Last-Modified is less spectacular, but its editorial consequences are more direct. RFC 9110 defines these headers as validators, Last-Modified in Section 8.8.2 and ETag in Section 8.8.3. They let a client that already holds a copy ask, through If-Modified-Since or If-None-Match, whether that copy is still valid; the server can then answer 304 Not Modified without resending the body. Section 13 describes this conditional request mechanism.
Cloudflare’s reasoning is consistent with that framework: the origin’s validators describe the HTML body, not the Markdown produced by conversion. Keeping them would have suggested that a conditional request was possible on a representation the network cannot compare. This explains their absence in the documented conversion, which has three implications editorial teams should understand.
The first concerns freshness. A client can no longer revalidate this representation using a conditional request based on those validators. Other freshness information remains usable; when a new fetch is needed, it cannot benefit from that conditional validation. Freshness then rests on Cache-Control, Expires and Age, which the documentation says are preserved. A site that relied on revalidation to get corrections to agents quickly should review the lifetimes it announces: a long one saves bandwidth but delays corrections, a short one does the opposite. That is an editorial trade-off as much as a technical one.
The second concerns dates. Last-Modified often reflected a deployment rather than a revision of the text, but it was still a time signal attached to the response. On the converted representation it is gone. If the update date matters, as it does for versioned documentation, a pricing page or a policy text, it has to be carried by the content itself: shown in the main body, or declared in JSON-LD, for example through dateModified, which the converter reproduces at the end of the document. The frontmatter has no date field.
The third concerns corrections. When an error is fixed in the HTML, the converted representation follows, since it is produced from the origin. But the two variants are stored separately in caches. A purge that targets only one variant, or an intermediate cache that ignores the Accept dimension, can leave an old Markdown version in circulation. The sections of the documentation consulted do not describe this case in detail; it has to be tested on one’s own chain, using the method in our cache guide.
6 MiB: a limit that says something about pages
The higher ceiling will be welcome for sites where some pages exceeded the old threshold. But what makes an HTML page weigh several megabytes once decompressed? Rarely the editorial text. More often it is data embedded for client-side hydration, inline styles, images encoded into the document, sprawling menus, scripts. Much of that volume will be stripped by pre-processing, yet the ceiling applies to the incoming HTML, not to the outgoing Markdown. A page whose useful text would fit in a few tens of kilobytes can therefore hit the limit because of everything around it.
For an editorial team, the useful question is not “are our pages under 6 MiB?” but “why are some of them getting close?” Legitimate cases exist: an exhaustive API reference, a long report published as a single page, an archive page. For those pages, a deliberately produced Markdown representation, or a split into sub-pages indexed by an llms.txt file, is likely to be more useful than a full conversion, even a successful one. An agent looking for one specific parameter does not need to receive the entire reference.
The documentation consulted does not describe, in the sections read, what a client receives when a page exceeds the limit; it only says conversion happens “when possible.” It would be unwise to assume that the original HTML is returned, that an error is returned or that the output is truncated. A site affected should observe the behavior on its own zone, on a real page, and record the date of the observation.
The ceiling also concerns only one floor of the building. An llms.txt file is meant to stay short; the llmtxt.info panel observed on October 19 records a median of 14,542 bytes for the files served, a different order of magnitude altogether from the conversion limit. A full corpus is generally produced as a plain text file, outside HTML negotiation. The 6 MiB figure only matters for HTML pages converted on the fly.
Versions, languages, dates and images: the metadata that does not come through
The sites that are hardest to convert are not necessarily the heaviest, but those that publish several states of the same content: languages, versions, updates, images that carry information.
On a well-built multilingual site, each edition has its own URL, and the lang attribute on the html element declares the language, and the <head> the canonical URL and alternates. Negotiation on Accept leaves that organization intact: the Markdown version of a French page is served at the French URL. That is a real advantage over separately produced .md files, which have to reproduce the URL structure and its alternate links. But the documented frontmatter contains neither the language nor the alternates. An agent knows what the page says, not necessarily which language it was published in or where its equivalents live. If that matters, the information has to be in the content or in JSON-LD, for example through inLanguage.
Versions raise a related problem. Multi-version documentation often places the version switcher and the “you are viewing an older version” warning in the header or navigation, precisely the zones pre-processing removes. An agent could receive the content of an obsolete version without the signal that qualifies it. Here the llms.txt proposal offers a better-suited tool: one file per subpath, such as /docs/v2/llms.txt, with the most specific file taking precedence. Our article on routing for versioned and multilingual documentation covers that organization in detail. The state of a piece of content, its version and its validity, belongs in the content.
A visible “last updated” date, moreover, is only worth something if it is maintained. A date regenerated at every deployment gives an impression of freshness while saying nothing about revisions to the text. The conversion will reproduce it as is; it will not correct it.
Images are the most delicate point. The only image field in the frontmatter is og:image, the sharing image, not the article’s figures. The sections consulted do not detail how images in the body are handled. One can only reason from principle: a converter can pass on only what is written about an image, its address, its alt text, its caption if marked up as text. A chart whose message exists only in pixels risks losing its information for a text-only reader. An agent able to inspect the image separately may have other ways to read it. This is the requirement accessibility has imposed for years: if a figure carries information, that information has to exist as text. Decoration, on the other hand, does not need describing; an empty alt attribute suits it.
Access and rights: what conversion does not settle
Conversion is a matter of format. It decides neither who may access content nor what it may be used for. Confusing those planes is the costliest oversimplification of all, because it touches on rights.
First, a usage signal is added by default. If the origin response carries no content-signal header, Markdown for Agents adds Content-Signal: ai-train=yes, search=yes, ai-input=yes, signaling that the content may be used for AI training, for search results and as AI input, including agentic use. If the origin sets its own value, that value is preserved and is authoritative. The Content Signals framework expresses preferences; it is not an enforcement mechanism. But a publisher that turns on conversion without defining its own policy is, in effect, publishing a signal that favors training on its Markdown representations. Our French-language article on publication policy and data use discusses those choices.
That point should be read alongside another announcement. On September 15, 2026, Cloudflare introduced three independent controls for search, AI training and AI agents, a Disallow AI Training setting, and the replacement of its managed robots.txt feature with Bot Preference Sync. The sources consulted do not describe how those settings interact with the header added by default to converted responses. Claiming an inconsistency would be as risky as claiming consistency. Each site should compare the signal actually emitted on a Markdown response with the policy it believes it is applying.
Second, robots.txt is not access authorization. RFC 9309, which standardizes the Robots Exclusion Protocol, says so in its introduction and reminds readers that controlling access requires a real security measure at the application layer. Not every client is subject to the same rules, either. OpenAI distinguishes OAI-SearchBot, used for search, GPTBot, tied to training, and ChatGPT-User, which acts during user-initiated actions and for which the company says robots.txt rules may not apply. The format of the response changes none of that.
Third, and more delicate: what the HTML contains is not always what the reader sees. Some sites deliver the full text of an article in the HTML and then hide it with a script or a stylesheet, for a paywall or a registration wall. The converter works on the HTML received from the origin and strips scripts and styles. By construction, text that is present in the HTML but hidden on the client side is liable to end up in the Markdown. That is an inference from the documented behavior, not an observation of any particular site, but it is enough to require a check before enabling conversion on restricted content. What should not be read should not be sent. Likewise, since Set-Cookie is preserved, a site that personalizes its responses needs to make sure a user-specific representation cannot be served to others from a cache.
A final reminder, as Cloudflare presented on October 15 the early results of a discovery pilot with OpenAI launched in July: according to Cloudflare, a signal about a page does not grant permission to access it, and the work relies on discovery metadata rather than on sharing page content for model training. The same reasoning applies to Markdown. Offering a readable representation is neither consent to every use nor a guarantee of being found.
What our observations show, and what they do not
llmtxt.info data llmtxt.info maintains a fixed panel of public hosts, observed at set dates. The dated data copy provided with this article, observatory-2026-10-19.json, corresponds to the observation of October 19, 2026. The panel has 219 hosts; 217 were tested and 2 were unreachable. 125 served an llms.txt file, or 57.6% of tested hosts. That percentage does not describe the web: the panel is a curated panel, deliberately weighted toward developer tools, as the data file itself states. The median size of the files served is 14,542 bytes, and those are bytes, not tokens.
The same observation looks at relations between files and pages. According to the method in the frozen discovery data, up to three linked pages and four common subpaths were tested for each verified root file, and discovery relations were checked in both the HTML and HTTP Link headers. Of 330 links checked, 321 were reachable at the time. 46 pages exposed an alternate relation to a Markdown version, 18 a describedby relation, and 23 hosts published llms.txt files on subpaths.
These figures say something precise and limited. The mechanisms the llms.txt proposal offers for linking index and representations are in use by part of this panel, and subpath files, useful for versions and languages, are not a theoretical curiosity. Nine of the 330 links were unreachable at the time of the check. This snapshot does not establish whether they worked before or how long they had been failing. On the other hand, the data does not say what proportion of the pages examined carried those relations: the file does not provide that denominator, and we do not reconstruct one. Nor does it say that an agent read those pages, that they were cited or that they gained traffic; the file itself rules out any conclusion about agent use or search performance.
A separate study, observed on September 2 across 2,000 domains and published in our observatory, found 192 recognizable files. Its sample and method differ, so the two observations do not form a time series. Neither measures Markdown for Agents, a network feature that stays invisible until it is queried with the right header.
The real cost: four architectures to maintain
Debates about Markdown for agents usually focus on the supposed gain and rarely on maintenance cost, which is what separates a sustainable decision from an announcement. Four architectures are on the table, often in combination.
| Architecture | What it provides | Main cost | Watch points: October 9 changelog and current documentation |
|---|---|---|---|
| HTML only | A single, canonical representation | Nothing beyond the existing site | Agents convert on their own, with their own rules |
| On-the-fly conversion | A Markdown representation at the same URL | Configuration, parity checks, dependence on the vendor and plan | No validators, default usage signal, 6 MiB input limit |
Published .md files |
A representation written and versioned by the publisher | Generation pipeline, alternate links, continuous parity checks |
Unaffected by the change, but must stay aligned with the HTML |
| llms.txt index | A selective map of the site or a subpath | Curation, link review, keeping descriptions current | Unaffected by the change; links to representations need checking |
HTML only is not the absence of a choice: it means letting each client convert with its own tools, without control over what it extracts from a cluttered page.
On-the-fly conversion is cheap to set up and not cheap to verify. It requires checking parity on every template, understanding how variants are cached, defining one’s own content-signal, and accepting that a vendor can change the behavior, as it did on October 9. That dependence is not a flaw in itself, but it has to be acknowledged and monitored: a header change in a changelog becomes an event to track, like a library update.
Published .md files offer the most control and the most durable cost. They have to be produced at every publication and correction, exposed through alternate relations, possibly listed in an llms.txt file, and checked so they do not drift from the HTML. A Markdown file that keeps an outdated price or instruction after the page has been corrected is worse than no file at all, because it looks like the reference version. In return, the publisher controls its validators: a static file served by its own server can carry an ETag and a date that genuinely belong to it.
An llms.txt index costs mostly editorial time: choosing, describing, ranking, reviewing. The volume is small, but it cannot be fully automated without losing what makes the file worthwhile, which is selection. An index generated mechanically from the sitemap adds little that the sitemap does not already offer. The proposal recommends testing a file by asking an agent questions with nothing but that file as a starting point: a qualitative method, not a benchmark, but one that puts the decision in the right place.
The full corpus is not in the table because it follows a different logic: a derived product for a specific audience, such as a library’s developers or partners, who know what they will do with it. Its main cost is freshness. A full export ages all at once and has to be corrected all at once.
Three hypothetical situations to work through
The following examples are constructed to illustrate reasoning. They do not describe real sites, rest on no test and contain no performance figures.
First case: a technical documentation publisher, behind Cloudflare on an eligible plan, publishes three versions of its product in two languages, for developers who use coding agents. A per-subpath llms.txt index is probably the most useful layer, because it lets an agent start from the right version and the right language. On-the-fly conversion may be enough for narrative pages if the template marks up the main content well; for the API reference, large and precise, files generated from the documentation source will be safer. The obsolete-version warning should move from the navigation into the body of the pages concerned.
Second case: a news publisher with part of its articles reserved for subscribers. Here the rights question comes before the format question. Before enabling anything, the publisher needs to know whether its HTML contains the full text of restricted articles, which usage signal it wants to send and which clients it accepts. According to the documentation, a configuration rule can limit the feature to certain open-access paths, which is more cautious than enabling it across the whole zone. An llms.txt file, if the publisher has one, should point to what it wants read, not to its entire archive.
Third case: a small services business with about twenty pages on a plan that is not eligible. The feature is unavailable, and that is no disaster. A short llms.txt file presenting the business, its terms and its reference pages, plus clean HTML with visible headings and dates, covers the essentials. Producing .md files for twenty pages is feasible, provided someone is responsible for keeping them current.
The three cases end up with different combinations. The useful question is not “should we turn on Markdown for Agents?” but: for which content, at which layer, under whose responsibility?
Editorial responsibility: who answers for the text the agent reads
One question remains that October 9 does not raise explicitly but makes hard to avoid. When an agent reads the Markdown representation of a page and then builds an answer for a user, which text is the publisher responsible for?
Legally, this article settles nothing; the rules vary by country and by contract, and none of the sources consulted addresses them. Editorially, the answer is simpler: the publisher answers for what it serves. Whether conversion happens on its server, on a delivery network or inside the agent, the starting content is the publisher’s. If a representation drops a caveat that lived in a banner, keeps a price corrected elsewhere or exposes a passage the page hid, the end reader will not blame the converter.
Three practical consequences follow. Parity: whatever matters for understanding a text has to be in the main content, not in the margins. Traceability: a publisher should be able to say, for a given page and date, which representation it was serving and by what mechanism; for on-the-fly conversion, that means at least recording the activation date and the vendor’s documented changes. Correction: a procedure that does not cover alternate representations is incomplete.
There is also a responsibility in how the work is described. It would be tempting to announce that conversion will get a site cited by assistants or improve its ranking. None of the sources consulted demonstrates that. Cloudflare claims a benefit in structure and tokens. The pilot presented with OpenAI on October 15 concerns page discovery, within a cohort of about 1% of the network according to Cloudflare, with no published per-publisher results on revenue or citation and no established link to Markdown for Agents. The llms.txt proposal describes use mostly at inference time, for documentation. None of that supports a promise about visibility, and an editorial lead gains credibility by saying so.
The most honest framing is readability: a clear index, faithful representations and well-built HTML make a site easier for automated clients to read. That is a verifiable goal, independent of the decisions of any search engine or assistant.
What October 9 says about what comes next
Back to the October 9 entry: a new engine, a higher limit, two counters removed. The current documentation also describes absent validators, without dating that change. None of that changes the nature of Markdown for Agents, which remains a representation computed from HTML, negotiated through a header and served by an intermediary. But the change does away with two convenient illusions.
The first was a ready-made metric. Token counters provided a number, not evidence. Their removal forces a return to measures one can explain: bytes, requests, status codes, content comparisons. They are more modest, and they have the advantage of claiming nothing about what models do.
The second was a representation that needs no upkeep. By removing ETag and Last-Modified, the documentation makes clear that the Markdown version is not the original document and cannot claim its metadata. It has to carry its own signals in its content: its date, its language, its version, its caveats. That work cannot be delegated to a converter.
For a site that wants to be readable by agents, the decision is built layer by layer. An index, when the site has a path to offer. Representations, when certain pages deserve to be read without the noise around them, produced by conversion or by publication depending on the control required. A corpus, only for those who have a use for one. And underneath all of it, HTML that says clearly what it contains. The October 9 change does not move that architecture; it makes its foundations more visible, along with the responsibility that comes with them.
Sources
- Cloudflare changelog, “More efficient Markdown for Agents conversion,” October 9, 2026, accessed October 21, 2026
- Cloudflare documentation, “Markdown for Agents,” updated October 9, 2026, accessed October 21, 2026
- The llms.txt proposal, version 2, accessed October 21, 2026
- RFC 9110, HTTP Semantics, June 2022
- RFC 9111, HTTP Caching, June 2022
- RFC 9309, Robots Exclusion Protocol, September 2022
- RFC 8288, Web Linking, October 2017
- OpenAI documentation, “OpenAI Crawlers,” accessed October 21, 2026
- Cloudflare press release, September 15, 2026, on search, training and agent controls
- Cloudflare blog post on early results of the discovery pilot with OpenAI, October 15, 2026, accessed October 21, 2026
- Content Signals
- llmtxt.info, dated data copy provided with this article, fixed-panel observation of October 19, 2026: /snapshots/2026-10-19/observatory-2026-10-19.json
- llmtxt.info, Firecrawl study of 2,000 domains, observed September 2, 2026
Read also : Cloudflare and OpenAI: being found is no longer enough · When AI uses an article: who pays, and on what proof?