Making a website AI friendly: what the bots actually fetched from mine

October 18, 2026

I kept reading lists of files to add to make a website “AI friendly”, and I wanted to know whether anything actually reads them. This site has most of them: a robots.txt that welcomes AI crawlers, an llms.txt, a Markdown copy of every page, full-text feeds, structured data, an AI Catalog and an MCP server of its own. So on 18 October 2026 I looked at what actually came through each door. I read 18 days of the site’s Cloudflare logs (30 September to 17 October 2026), three and a half days of its MCP server’s telemetry, and the standards and vendor documentation behind each door as they stood that day. Some of it surprised me.

In brief

1. Most AI crawler traffic went to the oldest doors: pages, robots.txt and the sitemap. ClaudeBot alone fetched the sitemap 122 times in 18 days.
2. No verified AI or search crawler fetched llms.txt in 17 days. The one verified AI fetch came from Claude-User, which fetches when a person asks. Google Search says it doesn’t use the file, and the larger studies find that its readers are coding agents.
3. The Markdown copies were found the day after they went live. Seven verified crawlers made 143 of the 152 requests for them that weren’t mine, most likely through the link in each page’s head.
4. Feeds are read by feed readers. Most of their requests only asked whether anything had changed.
5. An MCP server is reached by MCP clients and directories, not by crawlers. In three and a half days it saw about 300 connections and 29 tool calls, and clients asked for a dozen discovery addresses it doesn’t have. No mainstream client finds a site’s server on its own yet.
6. A crawler’s name in a request proves nothing. 113 requests said they came from PerplexityBot without Cloudflare verifying them. Count verified bots only.
7. You can measure most of this for free, with limits. Cloudflare’s Free plan keeps 31 days of counts per crawler and per path, but not the Accept header and not referrers.

The doors, and who knocks

Machines come to a website for different reasons, and each reason favours different doors. The big operators document three kinds of visitor, and keep them apart: a training crawler (OpenAI’s GPTBot, Anthropic’s ClaudeBot, Amazonbot), a search crawler that builds the index AI answers cite (OAI-SearchBot, Applebot, Googlebot), and a fetcher that gets one page because a person asked (ChatGPT-User, Claude-User, Perplexity-User). Two more show up in the logs and the studies: coding agents such as Claude Code and Codex, and plain feed readers. An agent that uses the Model Context Protocol (MCP) comes in by yet another door.

This is how they lined up on this site, by what each kind of visitor actually fetched. The training and search crawlers used the same doors, so they share a box.

I followed fourteen verified crawlers and fetchers by name: eleven that Cloudflare files as AI crawlers, AI search or AI assistants, and three search crawlers, OpenAI’s and Exa’s, that it files under search engines. Counting how many of the 14 used each door at least once gives a rough map of what matters. Keep in mind that the doors weren’t open equally long: pages, robots.txt and the sitemap for all 18 days, llms.txt for 17, the feeds for 13 to 17, the Markdown copies and llms-full.txt for 8, and the AI Catalog for 1.

Plain HTML still does most of the work

The page itself is the door every crawler uses, so it’s the one to get right first. All 14 crawlers fetched pages, and nothing else on the site came close in volume: 15,431 page requests in 18 days, against 432 for the Markdown copies.

What helps a machine read a page is mostly what has always helped. The extractors are documented in their code: Mozilla’s Readability, the library behind Firefox’s reader view, reads the page’s JSON-LD first, then its Open Graph and article: meta tags, and drops <footer>, <aside> and hidden elements. trafilatura, the extractor Hugging Face used for its FineWeb training set, reads the same meta tags and JSON-LD. When I ran one of this site’s articles through Jina Reader, a service that turns a page into text for language models, its title and date came from the page’s JSON-LD, and the footer with the author line was gone.

The one thing a modern site can easily get wrong is rendering. In Vercel’s measurement of December 2024, OpenAI’s, Anthropic’s and Perplexity’s crawlers didn’t run JavaScript, while Googlebot and Applebot did, and Anthropic’s web fetch tool says it doesn’t support pages rendered with JavaScript. A page whose text is built in the browser can look fine in Google and be empty to most AI crawlers. This site is static HTML with no client-side rendering, one <main>, real headings and JSON-LD built from the same content files as the page, which is the safe ground.

What you don’t need, according to Google, is anything special. Its guide to optimizing for generative AI features in Search says AI Overviews and AI Mode need no special markup, no special schema.org, and no llms.txt or Markdown files, and it lists overfocusing on structured data among the myths. Bing’s guidelines, which cover Copilot too, say structured data may support clearer grounding but guarantees nothing, and ask for markup that matches the visible page. The studies of schema and AI citations I found are vendors’ own and mostly found no lift.

robots.txt: say yes or no by purpose

A robots.txt file is where you tell crawlers what they may fetch, and since the operators split their bots by purpose, you can answer each purpose separately. The format is RFC 9309 (September 2022), and it’s worth remembering what the RFC itself says: the rules are not a form of access control.

Two details change how much it can do for you.

• User-triggered fetchers may not follow it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page. Anthropic’s page is the exception: it says all three of its bots honour robots.txt, Claude-User included.
• A name in the user agent can be anyone. Cloudflare verifies a bot by a published IP list, reverse DNS or, more and more, a signature on the request (Web Bot Auth, built on HTTP Message Signatures, RFC 9421, which OpenAI’s agent and some of Google’s agent requests already use). On this site, leaving images and scripts aside, 113 requests named PerplexityBot, 112 Kimi-SearchBot, 85 Bingbot and 75 Googlebot without Cloudflare verifying them. Some are spoofed, some come from operators that aren’t on the verified list, and the logs can’t tell which.

There’s also a newer layer that says what may be done with content after it’s fetched. Cloudflare’s Content Signals Policy (September 2025) adds a Content-Signal line to robots.txt with three answers, search, ai-input (using the content to ground an AI answer) and ai-train. The IETF’s AI Preferences working group is drafting something similar, a Content-Usage rule and HTTP header with search, ai-use and train-ai; both drafts missed their August 2026 milestone. Neither has an operator on record yet: none of the crawler documentation I read mentions either one, and Cloudflare’s scan found Content-Signal on 2.4% of about 109,000 top domains.

My own file is four lines: every crawler is allowed, and the Content-Signal line says yes to search, AI answers and AI training. I want the site read and cited, so the choice was easy. One practical note if you’re on Cloudflare: its managed robots.txt would add ai-train=no, so I keep it off.

Sitemap and IndexNow: say what changed

A sitemap with honest dates is the cheapest way to tell crawlers what changed, and the logs show it gets read: ClaudeBot fetched this site’s sitemap 122 times in 18 days, Googlebot 19 and GPTBot 10. Google uses lastmod only when it’s consistently accurate and ignores changefreq and priority; Bing says the same and calls sitemaps a foundational signal for Copilot too. Here, each page’s lastmod is the date its content last changed, never the build date.

IndexNow is a protocol Bing and Yandex launched in October 2021: a site pings one search engine when a URL is added, changed or deleted, and that engine shares the ping with all the others. The registry lists Bing, Yandex, Seznam, Naver, Yep, the Internet Archive and Amazonbot. Google isn’t part of it. Bing’s guidelines tie it directly to Copilot: timely IndexNow pings reduce outdated URLs in Copilot answers. No OpenAI document says IndexNow reaches ChatGPT search, so I’d treat that as unproven. This site sends the addresses each deploy changed, straight after the deploy.

llms.txt is cheap, and coding agents are its readers

llms.txt is a Markdown file at a site’s root that gives language models a short, curated map of the site: a title, a one-line summary and lists of links. Answer.AI proposed it in September 2024, and version 2 (August 2026) adds a way to find it from any page, a rel="describedby" link. llms-full.txt, one file with the whole site in it, is a convention from documentation platforms, not part of the proposal. This site has had an llms.txt since 1 October and an llms-full.txt since 10 October, both generated from the same content files as the pages.

In 17 days, no verified AI or search crawler fetched it. Of 75 requests for /llms.txt, 44 were my own tools, 24 came from browsers and clients that didn’t identify themselves, 5 from Chrome’s Lighthouse, 1 from an agent directory’s bot, and 1 from Claude-User. Not one came from GPTBot, ClaudeBot, OAI-SearchBot, Applebot, Amazonbot, Googlebot or Bingbot.

That matches what I found elsewhere:

• Google Search doesn’t use it, by its own guide, which adds that keeping one for other services neither helps nor harms a site in Search. Google’s Search Advocate wrote in June 2025 that no AI system used llms.txt.
• Ahrefs looked at 137,210 sites in May 2026: 28% had a valid file, 97% of those files got no request that month, and AI retrieval bots made 1.1% of the requests the rest got. GPTBot was the most frequent named bot, and Claude Code came second.
• Coding agents do read it. Gauge found Codex opening llms.txt in about a third of its sessions, through curl, which doesn’t say who’s asking. A count by crawler name, like mine, misses those reads.
• Chrome’s Lighthouse checks it. Its new Agentic Browsing category audits the file’s form when there is one, and doesn’t penalise a site without one.

So I’d keep an llms.txt, because it’s cheap and coding agents use it, but I wouldn’t expect it to change how search crawlers or AI answers see a site.

Markdown copies were found within a day

A Markdown copy of each page is the door that surprised me most. Since 10 October, every page here has a copy at its own address with .md added (/about.md for /about), served as text/markdown, written from the same content files as the page rather than converted from its HTML, and kept out of search results with noindex. Each page names its copy in its head:

<link rel="alternate" type="text/markdown" href="/about.md">

Verified crawlers started fetching the copies on 11 October, the day after they went live. Of 432 requests, 280 were my own tools and checks; of the other 152, seven verified crawlers made 143.

None of these crawlers had fetched llms.txt, so they didn’t find the copies there. The link in the head is the likely route, though the logs can’t show how a crawler found an address. A month of logs published on dri.es, another personal site, showed the same pattern: crawlers reached its Markdown only through the .md addresses and the head link, with Amazonbot, OAI-SearchBot and GPTBot fetching the most, in that order. Googlebot and Bingbot fetched no copy here.

The other way to serve Markdown is content negotiation. A client sends Accept: text/markdown to the page’s own address, and the server answers with Markdown and Vary: Accept (the text/markdown type is RFC 7763; the negotiation is ordinary HTTP, RFC 9110). Developer tools are the ones asking: Claude Code sends text/markdown, text/html, */*, and OpenCode and VS Code’s chat ask for Markdown first. No crawler operator documents sending it, and Cloudflare’s scan found 3.9% of about 200,000 top domains answering it. Since 18 October this site answers it with the same copies, through a small Worker in front of the pages, so the copy’s address comes back in a Content-Location header. In its first 49 minutes, one verified crawler used it: ExaSearchBot asked for an article as Markdown and got the copy. That’s far too early to say more.

If your site is on Cloudflare, Markdown for Agents does the negotiation for you by converting your HTML at the edge. It’s a switch on Pro plans and above (this site is on the Free plan, where the switch shows “Pro plan”), and it removes headers, footers and navigation as it converts. I chose my own copies because they hold the content files’ words exactly, diagrams as their source included, but conversion is a reasonable shortcut when you don’t have Markdown sources.

Feeds with the whole post

A feed with the whole post in it is an old door that still gets plenty of use, mostly by feed readers. The two feeds here carry each post in full, in content:encoded. In 18 days they got 691 requests: 572 from clients Cloudflare doesn’t verify, most of them feed readers (one ingesting service alone made 140), 29 from my own tools, a few dozen from Flipboard, small search engines and other named bots, and 1 to 4 each from ten verified AI crawlers, ClaudeBot, GPTBot, Amazonbot, Applebot and OAI-SearchBot among them. Most of the traffic is a reader asking whether anything changed: in a 30-day count, 401 of 551 answers on /rss.xml were 304 Not Modified.

None of the AI operators’ crawler pages mention feeds, so I can’t say any of them reads feeds by design. Google and Bing both accept a feed as a sitemap, which is reason enough to keep one.

An MCP server and the AI Catalog

An MCP server is the one door that lets an agent ask a site a question rather than read its pages one by one. MCP is an open standard that connects AI applications to outside systems; this site runs mcp-my-site, the open-source server I built, at ai.pournasserian.com, with a search tool that ranks the whole site and a fetch tool that returns one page as Markdown, from an index rebuilt each time the site deploys. The agents page explains how to connect to it.

Finding a site’s server is the part that isn’t settled yet. The intended route, from the MCP Server Cards proposal (final since 6 October 2026), goes like this:

An AI Catalog is a JSON file at /.well-known/ai-catalog.json that lists a site’s AI artifacts, such as MCP server cards. The card gives the server’s name, version and address, deliberately without its tools, and the live server/discover and tools/list calls give the rest (I wrote about server/discover in the MCP series). This site has carried its catalog since 17 October. On that first day, one crawler fetched it: OAI-SearchBot.

The server’s own telemetry shows who does come. In three and a half days (14 to 18 October), clients opened about 300 connections (308 initialize calls, 281 tool lists) and made 29 tool calls, 15 searches and 14 fetches. Many of the clients named themselves as MCP or agent directories and rankers. They also asked for a dozen discovery addresses the server doesn’t have: /.well-known/agent-card.json (the address A2A, another agent protocol, uses), /.well-known/mcp.json and /.well-known/mcp/server-card.json (from earlier Server Card drafts), /.well-known/ard.json (a rival proposal), and /mcp/server-card, where the final proposal puts a server’s card and where mine doesn’t serve one yet. Everyone is guessing at once.

No mainstream client follows the catalog by itself yet. Claude Code, Claude.ai, VS Code and Cursor all take a server’s URL from you, or from their own gallery; the only client code for Server Cards I found was in open pull requests. So if you run a server, expect to hand out its address yourself, as the agents page does. The 44 tools in my earlier roundup are the ways to get one without writing it.

Measuring it yourself

You can see most of this without paying for anything, as long as you count at the edge and count verified bots only.

• Cloudflare. AI Crawl Control is on every plan. On the Free plan it identifies crawlers by user agent; the GraphQL Analytics API behind it keeps 31 days of requests by user agent, verified bot category, path and status, which is everything in this article. It doesn’t show the Accept header, so you can’t see who negotiates Markdown, and referrers are a paid feature, so you can’t see visits that AI answers send you.
• JavaScript analytics can’t see crawlers. Cloudflare Web Analytics, like any tag in the page, counts browsers that run its script, and most AI crawlers don’t run scripts.
• Google Search Console. Clicks from AI Overviews and AI Mode are counted with other web clicks and can’t be separated. Its Generative AI report, on every site since 31 August 2026, shows impressions only.
• Bing Webmaster Tools. Its AI Performance report (in preview since February 2026) shows how often Copilot and Bing’s AI summaries cite your pages, and for which grounding queries, but not clicks.
• Lighthouse. The Agentic Browsing category in Lighthouse 13.5 (also in PageSpeed Insights) checks the accessibility tree, layout shift, llms.txt, the AI Catalog and WebMCP. This site scored 3 of 4: its catalog entry lacks a displayName that one of two competing specs requires.
• Your own server’s telemetry. If you run an MCP server, count calls by method and channel. Mine doesn’t log what people search for, because queries can carry personal data.

A checklist

If I were starting a site today, this is the order I’d do it in, by what the evidence says gets read:

1. Serve the text in the HTML. No client-side rendering of content, one <main>, real headings, and JSON-LD that matches the page.
2. Write a robots.txt that answers each purpose. Add a Content-Signal line if you have a view, knowing nobody has committed to reading it yet.
3. Keep a sitemap with honest lastmod dates, and ping IndexNow when you deploy.
4. Publish a Markdown copy of each page and name it in the head with rel="alternate". Answer Accept: text/markdown with Vary: Accept if you can.
5. Put the whole post in your feeds.
6. Add an llms.txt for coding agents. It’s cheap; just don’t expect search crawlers to read it.
7. If you want agents to search you, run an MCP server and publish an AI Catalog, and give people the address yourself.
8. Measure at the edge, and count only the bots your CDN verified.

Method and caveats

• The figures come from Cloudflare’s GraphQL Analytics API for pournasserian.com, 30 September 2026 00:00 UTC to 18 October 2026 00:00 UTC, read on 18 October 2026, and from the AI Crawl Control dashboard the same day. The dataset is sampled; the bot counts are close to exact and the totals are Cloudflare’s estimates. The MCP server’s figures are Application Insights counts from 14 to 18 October 2026, with no query text.
• “Verified” means Cloudflare put the request in a verified bot category. The fourteen crawlers I followed by name leave out a few verified AI assistants that send a generic browser user agent. Requests that only named a crawler are counted apart, and so are my own tools by their user agents (curl, the site’s indexer, audit scripts, Claude Code and the Claude app), along with one evening’s check that fetched 165 Markdown copies. Someone else using curl would have been counted as me.
• The doors were open for different lengths of time, and Markdown negotiation, the site’s search box and the agents page went live on 18 October, outside the period; the only figure for them is the first 49 minutes of Markdown negotiation, 15:31 to 16:20 UTC that day. The figures are counts of requests, not reads: a fetch says nothing about what a crawler did with the page.
• This is one small personal site. A larger or busier site will see more crawlers and different proportions. The studies I cite are mostly by vendors measuring their own platforms, and I’ve named them as such.
• The standards, the operators’ documentation and Cloudflare’s features are as I read them on 18 October 2026. Several are moving: llms.txt v2, the IETF’s AI Preferences drafts, Web Bot Auth, MCP Server Cards and Lighthouse’s Agentic Browsing audits were all changed within the last few months.