`, real headings, and JSON-LD that matches what the page shows. Fetch a page the way a crawler without JavaScript does, and look for a few plain words from a paragraph you can see in the browser: ```bash curl -s "$SITE/about" | grep -c "a few plain words from the page" curl -s "$SITE/about" | grep -o "[^<]*" | sort | uniq -c | sort -rn | head ``` The first should name your sitemap. The second counts the addresses that share each date: if one date covers nearly all of them, it's probably the day you last built the site. If your sitemap is an index of other sitemaps, run it on each of those. For IndexNow, the key file at your site's root should print your key, and each ping your deploy sends should get `200` or `202` back, both of which mean the ping was received: ```bash curl -s "$SITE/your-key.txt" ``` ## Publish a Markdown copy of each page A Markdown copy of each page, named in the page's head, was the door crawlers found fastest on this site: seven verified crawlers fetched the copies, starting the day after they went live, and Amazonbot most of all. Here each copy sits at its page's address with `.md` added, is served as `text/markdown`, is kept out of search results with `noindex`, and is named in the page's head: ```html ``` ```bash curl -s "$SITE/about" | grep -o ']*text/markdown[^>]*>' curl -s -o /dev/null -D - "$SITE/about.md" | grep -i "^content-type" ``` The first should print the link, and the second `text/markdown`. You can also answer a request for Markdown at the page's own address, which is how coding agents ask: Claude Code sends `Accept: text/markdown, text/html, */*`. The server answers with Markdown and `Vary: Accept`, and sends a browser, which asks for `text/html`, the HTML it always had, with the same `Vary: Accept`, so a cache keeps the two apart. ```mermaid sequenceDiagram participant B as A browser participant S as Your site participant A as A coding agent B->>S: GET /about, Accept: text/html S-->>B: The HTML it always had, Vary: Accept A->>S: GET /about, Accept: text/markdown S-->>A: Markdown, Vary: Accept ``` ```bash curl -s -o /dev/null -D - -H "Accept: text/markdown" "$SITE/about" | grep -iE "^(content-type|vary)" curl -s -o /dev/null -D - "$SITE/about" | grep -iE "^(content-type|vary)" ``` The first should print `text/markdown` and `Vary: Accept`, the second `text/html` and the same `Vary: Accept`. If your site is on a Cloudflare Pro plan or above, [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) does the negotiation for you by converting your HTML at the edge. ## Put the whole post in your feeds A feed that carries each post in full is still read, mostly by feed readers, and most of their requests only ask whether anything changed: in a 30-day count, 401 of 551 answers on this site's `/rss.xml` were `304 Not Modified`. Google and Bing both accept a feed as a sitemap, which is one more reason to keep it. ```bash curl -s "$SITE/rss.xml" | grep -o "" | wc -l curl -s "$SITE/rss.xml" | grep -o "" | wc -l curl -s -o /dev/null -D - "$SITE/rss.xml" | grep -iE "^(etag|last-modified)" ``` In an RSS feed the first two numbers should match: one full text for each post. The third should print an `ETag` or a `Last-Modified` header, which is what lets a reader ask whether anything changed. Send the `ETag` back, and the answer should be `304`: ```bash curl -s -o /dev/null -w "%{http_code}\n" -H 'If-None-Match: "the-etag-you-got"' "$SITE/rss.xml" ``` ## Add an llms.txt for coding agents An `llms.txt` is cheap to add and coding agents read it, but don't expect search or AI crawlers to: in 17 days, no verified AI or search crawler fetched this site's. It's a Markdown file at a site's root that gives language models a short map of the site, a title, a one-line summary and lists of links, as [the proposal](https://llmstxt.org/) describes. [Gauge found Codex](https://www.withgauge.com/blog/does-codex-read-llms-txt/) opening one in about a third of its sessions, while Google Search says it doesn't use the file. ```bash curl -s "$SITE/llms.txt" | head -5 ``` It should open with a `# ` title and a `> ` summary line, with the lists of links under `## ` headings after them. Lighthouse's Agentic Browsing category, also in [PageSpeed Insights](https://pagespeed.web.dev/), checks the file's form when there is one and doesn't penalise a site without one. ## Run an MCP server and publish an AI Catalog An MCP server lets an agent search your site rather than read it page by page, but you'll have to hand out its address yourself, because no mainstream client finds a site's server on its own yet. MCP, the Model Context Protocol, is an open standard that connects AI applications to outside systems. An AI Catalog is a JSON file at `/.well-known/ai-catalog.json` that lists a site's AI artifacts, such as its MCP server's card, the route the [MCP Server Cards proposal](https://github.com/modelcontextprotocol/modelcontextprotocol/blob/main/seps/2127-mcp-server-cards.md) sets out. This site runs [mcp-my-site](https://pournasserian.com/open-source/mcp-my-site), the open-source server I built, and the [44 tools in my earlier roundup](https://pournasserian.com/writing/mcp-servers-for-websites-2026) are the ways to get one without writing it. ```bash curl -s "$SITE/.well-known/ai-catalog.json" | head -12 ``` It should print JSON with an entry of type `application/mcp-server-card+json` for your server. Then connect a client to the server's address and list its tools; the official [MCP Inspector](https://github.com/modelcontextprotocol/inspector) does it, and I compared it with the other testing tools in [an earlier article](https://pournasserian.com/writing/mcp-server-testing-tools-2026). Last, check that a person can find the address on your site, as the [agents page](https://pournasserian.com/agents) gives it here. ## Measure at the edge, and count only verified bots Count crawlers in your CDN's or your server's logs, and only the ones verified, because a crawler's name in a request proves nothing: 113 requests here said they came from PerplexityBot without Cloudflare verifying them. Analytics that run as JavaScript in the page can't see them, since most AI crawlers don't run scripts. Cloudflare verifies a crawler by a published IP list, reverse DNS or a signature on the request; count what passes, and count a request that only names a crawler apart. ```mermaid flowchart LR named[A request names a crawler] --> list[A published IP list] named --> dns[Reverse DNS] named --> sig[A signature on the request] list --> verified[Verified, count it] dns --> verified sig --> verified named -. "none of them" .-> apart[Only named, count it apart] ``` On Cloudflare, [AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/) is on every plan, and the [GraphQL Analytics API](https://developers.cloudflare.com/analytics/graphql-api/) behind it keeps 31 days on the Free plan. This query lists who fetched one path in a week. `requestSource: "eyeball"` leaves out Cloudflare's own requests, the host filter leaves out your zone's other hostnames, and an empty `verifiedBotCategory` means Cloudflare didn't verify the request, whatever its user agent says: ```graphql query { viewer { zones(filter: { zoneTag: "YOUR_ZONE_ID" }) { httpRequestsAdaptiveGroups( filter: { datetime_geq: "2026-10-11T00:00:00Z" datetime_lt: "2026-10-18T00:00:00Z" requestSource: "eyeball" clientRequestHTTPHost: "example.com" clientRequestPath: "/llms.txt" } limit: 20 orderBy: [count_DESC] ) { count dimensions { verifiedBotCategory userAgent } } } } } ``` Off Cloudflare, you can check a crawler by reverse DNS yourself. Look up the address a request came from: [Google's crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) resolve to a name under `googlebot.com`, `google.com` or `googleusercontent.com`, and that name resolves back to the same address. ```bash nslookup 66.249.66.1 nslookup crawl-66-249-66-1.googlebot.com ``` ## Method and caveats - The steps, their order and every figure come from [the longer article](https://pournasserian.com/writing/ai-friendly-website-2026), which read this site's Cloudflare logs from 30 September to 17 October 2026 and the standards and documentation as they stood on 18 October 2026. Its own method and caveats apply here. - The checks are new in this article. I ran each one against this site on 18 October 2026: the shell commands with curl in bash (they're written for a Unix shell such as bash or zsh), the GraphQL query against this site's Cloudflare zone, and the two lookups against the Googlebot address shown. I didn't send an IndexNow ping for this article; `200` and `202` are the answers IndexNow's documentation lists for a ping it received. - A check that passes says a door is open, not that anyone uses it, and a count of requests says nothing about what a crawler did with a page. - This is one small personal site. A larger or busier site will see more crawlers and different proportions, and several of the standards here were still changing when I wrote this. I'm [Amir Pournasserian](https://pournasserian.com/about). I build AI and platform systems for a living, maintain [FluentCMS](https://pournasserian.com/open-source/fluentcms), [YeSvelte](https://pournasserian.com/open-source/yesvelte) and [mcp-my-site](https://pournasserian.com/open-source/mcp-my-site), and write here about what I find along the way.