mcp-my-site — Make Your Website Answer AI Agents

mcp-my-site is an open-source tool that lets a website answer AI agents. Point it at your site and ChatGPT, Claude and any other client of the Model Context Protocol (MCP) can search and read it. A command-line indexer crawls the site and builds a hybrid keyword and vector index. A small ASP.NET Core app serves that index as an MCP server with search and fetch tools, and as a JSON search API for the site’s own search box. One site per server, built for sites under 1,000 pages.

GitHub · ai.pournasserian.com (the live instance, with a playground) · NuGet · Container image

At a glance

My role: creator and maintainer
Started: October 2026, the day I published the idea it comes from, Sites that speak for themselves
Runs on: .NET 10 and ASP.NET Core, with the official MCP C# SDK; written in C#
Ships as: the mcpmysite .NET tool on NuGet and a container image on GitHub Container Registry, for amd64 and arm64
Search: keyword and vector search fused per query, or keyword-only with no API key at all
Listed in: the official MCP registry, as io.github.pournasserian/mcp-my-site (as of 17 October 2026)
Live: ai.pournasserian.com indexes this site, all of its pages, and rebuilds the index on every deploy
Community: 98 commits and 41 merged pull requests in its first week, one contributor so far (as of 17 October 2026)
License: MIT
Status: pre-1.0; latest release v0.2.1 (17 October 2026)

as of October 2026

Why I built it

For twenty years a small website had one kind of reader to plan for: a person who arrived from a search result and read the page. People now ask an assistant, and the assistant visits the site on their behalf. When it does, it guesses. It downloads a page or two, strips away the menus and footers as best it can, and hopes it picked the right page. The owner has no say in what gets read and never learns what was asked.

The big platforms have started to close that gap for their own customers: GitBook, Shopify and WordPress ship MCP endpoints. Hosted services will crawl your site and put an endpoint in front of it, for a monthly bill and a vendor. The open-source crawl-to-MCP tools are built for the reader, not the owner: a developer points one at someone else’s documentation and runs it on their own laptop. What was missing was the general version: any small site, on any host, with search that understands meaning as well as keywords, run by the owner.

I wrote that up as an idea on 10 October 2026 and started the repository the same day. mcp-my-site is the first two doors of that idea, the agent endpoint and the search API, with chat left for later.

What it does

Two parts share one index. The indexer, mcpmysite, reads the site from its sitemap or from a folder of built HTML, extracts the main content of each page as Markdown, splits it into heading-aware chunks, embeds the chunks, and writes the index. Every run is a full rebuild. The server loads that index, keeps it fresh by polling, and answers two kinds of callers with the same search code: MCP clients through the search and fetch tools, and the site’s search box through GET /api/search.

The index is the contract: the indexer and the server only agree on the files in the index folder, so either side can be replaced on its own.

Hybrid search. BM25 keyword search and vector search over embeddings, fused per query, with a keyword-only fallback when the embedding provider is unavailable.
Works without a key. A keyword-only index needs no embeddings provider at all; switch to hybrid search later with one setting and a rebuild.
ChatGPT and Claude ready. A Streamable HTTP MCP endpoint with search and fetch tools in the shape ChatGPT connectors use, and server instructions generated from the site itself. It works with Claude Code, Claude Desktop, VS Code, Cursor and the MCP Inspector.
A search API for your own site. GET /api/search with CORS, rate limiting, an OpenAPI 3.1 document and an interactive reference, plus a drop-in snippet for a search box.
A playground. /playground is a small MCP client in the page, so anyone can try the server without installing anything.
Hot swap. The server polls the index source and swaps a new build in without a restart.
Telemetry. OpenTelemetry traces, metrics and logs to Application Insights or any OTLP collector. Query text is not logged unless you turn it on.
Any embeddings provider. OpenRouter by default; any OpenAI-compatible endpoint works.

What it is not: one site per index and per server, with no multi-tenancy. HTML only, so PDFs and Word documents are not indexed, and a JavaScript app has to be prerendered to a folder of HTML first. Search runs on submit, with no search-as-you-type. And there is no chat endpoint in this version.

Getting started

You need the .NET 10 SDK and nothing else: the first index is keyword-only, so no API key is involved.

dotnet tool install --global mcpmysite
mcpmysite init https://example.com --keyword-only   # writes appsettings.json
mcpmysite build                                     # crawls the site into ./index
mcpmysite search "how can I contact you"            # checks the index
docker run --rm -p 5000:8080 -v "$PWD/index:/index:ro" -e Index__Path=/index ghcr.io/pournasserian/mcp-my-site:latest

The server is then at http://localhost:5000: /mcp for agents, /playground to try it in the browser, /api/search?q=contact for JSON and /healthz for the loaded index. Connecting Claude Code takes one line:

claude mcp add --transport http mysite http://localhost:5000/mcp

Hybrid search needs an API key for an OpenAI-compatible embeddings endpoint, read from the AI_API_KEY environment variable and never written to a file. A small site embeds for a fraction of a cent.

Where it sits

The idea page sorted what exists into four groups, each leaving a small site owner without something. mcp-my-site is the row I wanted to add.

Option	Built for	What a small site owner is left without
Platform built-ins	Customers of that platform	Anything at all, if the site is hosted elsewhere
Hosted services	Teams with a budget	Control over where the index lives and what it costs
Open-source crawlers	A developer reading other people’s sites	A public endpoint the owner runs for their own site
Proposed standards	The whole web, eventually	Something that works this year
mcp-my-site	The owner of a small site, on any host	Chat, for now

Under the hood: for developers

Picking the content. For each page the indexer chooses one element: <main>, or the single <article> inside it when that article holds at least half of the text; a lone <article> when there is no <main>; otherwise a density heuristic that scores blocks by text length, penalises text that sits inside links, and takes the best one. Navigation, asides, forms, scripts, hidden elements and footers that look like page chrome are removed. There are no per-site CSS selectors to maintain; the run report says which strategy each page used, so you can see whether your markup is being understood.

Chunks that carry their headings. Markdown is split by headings into sections, each remembering its heading path. Sections longer than the limit are split at paragraph boundaries, then at sentence ends; fenced code blocks are never split. What gets embedded and keyword-indexed is Page title > Heading > Subheading followed by the chunk text, so a chunk about “Enterprise” under “Pricing” matches a question about enterprise pricing even if its body never says “pricing”.

A broken build never replaces a working index. Before writing anything the indexer checks that enough pages were indexed, that the page count did not fall too far from the previous index, and that not too many fetches failed. Any failure leaves the previous index untouched and exits with code 1. Embedding happens after those checks, so a broken crawl never spends money. Build time fails closed; query time fails open: a missing embedding degrades to keyword search rather than an error, and a bad new index leaves the old one serving.

Two stores, one result. The index is a folder of JSON files with a flat vector file, or one SQLite database with FTS5 for keyword search. Both hold the same data, merge the keyword and vector rankings with reciprocal rank fusion, collapse chunks to one hit per page, and build a snippet around the first matched term. Vector search is brute force: a site this size has thousands of chunks, not millions, and it needs no extra service. The manifest is written last, so a reader never sees a half-written index.

A stateless server. The MCP endpoint speaks Streamable HTTP in stateless mode: no sessions and no server-to-client requests, so any number of instances can serve the same index behind any load balancer. Its two tools are read-only and shaped the way ChatGPT connectors expect: every search result has an id, and fetch takes that id and returns the page as Markdown with its title, description, language and last-modified date. The indexer writes the server’s instructions into the manifest from the crawled pages: what the site is, how many pages the snapshot holds and when it was taken, when to use the tools, and an outline of the site’s sections, kept under 2,000 characters.

Protection without a gateway. Every MCP and search request counts against a fixed one-minute window per client IP, 60 by default. Behind Cloudflare the visitor’s address is read from CF-Connecting-IP, but only when the connection comes from Cloudflare’s own ranges, because anyone can send that header. The search API answers browsers only from the origins you list, and the MCP endpoint can be closed with a static API key. Snippets are HTML-escaped with only <mark> added, full-text queries are built from quoted tokens, and nothing in the index, the configuration or the logs contains a key.

Shipping a new index without touching the server. The server reads the manifest from a folder or a URL on its poll interval. When the build time or the page count differs from the loaded index, it opens the new files, validates them, and swaps the store in; requests in flight finish on the old one. Build in CI, publish the folder, and the server picks it up.

A few of the decisions, and what they were weighed against:

Decision	Alternatives considered	Why this one
Stateless MCP transport, tools only	Stateful sessions with resources and prompts	Horizontal scaling without affinity; the two tools cover the use case; matches the 2026 protocol direction
Reciprocal rank fusion	Weighted score blending	No calibration between BM25 and cosine scales; robust defaults; easy to reason about
Brute-force vector search	HNSW or an external vector database	Sites this size have thousands of chunks, not millions; zero extra infrastructure
Automatic content selection, no selectors	Per-site CSS selector overrides	Nothing to maintain or break on redesigns; the report shows what was picked
Full rebuild with an embedding cache	Incremental crawl with change detection	Deleted pages vanish for free; the cache removes most of the cost of a full rebuild
Markdown as the stored format	HTML, plain text	Readable by models and humans, keeps structure, small
Fail closed on build, fail open on query	Fail open everywhere, or fail closed everywhere	A broken index is worse than a stale one; a degraded answer is better than no answer

Running it for this site

ai.pournasserian.com is the first instance, and this site is its first user. Every time the site deploys, the indexer crawls the live pages, embeds only the chunks that changed, and publishes the index; the server polls the manifest and swaps the new index in within minutes, without a restart. The site advertises the server to agents in an AI Catalog at /.well-known/ai-catalog.json, and the server is listed in the official MCP registry.

To ask this site something through Claude Code:

claude mcp add --transport http pournasserian https://ai.pournasserian.com/mcp

Or open the playground and try the search tool in the browser.

My role

I created mcp-my-site in October 2026 and am its only contributor so far: the index format, the indexer, the server, the documentation, and the release pipeline that publishes the tool to NuGet and the image to GitHub Container Registry on every version tag.

Where it’s headed

mcp-my-site is pre-1.0 and a week old. The third door of the idea, chat, is planned for the next version: an OpenAI-compatible chat API, with Chat Completions and Responses and streaming, built on Microsoft Agent Framework and using the same search as its retrieval tool. The index and search layers need no change for it.

The code has three seams for what comes after: a new index store, such as Postgres or a hosted search service, behind the store interfaces; a new content source behind the source interface; and another embedding protocol behind the generator factory. The idea also wanted the owner to see what agents ask; the server records telemetry today, with query text off by default, and an owner’s view of it is not built.

Stack

.NET 10 · ASP.NET Core · ModelContextProtocol C# SDK · AngleSharp · ReverseMarkdown · System.CommandLine · SQLite (FTS5) · OpenTelemetry · Scalar · Docker · GitHub Actions · NuGet · GitHub Container Registry