Validating a vector database built for RAG: four layers of checks, with the code to run them

November 5, 2026 · 30 min read

When a RAG answer is wrong, the model usually gets the blame, and the index that fed it gets a pass. I’ve learnt to look at the index first. Retrieval-augmented generation (RAG) puts a retrieval step in front of a language model: the question is matched against a store of document chunks and the best matches go into the prompt. That store is usually a vector database, an index of the chunks’ embeddings in which a query embedded with the same model lands near the chunks that mean the same thing. On 23 October 2026 I read what the papers, the vendor documentation and 25 evaluation tools say about checking such an index, and wrote a small set of scripts that does it in four layers, from the documents to the answer. I ran them the same day against the 31 pages of the MCP specification, embedded with OpenAI’s text-embedding-3-small, and put the code on GitHub as rag-index-validation, under an MIT licence. A few of the results surprised me.

In brief

1. The index fails before the model does, and the model rarely says so. Three of the seven failure points of a RAG system catalogued in 2024 are decided before generation starts, and in one benchmark the model that was best at declining to answer from a useless context still answered 55% of the time.
2. Cheap, deterministic checks catch the common faults. A document count against the source, the chunk token histogram, a near-duplicate pass, a norm and dimension check, a fresh embedding of a sample, and recall against a brute-force search. On my sample corpus they flagged the generated schema reference at once: 77% of its bytes inside markup, 162 of the 536 chunks, and all 11 near-duplicate pairs.
3. Label pages, not chunks, and measure recall at the k your prompt carries. With 50 questions, the index put the right page in the top 5 for 49 of them. The misses were paraphrases in plain words.
4. A page-level hit can hide a chunk-level miss. For one question the chunk that held the answer sat at rank 6, under a chunk of the same page at rank 2, and the generator answered “I don’t know”.
5. Judged context relevance is the least trustworthy number on the sheet; faithfulness is the judged metric worth paying for. Three independent studies found LLM-judged context relevance far behind a small fine-tuned classifier, while faithfulness had the best human agreement. Whatever judge you use, calibrate it against hand labels and report chance-corrected agreement.
6. Approximate search is a tuning problem you can measure in an afternoon. Run the golden queries through the index and through an exact search over the same vectors, and divide the overlap. On my 536 vectors the HNSW index lost nothing even at the lowest search setting; the vendors call 0.95 typical for a tuned index at scale.
7. Of 25 tools, none checks the index itself. They score answers and judge retrieved context. No tool compares approximate with exact search, counts chunks against the source, or checks that the query path uses the index’s embedding model.

Why the index gets its own tests

A generation score can’t tell you which layer failed, and three of the seven failure points are decided before the model runs. Seven Failure Points When Engineering a Retrieval Augmented Generation System (2024) reports on three RAG systems built by one group and names the points where they broke. The first three are the index’s: missing content, where the question can’t be answered from the documents and the system answers anyway; missed the top ranked documents, where the answer is in the corpus but ranks below the k chunks passed on; and not in context, where the relevant documents are retrieved and then dropped as the prompt is assembled. The paper’s lessons table ties the first two to chunk size, embedding and retrieval strategy, and says that adding the file name and chunk number to the context improved both retrieval and extraction. Its authors say offline evaluation is promising but needs labelled question-answer pairs, which is exactly what a retrieval test set is.

Where a passage lands in the prompt matters as much as whether it lands. Lost in the Middle (2023) gave GPT-3.5-Turbo 20 documents, one of them holding the answer: accuracy was 75.8% with that document first and 53.8% with it at position 10, below the 56.1% the model scored with no documents at all. The variant with a 16K context did no better. The Power of Noise (2024) measured the other side: with the gold passage alone Llama2 scored 0.5642, and with 18 passages that looked relevant but held no answer, the gold one in the middle, it fell to 0.1795. Random, unrelated passages hurt far less (0.3466). The authors recommend retrieving a few passages, three to five, because every extra one is a chance to add a distractor.

And the model will not tell you the index is bad. In the RGB benchmark (2023), ChatGPT’s accuracy fell from 96.33% to 76.00% as the share of irrelevant retrieved documents rose from none to 0.8, and when no document held the answer, the best model declined to answer only 45% of the time. One more caution comes from Evaluating Retrieval Quality in Retrieval-Augmented Generation (2024): labels that say a passage is about the topic correlate only weakly with what the generator does with it (a Kendall’s tau of at most 0.22 on its datasets, against 0.49 to 0.63 for scoring each passage by whether the model can answer from it alone). So the relevance label you want is “can the answer be written from this chunk”, not “is this chunk on topic”.

That’s why I check in layers, cheapest first, and only reach for a judge at the end.

Layer 1: the documents and their chunks

Most index faults show in a plain table of chunks before anything is embedded, and finding them costs nothing. The checks below are the ones I run; each has a source that describes the failure it catches.

On my corpus the first layer earned its keep immediately. The 31 pages became 536 chunks under a cap of 400 tokens, split by heading first and then by paragraph, and one page stood out everywhere: the generated schema reference, 726 KB of which 77% is inside HTML tags, which became 162 of the 536 chunks (the median page has 9), carried 49 of the chunks that had to be cut by force at the cap, and held all 11 near-duplicate pairs (the list results of tools/list, prompts/list and their siblings, which differ by one word). Five chunks were under 20 tokens, a heading with one line under it. And 31% of the chunks ended without punctuation, most of them list items, which is a thing to read rather than a failure.

How big a chunk should be is the usual question, and the evidence disagrees because the studies measure different things. Chroma’s technical report of 2024 scored retrieval at the token level over 472 generated queries and five corpora with text-embedding-3-large, five chunks per query. Its own table is below; the recall gap between strategies reached nine points, big overlapping chunks were the least efficient, and recursive splitting at about 200 tokens with no overlap was a strong cheap baseline.

Two later studies measured something else and leaned the other way. NVIDIA’s post of June 2025 scored answer accuracy with a 70B generator across five document sets and found page-level chunking best on average (0.648 against 0.603 to 0.645 for token sizes from 128 to 2,048), with the middle sizes beating the extremes. Rethinking Chunk Size for Long-Document Retrieval (2025) found the best size depends on the question: 64 to 128 tokens for short factual answers, 512 to 1,024 where the answer needs its surroundings, with recall at rank 1 on one technical set swinging from 4.9% at 64 tokens to 61.9% at 1,024. Semantic chunking, which cuts where the embedding of adjacent sentences changes, won by 12 to 20 points in one study only on corpora stitched together from unrelated passages, and by under a point on real documents.

What the sources agree on is context on the chunk. Anthropic’s contextual retrieval post of 2024 has a model write a 50 to 100 token sentence situating each chunk in its document before embedding it, and reports the share of relevant chunks missing from the top 20 falling from 5.7% to 3.7%, to 2.9% with a lexical index added, and to 1.9% with a reranker, at a cost it puts at $1.02 per million document tokens. The heading path my chunker prepends is the free version of the same idea. My own verdict, with its caveats: split by structure first, cap at a few hundred tokens, keep the heading path on the chunk, and decide the size on your own questions rather than on anyone’s table, mine included.

Layer 2: the embeddings and the approximate index

The faults in this layer are silent, which is why they need a check rather than a glance: a store returns results happily when the query was embedded by a different model than the documents, when a vector was never normalised, or when the graph misses a neighbour. Microsoft’s documentation for Azure AI Search puts the rule in one line: a vector index is an embedding space made of vectors from one model, and the query text goes through the same model so its vector lands in that space. A dimension mismatch fails loudly (Azure’s indexer rejects the document). A model mismatch with equal dimensions does not: Voyage’s migration guide says vectors from different models are not comparable even when their dimensions and types agree, so a new model means re-embedding everything.

Some models also treat queries and documents differently. The E5 models require every input to start with query: or passage: and their card warns of a performance drop without them; Nomic’s require a search_query or search_document prefix; Voyage prepends its own prompts when you pass input_type and says vectors with and without it stay compatible. If a prefix is skipped on one side, the two sides drift apart with no error anywhere.

Normalisation and the distance metric are the other pair to settle once. OpenAI’s embeddings are normalised to length 1, so cosine equals the dot product and ranks the same as Euclidean distance, and a vector you shorten by hand must be renormalised. Qdrant normalises on upload when a collection uses cosine and compares by dot product underneath; pgvector suggests inner product for normalised vectors, which is wrong for unnormalised ones, where longer or denser chunks would score higher regardless of topic. Absolute scores don’t transfer between models either: E5’s card notes its cosine scores cluster between 0.7 and 1.0 and only the order matters, so a threshold tuned on one model means nothing on another.

The checks I run on the stored vectors:

• Dimension, finite components and norm on every vector, and on every embedding response before it’s stored, with the response indices required to cover each input exactly once.
• Duplicate vectors, identical to four decimals, which mean duplicate chunks or a chunker that emitted the same text twice.
• The embedding model recorded on the collection when it was built, compared with the model configured on the query path.
• A fresh embedding of a sample of chunks with the production client, compared with the stored vectors by cosine. On my index the lowest was 0.9991; anything well under 1 means the model, the dimensions or a prefix changed in between.
• Self-retrieval: the store queried with a stored vector must return that point first. Chroma’s own documentation shows the failure: with ef_search at 10 its example index did not return the query vector itself, and did at 100 with a bigger build, about 42% slower. My index returned 100 of 100.

Then the check I think gets skipped most often: how much the approximate index loses against an exact search over the same vectors. Every production vector store uses an approximate nearest-neighbour (ANN) index, most often HNSW, a layered graph that trades a fraction of the true neighbours for speed. The HNSW paper puts its one real construction parameter, M, at 5 to 48 and says to build with enough effort that construction recall is close to 1, 0.95 being enough for most cases; the hnswlib authors say a recall below 0.9 means there’s room to improve; Qdrant’s tutorial calls 0.95 or higher typical for a tuned index; Weaviate’s own benchmark reports recall@10 of 97.24% at 5,639 queries per second on a million 1,536-dimensional vectors. The method is the same everywhere: run each query twice, through the index and brute force over the same vectors, and divide the overlap of the two top-k sets by k.

# check_index.py, abridged
def exact_top_k(matrix, query, k):
    scores = matrix @ query  # unit vectors, so the dot product is the cosine
    return list(np.argsort(-scores)[:k])

truth = [set(ids[j] for j in exact_top_k(matrix, q, 10)) for q in queries]
for ef in (10, 32, 100):
    col.modify(configuration={"hnsw": {"ef_search": ef}})
    res = col.query(query_embeddings=queries.tolist(), n_results=10, include=[])
    recall = np.mean([len(set(res["ids"][i]) & truth[i]) / 10 for i in range(len(queries))])

Each store has a switch for the exact side, or a documented way around its absence:

On my small index the result was recall@10 of 0.998 for the golden questions at ef_search 10, 32 and 100 alike, 1.000 for a sample of stored vectors, at about half a millisecond per query. On 536 vectors the graph is nearly exact, and that is the point of running the check early: it costs minutes, and the number is there to watch when the index reaches the sizes, the filters and the compression where it stops being 0.998.

Three things move it. Filters, first: pgvector’s README does the arithmetic, a filter that matches 10% of rows applied after an HNSW scan with ef_search 40 returns about 4 rows, and its iterative scans exist to fix that; Qdrant’s documentation says its graph “starts to fall apart” when filters are too strict; Weaviate falls back to a flat search under 40,000 matching objects; Azure’s pre-filtering guarantees k results at about seven times the cost of post-filtering when under 2% of a million vectors match. So repeat the recall measurement with the production filters at the selectivities users hit. Quantization, second: Qdrant reports recall@100 of 0.9966 for text-embedding-3-large at 3,072 dimensions under binary quantization with three times oversampling and rescoring, and 53 to 55% for text-embedding-3-small at 512 dimensions with one bit and no rescoring; scalar quantization’s error is usually under 1%. Repeat the measurement with and without rescoring. A model swap, third: Voyage’s procedure is to evaluate the new model on a sample of your own data, write its vectors to a new field or collection while the old stays live, switch queries over, verify, and only then delete the old one. Drift-Adapter (2025) is the one exception I found to “re-embed everything”: a small adapter trained on paired old and new embeddings lets new queries search the old index and recovers 95 to 99% of a full re-embedding’s recall at over a hundred times less compute.

Layer 3: retrieval against a golden set

This is the layer that answers the question you actually have, whether the right chunks come back, and it needs no judge at all. It needs questions with labels, a handful of definitions, and a few decisions about k.

The metrics are old and well defined. Precision@k is the share of the top k that is relevant; recall@k the share of a query’s relevant items that made the top k; hit rate scores a query 1 if any relevant item is in the top k; the mean reciprocal rank (MRR) averages 1 over the rank of the first relevant item; and normalised discounted cumulative gain (nDCG@k) sums graded relevance discounted by rank and divides by the ideal ordering’s sum, from the 2002 paper that defined it. Two traps when comparing numbers across tools: whether the gain is linear or exponential, and whether the ideal is computed over all judged relevant items or only the retrieved ones. BEIR and MTEB report nDCG@10 because precision and recall ignore rank and MRR can’t use graded labels; MS MARCO reports MRR@10 because 94% of its queries have one labelled passage. The Ragas framework’s “context precision” is average precision over relevance an LLM judged, and its LLM “context recall” counts the claims of a reference answer the context supports: both useful, neither the recall above, even when the names match.

Which one you watch follows from what the prompt carries. Recall@k at the k you pass to the model is the number that says whether the answer-bearing chunk was there at all; Anthropic measures its retrieval as 1 minus recall@20 because 20 is what it passes on. MRR when a single passage answers. nDCG when relevance is graded and order inside the top k matters. Precision when the context window is tight. And k itself is a trade: Lost in the Middle found reader accuracy plateaued past 20 documents (about 1.5 points for GPT-3.5-Turbo) while retriever recall kept rising, and The Power of Noise recommends three to five. The common way out is to retrieve wide and keep few: Anthropic retrieves 150 chunks, reranks them and keeps 20; Cohere’s demo retrieves 100 and keeps 3; Pinecone’s retrieves 25 and keeps 3.

Building the question set is the hard part, and the tooling has three sources for it. Synthetic questions from the chunks themselves: Ragas builds a knowledge graph over the documents and draws single-hop specific, multi-hop abstract and multi-hop specific queries from it; DeepEval’s synthesizer evolves a seed question in seven ways, of which only four stay tied to the context; LlamaIndex’s generate_question_context_pairs writes questions per chunk and keeps the chunk ids as labels; Giskard’s RAGET splits a set equally across six question types. Expert-written questions, the way the Ragas paper built its own evaluation set, 50 pages with one question each and two annotators who agreed about 95% of the time. And questions mined from real logs, which LangSmith, Braintrust and Phoenix let you promote from a trace into a dataset with a click.

The pitfalls are documented too. Generated questions echo their chunk and are too easy: the Hugging Face cookbook scores each generated pair on groundedness, relevance and stand-alone quality from 1 to 5 and keeps only those scoring at least 4 on all three, which drops about half, so it generates more than 200 to keep about 100; Promptagator and ARES keep a generated query only if its source passage ranks first. A 2024 study found fully synthetic test collections ranked systems like real ones (a Kendall’s tau of 0.86) but inflated every system’s absolute score, and that GPT-4’s relevance judgements agreed with human assessors only fairly (a Cohen’s kappa of 0.26). Answers that live in several chunks need multi-hop questions, and a dataset built by reading only the first relevant id drops the second hop. And a model judging questions it wrote prefers them: GPT-4 recognises its own text 73.5% of the time, and self-recognition and self-preference move together.

How many questions depends on the gap you want to see. TREC runs about 50 topics a year; ARES needs 150 to 300 human labels to separate systems; the cookbook aims at about 100 kept questions. Topic set size design, the IR method for choosing the number, takes the minimum difference you care about and the variance seen on a pilot: one worked example lands at 41 topics with a good variance estimate and 58 with a poor one, for a difference of 0.15. A Vespa experiment on 100 queries shows the scale: recall@10 of 0.937 against 0.845 came with bootstrap intervals of [0.891, 0.973] and [0.778, 0.909], which overlap, so a nine-point gap on 100 queries was its less certain finding. Tens of questions separate large differences; a few hundred are needed for a few points. When you compare two indexes, compare them query by query on the same set with a paired t-test, the default ranx uses; the 2019 study of significance tests in IR recommends it and says the Wilcoxon and sign tests should be discontinued.

Two decisions keep the set alive. Label documents, not chunk ids: TREC’s relevance files name a document, and a chunk id changes the moment you change the splitter, so the labels would go stale with every experiment. And version the set with the corpus snapshot in the same repository, the way the IR tradition keeps topics and judgements beside the collection.

My set has 55 questions written from the spec pages: 30 factoid, 6 multi-hop across pages, 8 paraphrases in plain words with none of the spec’s vocabulary, 6 keyword queries, and 5 out of scope, each labelled with the pages that answer it. The script embeds the questions, takes the top chunks from the index, reduces them to pages in rank order and scores with ranx.

# eval_retrieval.py, abridged
qrels[g["id"]] = {page: 1 for page in g["relevant_pages"]}
run[g["id"]] = {page: 1.0 / (rank + 1) for rank, (page, _) in enumerate(ranked)}
scores = evaluate(Qrels(qrels), Run(run), ["hit_rate@5", "recall@5", "recall@10", "mrr@10", "ndcg@10"])

By type, every factoid, keyword and multi-hop question had a right page in its top 5; the paraphrases had a hit rate of 0.88 and an MRR of 0.68. The one miss asked, in plain words, whether the server remembers anything between two calls on the same connection, and got the versioning, transports and cancellation pages instead of the statelessness section. The five out-of-scope questions had a median top-1 similarity of 0.481 against 0.625 for the answerable ones, but the ranges overlapped (an answerable question at 0.402, an out-of-scope one at 0.587), so a similarity threshold alone would not have separated them.

The result I keep thinking about is the one the metrics called a hit. The question “Is it OK to ask the user for their password in a pop-up form?” is answered by one sentence of the elicitation page, which says servers must not request passwords through a form. That page was in the top 5, at rank 2, through a different chunk about identifying the user; the chunk with the sentence sat at rank 6, with a similarity of 0.381, under a chunk of the prompts page about validating inputs. The page-level label counted a hit, the generator was given the top 5 chunks, and it answered “I don’t know”. That is failure point two from the 2024 paper, and the eRAG paper’s warning in miniature: a label about the page tells you less than a check of whether the model could answer from what it was given.

Layer 4: the answer, with a judge

A judge is the only way to score the answer itself, and it is also the layer where the numbers deserve the least trust, so I keep it last, small and calibrated. The metrics are standard across the frameworks, with differences in the fine print.

DeepEval’s five RAG metrics follow the same shape with two differences worth knowing: its faithfulness passes a claim that the context doesn’t contradict, where Ragas requires that the claim can be inferred from the context, and it keeps a context relevancy metric that scores chunks against the question with no reference. Everything that scores the retriever against a ground truth exists in a judge-free form once you hold reference chunks or their ids, which is the previous layer.

The evidence on judges is consistent. In the Ragas paper, on 50 items with a 2023 model, agreement with human preference was 0.95 for faithfulness, 0.78 for answer relevance and 0.70 for context relevance, which its authors call the hardest. The MT-Bench study found GPT-4 agreeing with human experts 85% of the time, against 81% for humans with each other, but also measured the biases: swapping two answers’ positions left GPT-4’s verdict unchanged 65% of the time, GPT-3.5’s 46% and Claude-v1’s 24%; padding an answer with a rephrased copy of its own list fooled GPT-3.5 and Claude-v1 on 91.3% of cases and GPT-4 on 8.7%. A 2026 study of 21 judge models and about 541,000 judgements found them repeatable (test-retest above 0.95) yet found chance-corrected agreement, Cohen’s kappa, 33 to 41 points below raw agreement, and judge rankings moving by up to 14 places between benchmarks, so a judge can be repeatable without being right.

Judged context relevance is the weak link in three separate studies: ARES beat Ragas’s context relevance by about 60 accuracy points with a fine-tuned DeBERTa classifier and 150 to 300 human labels; RAGBench found TruLens’s context relevance at a root mean squared error of 0.73 on one dataset where the fine-tuned classifier got 0.11; and eRAG found LLM relevance annotations correlating with downstream performance at a tau of at most 0.06 to 0.19. Faithfulness fares best, and gets better with examples: FaithJudge (2025) took a zero-shot GPT-4o faithfulness judge from 56 macro-F1 to 82 by showing it human-annotated responses to the same source.

The practical guidance I read, from practitioners’ essays to Anthropic’s and OpenAI’s own documentation, agrees on a few points. Label about 30 examples by hand to start and keep going until no new failure mode appears, then validate the judge on roughly 100 per failure mode. Report kappa, or precision and recall, rather than raw agreement, because a judge that always says pass looks accurate when failures are rare. Ask for a binary verdict with a written reason rather than a score out of ten. Use a judge that didn’t write the answers it grades. Pin the judge’s model id and prompt, and re-run the calibration after a model update. Keep the judge’s own evaluation as a small test suite of its own.

Judges cost money on every run, so the model choice is part of the design. As read on 23 October 2026, per million tokens in and out: gpt-5.4 $2.50 and $15 (DeepEval’s default), gpt-5.4-mini $0.75 and $4.50, gpt-5-mini $0.25 and $2, gpt-4.1-mini $0.40 and $1.60, gpt-4o-mini $0.15 and $0.60, Claude Haiku 5.5 $0.10 and $0.50, Claude Sonnet 5.5 $2 and $10, with batch tiers at half. Ragas and Phoenix ship no default judge and require a model id; prompt caching matters because the rubric repeats on every call.

My judge is deliberately small: two prompts, a different model from the generator, both pinned in one place.

Split the answer into its atomic factual claims, one per item, each a complete sentence
that stands on its own. Return JSON: {"claims": [...]}. If the answer is only
"I don't know", return an empty list.

You check whether a claim is supported by a context. A claim is supported only if it can
be inferred from the context; general knowledge does not count. Return JSON:
{"supported": true or false, "reason": "one sentence"}.

On six questions, with gpt-5-mini writing answers from the top 5 chunks and gpt-5.4-mini judging, five answers scored 1.0 on every claim, and the sixth was the “I don’t know” from the previous layer, which has no claims to judge. The control, an answer with a planted false sentence about a default port, scored 0.67 and the planted claim was the one flagged. The seven verdicts cost the judge 35,127 tokens in and 1,632 out, about three cents at the prices above; the six answers cost the generator under a cent. The rule I take from all this: validate the index with deterministic recall and precision against labelled chunks, validate the pipeline with a judged faithfulness metric, treat judged context relevance as the least trustworthy number on the sheet, and calibrate whichever judge you use against a hand-labelled sample.

Keep it running

The checks pay off when they run on every change to the chunker, the embedding model or the index settings, and after every re-index, and the retrieval checks are cheap enough to run every time. Every framework documents a way to gate a build: Ragas’s pytest guide asserts context recall and context precision at or above 0.95 and answer relevancy at 0.9, with an in_ci mode that trades cost for reproducibility; Promptfoo’s sample workflow fails under a 95% pass rate; DeepEval runs as pytest with a cache keyed on the test case and the metric configuration; LangSmith’s plugin records HTTP cassettes you commit so CI replays them; Braintrust runs a sampled smoke test on pull requests and the full set on merge. The retrieval-only metrics need no judge, so a run costs one embedding per chunk and per question, and the question embeddings can be cached beside the golden set since they’re deterministic for a fixed model. The example workflow in my repository runs layers 1 to 3 and fails below a recall@10 of 0.90.

In production the traces already hold what you need. The OpenInference conventions that Phoenix and Arize use record a retriever span with each document’s id, score and content; LangSmith has a retriever run type; Langfuse has retriever and embedding observation types with the query, the top k and the model. Online evaluators run on a sample of those traces with a budget: LangSmith has a sampling rate and a weekly spend cap per evaluator, Langfuse runs judges per observation on a queue, Azure AI Foundry’s continuous evaluation defaults to 100 runs an hour. The signals of a degraded index come from the same data: a falling top-1 or mean top-k score over time, more refusals (Evidently has a descriptor that labels a response as a decline), more queries with nothing above a threshold (remembering that my out-of-scope and answerable similarities overlapped), and drift in the queries or the corpus. Evidently’s embedding drift has four methods with defaults, a classifier that tells reference from current (ROC AUC above 0.55 is drift), the share of drifted components (0.2), the distance between mean embeddings (0.2) and maximum mean discrepancy (0.015); Arize measures the Euclidean distance between the centroid of production and the centroid of a baseline.

Re-indexing is where a good index turns bad quietly, and the pattern that protects it is older than vector stores. Build the new index under its own name, validate it with the golden set while traffic stays on the old one, compare the two indexes’ top-k lists per query (the Jaccard overlap for an order-insensitive view, rank-biased overlap when the top matters more), then move an alias in one request. Elasticsearch and OpenSearch, where the pattern comes from, make the alias change atomic; Qdrant does the same, with the delete and create of an alias in one request; Weaviate added collection aliases in version 1.32; Azure AI Search aliases propagate within about 10 seconds and can’t be the target of an indexer. Pinecone has no alias, so its migration guide says to pause writes or log them and replay them into the new index; pgvector has none either, so the swap is two table renames in one transaction.

Incremental updates have their own traps. LangChain’s indexing API hashes each document and offers cleanup modes, and only its full mode removes a document the loader no longer returns, so a loader that returns part of the corpus deletes documents it shouldn’t; LlamaIndex’s ingestion pipeline needs its UPSERTS_AND_DELETE strategy for the same effect. Deleted points linger in the graph after they stop being returned: Qdrant rewrites a segment once a fifth of it is deleted, Weaviate cleans tombstones every 300 seconds by default, Milvus compacts at the same deleted ratio. The validation that matters is a probe set of recently deleted and recently added ids, plus a count reconciliation against the source, run after the write and again after the cleanup window.

The tools

The evaluation frameworks score answers and judge retrieved context; almost none of them measures the index. Here are 21 of the 25 I read, as they stood on 23 October 2026. The legend is Yes for a documented feature, Part for a limited form or a paid tier, – for not offered on the pages I read, and ? for unknown.

A few notes the table can’t hold. Ragas’s repository had no commit between 24 February and 23 October 2026 while issues kept arriving, so treat it as slow or paused; its organisation is now Vibrant Labs. LangSmith’s SDK repository was archived on 16 October 2026 and the code moved to new repositories, which is why its star count is tiny; Braintrust split its Python SDK out the same way. Phoenix computes nDCG, MRR and precision@k, but from relevance scores an LLM produced, not from labels. UpTrain has had no commit since 2024, Tonic Validate’s interface was sunset on 1 August 2025 and RAGChecker has not moved since December 2024. Inspect AI has no retrieval scorer and Vertex AI’s evaluation service has groundedness but no context relevance or recall, so both are out of the table. Only three tools publish any agreement between their judges and humans: TruLens, with its own figures (groundedness F1 0.81), RAGChecker, with a 280-pair study in which its score correlated with human preference at a Pearson of 61.93 against 48.31 for the best Ragas metric, and Phoenix, with a blanket claim of an F1 of 85% or higher.

What none of them covers is the index. Not one page I read compares an approximate index’s results with an exact search over the same vectors, mentions build parameters or quantization, counts chunks against the source, checks for duplicates or empty chunks, or verifies that the query path and the index share an embedding model and dimension. The labelled-retrieval tools score a ranked list against relevance labels and are blind to whether the list came from an exact or an approximate search. Labelled query-to-chunk ground truth is scarce too: LlamaIndex, Giskard and Ragas’s generator produce it; the hosted platforms manage question-answer pairs and judge retrieval with an LLM instead. That gap is what the first two layers of this article, and the first two scripts in the repository, are for.

Method and caveats

Everything here was read on 23 October 2026: the papers by their arXiv pages, the vendor documentation and the tools’ repositories and package pages, with star counts, versions and prices as they stood that day. Where a claim rested on a page I could not open, or on a single secondary source, I left it out; that includes several often-quoted figures, among them OpenAI’s launch benchmarks for its embedding models, a recall expectation attributed to Pinecone’s older pods, and every incident number in practitioner posts about indexes going wrong, for which I found no official account.

The code was run the same day. The corpus is the 31 Markdown pages of the MCP specification, revision 2026-07-28, fetched at a pinned commit of its public repository and not redistributed; the questions and the code are MIT. The embeddings are text-embedding-3-small called through OpenRouter, which lists OpenAI’s models at OpenAI’s prices; the generator is gpt-5-mini and the judge gpt-5.4-mini through the same endpoint. The store is Chroma, embedded, with its default HNSW settings (16 neighbours, a build effort of 100 and a search effort of 100); I had meant to use Qdrant’s local mode and dropped it on finding that it is a brute-force prototype with no approximate index, so an approximate-against-exact check there would read 1.0 by construction.

536 vectors are far too few for an HNSW graph to miss anything, so the recall check demonstrates the method, not a result. I ran one embedding model, no hosted store, no quantization and no filtered search. The question set is 55 questions I wrote myself, labelled by page, and the judge saw six answers; none of the synthetic generators was run. Your numbers will differ with the model, the chunker, the corpus and the day. The research notes behind the article, with every source and the day it was read, are not published; the repository has the code and the questions.