Evaluating a RAG Retriever

This session continues the RAG-evaluation roadmap. Over the next three classes, the goal is to create a reusable RAG evaluation suite at three levels: component level, pipeline/workflow level, and application level. Together, these checks become an eval suite that can later support regression testing.

Today focuses on the component level. The two most important RAG components are the retriever, which fetches relevant context from a vector database, and the generator, which uses the user question plus retrieved context to produce an answer. This lesson starts with the retriever.

The project is organized into data/ for lecture transcripts, src/ for application code, evals/ for evaluation pipelines, and goldens/ for golden datasets. A virtual environment and the required libraries are set up, with secrets stored in an environment file rather than in source code.

The important engineering principle is to evaluate as you build. Do not build the entire RAG app first and test it only at the end. Build the retriever, evaluate it, then build and evaluate the generator, then evaluate the full pipeline, and finally evaluate the application.

The retriever takes a query, converts it to an embedding, searches a vector database containing embedded text chunks, and returns the nearest k chunks as context. To create it, load the documents, split them into chunks, embed those chunks, store them, and expose a retriever object that can fetch relevant context for a query.

A retriever can fail in two opposite ways. First, it may miss relevant context: information needed to answer the question exists in the knowledge base but was not retrieved. This is measured by recall. Second, it may retrieve too much irrelevant context (noise), which distracts the generator and wastes context window. This is measured by precision.

An ideal retriever has both high recall and high precision. However, there is usually a trade-off. Increasing k, the number of retrieved chunks, tends to raise recall because more relevant chunks may be included; it can lower precision because more irrelevant chunks are also included.

Recall and precision are reference-based evaluations. To know whether the retriever found the correct information, you need a golden reference for each query. A traditional approach labels the exact relevant chunks. But if chunk size or chunking strategy changes, those chunk labels must be redone.

The preferred approach in this lesson avoids that brittle dependency. For each realistic user question, store an ideal answer. The ideal answer remains valid even if the document chunks are changed. An LLM judge can break the ideal answer into atomic claims and check whether the retrieved context supports those claims.

For contextual recall, the LLM judge decomposes the ideal answer into claims. It then inspects all retrieved chunks and identifies which claims are supported anywhere in the context. Contextual recall is the proportion of ideal-answer claims covered by the retrieved context. If all required claims are present, recall is 1.0; if only one of two claims is supported, it is 0.5.

This differs from simple Recall@K. Recall@K depends on pre-labelled relevant chunks; contextual recall uses an ideal answer and an LLM judge, making it more resilient to changes in chunking.

Contextual precision asks whether relevant information appears early in the retrieved list and whether the list is free of noise. The judge examines each retrieved chunk in rank order and marks whether it helps answer the question. Relevant chunks near the top improve the score; irrelevant chunks, especially near the top, reduce it.

Both contextual recall and contextual precision are useful. High recall with poor precision means the correct material exists somewhere in a long noisy context. High precision with poor recall means the returned chunks look relevant but omit essential facts.

A golden dataset contains at least a user-like question and its ideal answer. The lecture reviews four ways to create it: write examples manually, have a capable LLM generate examples under strong instructions, use a synthetic-data tool, or mine successful production interactions. Production logs are valuable later but cannot be the only starting point.

For this course chatbot, the chosen approach is LLM-assisted authoring with detailed instructions and manual quality control. Examples are generated in small batches so they sound like real student questions—not artificial or overly technical questions copied from random transcript text.

For every row in the golden dataset, send the question to the retriever, collect the returned chunks, and pass the question, ideal answer, and retrieved context to the evaluation metrics. Aggregate the per-question contextual recall and contextual precision scores. Inspect low-scoring cases—not only the average—to discover whether chunk size, embedding model, retrieval k, metadata filters, or the source documents need improvement.