1Retriever: the job
Retriever: given a query, fetch relevant context from the vector database for the generator to use.
A strong RAG answer begins with high-quality retrieval.
2How retrieval works
Query→Embedding→Vector search→Top-k chunks→Context
3Build before evaluating
Load documents→Chunk→Embed→Index→Retrieve
Evaluate each component as you build—not only at the end.
4Failure mode 1: missing context
Information needed to answer the question exists in the knowledge base but was not retrieved.
Measure with recall.
5Failure mode 2: noisy context
Too many irrelevant chunks are retrieved, distracting the generator and using context-window space.
Measure with precision.
6The ideal retriever
High recallFind all facts required for the answer.
High precisionReturn mostly useful chunks, early.
The target is high performance on both metrics.
7Recall–precision trade-off
Increasing k returns more chunks. This can improve recall but often lowers precision because extra chunks may be noise.
Example: k = 5 → 3 relevant; k = 10 may find the missing 2 but add 5 irrelevant chunks.
8Reference-based evaluation
Recall and precision need a reference for what the correct information should be.
Reference: a golden dataset with user question + ideal answer.
9Why not label chunks?
Exact relevant chunk IDs change if you alter chunk size or chunking strategy.
Better reference: an ideal answer remains valid even when chunks are reorganized.
10Golden dataset row
realistic questionideal answerexpected facts / claims
Use student-like questions for a student doubt-solving RAG app.
11Contextual recall
Supported ideal-answer claims ÷ total ideal-answer claims
Did the retrieved context contain all information needed for the ideal answer?
12Contextual recall workflow
Ideal answer→Break into claims→Check retrieved context→Coverage score
13Recall example
Ideal answer has 3 atomic claims. Retrieved chunks support all 3.
Contextual recall = 3 / 3 = 1.0
If only one of two claims is found: 1 / 2 = 0.5.
14Contextual precision
Rank-aware relevance of the retrieved chunks
Are useful chunks near the top of the context, with minimal noise?
15Precision intuition
- Relevant chunk at rank 1 → good
- Relevant chunk buried after noise → worse
- Many unrelated chunks → lower precision
- Top-ranked context should help answer the query
16Contextual vs Recall@K
| Recall@K | Contextual recall |
|---|
| Reference | Relevant chunk IDs | Ideal answer claims |
| Chunking change | Can break labels | Still valid |
| Judge | Exact match | LLM-as-a-judge |
17LLM as judge
The judge decomposes the ideal answer into claims, then determines whether each claim is supported in the retrieved context.
Benefit: semantic evaluation without fragile exact chunk labels.
18Golden data sources
- Manual authoring by a domain expert
- LLM-assisted generation + human review
- Synthetic-data tooling
- Successful production interactions
Warning: synthetic questions can be unnatural or off-topic.
19How to improve recall
- Increase
k carefully - Improve chunking strategy / chunk size
- Use stronger embedding models
- Improve document coverage
- Add metadata filtering where appropriate
20How to improve precision
- Lower
k when safe - Improve metadata filters
- Use reranking
- Improve chunk boundaries
- Remove noisy or duplicate source content
21Evaluation loop
Golden question→Retrieve top-k→LLM judge→Recall + precision→Inspect failures
22What to inspect
Do not rely only on average scores. Read low-scoring questions and discover whether the problem is embedding, chunking, top-k, filtering, source data, or relevance ranking.
23Mental model
A retriever is good when it brings all the facts needed to answer the question and avoids irrelevant noise.
Rule: high contextual recall + high contextual precision = useful RAG context.