1RAG: the core idea
RAG: retrieve relevant context from a knowledge base, then generate an answer grounded in that context.
A RAG chatbot is an LLM-powered application—not just a standalone model.
2Case study: Campus X
A student doubt solver answers questions from course transcripts and learning material.
Student question→Retriever→Context→Generator→Answer
3Why RAG eval is different
An incorrect answer can come from retrieval, generation, the pipeline connection, or the broader product experience.
Therefore: test at more than one level.
4Complete framework
Component level→Pipeline level→Application level
These layers together form the RAG evaluation framework.
5Component level
RetrieverDid it fetch the right context?
GeneratorDid it write a good answer from the context?
6Retriever job
- Turn a query into an embedding
- Search a vector database
- Return the most relevant chunks
- Give the generator useful context
Risk: missing facts or retrieving noisy chunks.
7Generator job
- Read question + retrieved context
- Produce a relevant response
- Stay grounded in the context
- Use a helpful, suitable style
Risk: hallucination or an irrelevant answer.
8Pipeline-level eval
Tests the end-to-end RAG flow: can the system answer the user question correctly after retrieval and generation work together?
Focus: the answer—not isolated component scores.
9RAG Triad
context relevancefaithfulnessanswer relevance
The three key dimensions for evaluating an end-to-end RAG response.
10Context relevance
Question ↔ retrieved context
Is the retrieved information actually useful for answering this specific question?
11Faithfulness
Answer ↔ retrieved context
Are the answer's claims supported by the retrieved context, with no invented facts?
12Answer relevance
Question ↔ final answer
Does the answer directly address what the user asked?
13Triad diagnosis
| Symptom | Likely issue |
|---|
| Low context relevance | Retriever problem |
| Low faithfulness | Generator hallucination |
| Low answer relevance | Pipeline/prompt/task issue |
14Application level
Tests the user-facing product beyond the RAG answer itself.
- UX and clarity
- Latency and reliability
- Safety and guardrails
- Conversation / session behavior
- Real user satisfaction
15Golden dataset
Golden data: realistic user questions with trusted expected answers or evaluation references.
Use it to make evaluation repeatable across RAG versions.
16Reference-based vs free
Reference-basedNeeds a golden answer / context.
Reference-freeUses context or judging without a gold answer.
17Eval suite
Eval suite: a maintained collection of evaluation cases, metrics, and pass/fail thresholds.
Run it whenever you change a model, prompt, chunks, embedding, or retrieval logic.
18Regression testing
Old versionvsNew version→Same eval suite→Compare scores
Goal: ensure a change improves the product without breaking earlier behavior.
19DeepEval
An evaluation library that can structure test cases, run metrics, use LLM-as-a-judge scoring, and support regression testing workflows.
Use: automate repeatable RAG evaluation.
20Online evaluation
After deployment, monitor real interactions, feedback, latency, errors, and failure patterns.
Rule: offline evals start quality assurance; production signals keep it honest.
21Offline → online loop
Golden evals→Deploy→Observe users→Add failures→Re-evaluate
22Best build order
Build retriever→Evaluate→Build generator→Evaluate→Evaluate full app
23Mental model
Don’t ask only: “Does the chatbot answer questions?” Ask: “Does it retrieve relevant context, generate faithful answers, and deliver a good product experience?”
Rule: component + pipeline + application = trustworthy RAG evaluation.