LLM EVALUATION • RAG APPLICATION EVALUATION

How Do You Evaluate Your RAG Chatbot?

Detailed English study notes from the lecture: playlist recap, RAG evaluation framework, component/pipeline/application-level evaluation, Eval Suite, DeepEval, regression testing, CI/CD gating, online evaluation, observability, drift detection, and the self-improving loop.

Case Study: Campus X Doubt Solver Focus: RAG Evaluation Library: DeepEval Interview Question: RAG Evaluation

1. Where We Are in the LLM Evaluation Playlist

The first few lectures focused on the fundamentals of LLM evaluation and why evaluation is necessary for LLM-based systems.

Reference-Based Eval

Evaluation where a reference or expected answer is available.

Reference-Free Eval

Evaluation where a reference answer is not necessarily required.

Online vs Offline

Offline evaluation happens before deployment; online evaluation continues after deployment.

The Two Major Types of LLM Evaluation

Model Evaluation

Evaluates the underlying LLM/model itself.

Application Evaluation

Evaluates an LLM-powered application. This becomes the major focus from this point onward.

Model Evaluation Covered So Far

  • Standardized evaluations / benchmarks: standardized tests used to evaluate model capabilities.
  • Custom model evaluations: evaluations created using your own dataset, especially when selecting a model for a particular application.
Major milestone: The playlist now moves from model evaluation to the most important practical part — application evaluation.

2. Types of LLM-Based Applications

LLM applications can take many forms. The lecture highlights several examples:

General Chatbot

A normal conversational application without RAG or agent functionality.

RAG Chatbot

A chatbot that retrieves relevant information from an external knowledge base before generating an answer.

Agents

LLM-based systems capable of performing more complex workflows and actions.

Multimodal Apps

Applications that work with modalities beyond text, such as image generation.

Fixed-Schema Output

Applications where the LLM produces output following a predefined schema, such as email classification.

Image-Based Applications

Applications in which images are an important input or output modality.

Course selection: It is not practical to teach evaluation for every possible application type. The course therefore focuses on RAG applications and agent applications, because these patterns are especially important in professional projects.

3. Case Study — Campus X Doubt Solver

Problem Statement

Build a RAG chatbot for the LLM course

The course contains lectures, and each lecture has a transcript. These transcripts become the knowledge documents for the chatbot.

A student can ask questions or doubts about the LLM course, and the chatbot should answer using the available lecture transcripts.

1. Lecture TranscriptsCourse lecture transcripts become source documents.
→
2. RetrievalRelevant transcript chunks are retrieved for a user query.
→
3. GenerationThe LLM uses the question + retrieved context to generate an answer.
→
4. Answer + CitationThe chatbot answers and can identify the relevant lecture/session.
The goal is intentionally not to create a highly complicated RAG system. The main objective is to learn how to evaluate even a simple RAG application properly.

4. Complete RAG Evaluation Framework

The central idea is that an LLM application should not be evaluated with one single test. It needs an evaluation suite covering multiple perspectives.

Level 1

Component Level
Evaluate individual RAG components such as the retriever and generator.

Level 2

Pipeline Level
Evaluate the complete RAG pipeline after retriever and generator are connected.

Level 3

Application/System Level
Evaluate the complete user-facing application.

Overall Development + Evaluation Flow

Step 1: Build the retriever.
Step 2: Evaluate the retriever independently.
Step 3: Build the generator.
Step 4: Evaluate the generator independently.
Step 5: Connect retriever + generator to create the RAG pipeline.
Step 6: Evaluate the RAG pipeline using the RAG Triad.
Step 7: Evaluate application quality.
Step 8: Evaluate application safety.
Step 9: Evaluate operational metrics.
The key principle: Do not build the entire RAG chatbot first and test only at the end. Build and evaluate continuously — just like software is tested at function, feature, integration, and system levels.

5. Component-Level Evaluation

1

Retriever

The retriever is responsible for finding relevant documents/chunks from the vector database for a user query.

Basic Retriever Flow

  1. Load documents.
  2. Chunk the documents.
  3. Convert chunks into vectors using an embedding model.
  4. Store the vectors in a vector database.
  5. Receive a new user query.
  6. Retrieve the most relevant documents/chunks.

Retriever Metrics

MetricWhat it asks
RecallOut of all the relevant/correct documents that should have been retrieved, how many did the retriever actually retrieve?
PrecisionOut of all the documents retrieved, how many were actually useful/relevant?
Retriever goal: retrieve the right context — not simply retrieve a large number of documents.
2

Generator

The generator is the LLM that receives a question and relevant context and produces an answer.

At this stage, the generator is evaluated in isolation. It is not yet connected to the retriever.

Generator Metrics

MetricWhat it checks
FaithfulnessWhether the generated answer is supported by the supplied context rather than containing unsupported hallucinated information.
Answer RelevanceWhether the generated answer is relevant to the user's question.
Citation AccuracyWhether the citations/references provided by the application correctly point to the source information.
Golden-data idea: At the isolated generator stage, questions and contexts can be supplied manually as a controlled/golden dataset rather than coming from the retriever.

6. Pipeline-Level Evaluation — The RAG Triad

After the retriever and generator have independently passed their evaluations, they are connected to create the RAG pipeline.

User QueryThe original question.
+
Retrieved ContextContext fetched from the vector database.
→
Generated AnswerAnswer produced by the generator.

The Three RAG Triad Metrics

PairMetricPurpose
Question + ContextContext RelevanceChecks whether the retrieved context is relevant to the question.
Context + AnswerFaithfulnessChecks whether the generated answer is grounded in the retrieved context or contains hallucination.
Question + AnswerAnswer RelevanceChecks whether the generated answer is relevant to the original question.
RAG Triad = Context Relevance + Faithfulness + Answer Relevance. Together, these metrics provide a compact way to inspect whether the RAG pipeline is functioning properly.

7. Application-Level Evaluation

Once the pipeline works, evaluate the complete Campus X Doubt Solver as a user-facing application.

Correctness

Is the answer actually correct?

Completeness

Does the answer address every part of the user's question? A response can be partially correct but still incomplete.

Style

Does the response style match the desired Campus X teaching/explanation style?

Safety Evaluation

Toxicity

Does the response contain toxic or inappropriate content?

PII Leakage

Does the application expose personally identifiable information?

Jailbreak Resistance

Can the chatbot be manipulated into violating intended behavior?

Operational Evaluation

Latency

How long does the application take to answer?

Cost per Query

How much does each interaction cost?

Token Usage

How many tokens are being consumed?

8. What Is an Eval Suite?

An Eval Suite is the complete collection of evaluations used to test the application from multiple perspectives.

For the RAG chatbot, it includes component, pipeline, application, safety, and operational evaluations.

Conceptual Project Structure

rag-project/
│
├── src/
│   ├── retriever.py
│   ├── generator.py
│   ├── rag_pipeline.py
│   ├── api.py
│   └── ui.py
│
├── evals/
│   ├── eval_retriever.py
│   ├── eval_generator.py
│   ├── eval_rag_pipeline.py
│   ├── eval_application.py
│   ├── eval_safety.py
│   └── eval_operations.py
│
└── run_evals.py

Role of run_evals.py

The idea is to have one entry point that triggers the different evaluation files and produces an overall report.

This makes the complete evaluation suite repeatable and useful for regression testing.

9. DeepEval — Evaluation Library

Instead of writing all evaluation logic from scratch, the lecture chooses DeepEval as the primary evaluation library for the practical RAG evaluation work.

Many of the required metrics are already available, including RAG and safety-related metrics.

RAG Metrics

Answer relevance, faithfulness, contextual precision, contextual recall, contextual relevance, and related evaluation capabilities.

Safety

Toxicity, PII leakage, and other safety-oriented evaluations.

Broader Scope

The lecture highlights use cases beyond RAG, including agents, multi-turn chatbots, non-LLM applications, and image-based applications.

Why DeepEval instead of Ragas in this course?
  1. Ragas has already been covered in the advanced RAG course.
  2. DeepEval is presented as a broader evaluation library with a wider scope.
  3. Its syntax is based around Pytest, making it familiar to people who have worked with Python software testing.
The lecture's broader advice is to understand the concepts, not become dependent on a particular tool. Tools can change from company to company.

10. Regression Testing

After building the evaluation suite, the next important step is regression testing.

The idea is simple: run the complete evaluation suite on a new version of the application and compare the results with a previous/baseline version.

Version 1

Baseline configuration + baseline evaluation metrics.

Version 2

New configuration + new evaluation metrics.

What Regression Testing Answers

Three Levels of Regression Testing Discussed

Level 1 — Simple

Run the eval suite and manually compare new results with the baseline.

Level 2 — Experiment Tracking

Log configurations and metrics for repeated experiments and compare them visually.

Level 3 — CI/CD

Automatically run evaluations after code changes and gate deployment based on thresholds.

Experiment Tracking Example

Suppose the first run uses a particular chunk size, overlap, temperature, and embedding configuration.

The evaluation produces values such as:

MetricExample Baseline
Retriever Recall82
Retriever Precision68
Other Eval MetricsLogged for comparison

Then you change chunking size or overlap and run the suite again. The new results are logged and compared with the baseline.

Experiment Tracking Tools Mentioned

The lecture mentions tools such as MLflow, Confident AI, and Weights & Biases as possible options.

CI/CD Gating

  1. A developer changes code or configuration.
  2. The change is pushed to the repository.
  3. A CI system such as GitHub Actions triggers the evaluation suite.
  4. New metrics are compared with the baseline.
  5. A threshold can be defined — for example, a metric should not drop by more than an allowed amount.
  6. If the new version passes, deployment is allowed.
  7. If the new version regresses beyond the threshold, deployment is stopped.
Core idea: Evaluation becomes a deployment gate. A change should not automatically reach production if it causes unacceptable regression.

11. Online Evaluation After Deployment

Evaluation does not stop when the application is deployed. Once the chatbot is live, the system should continue to be monitored and evaluated.

Observability / Tracing

The lecture mentions tools such as LangSmith, Langfuse, and Confident AI for tracing and observability.

Tracing code can capture live interaction data such as:

Latency

How long the response took.

Cost

Cost associated with the interaction.

Tokens

Token consumption.

User Feedback

Thumbs up / thumbs down signals.

Traces

Information about the execution path of the application.

Dashboard

Visualize captured production signals.

Online Quality Evaluation

Some metrics that were evaluated offline can also be evaluated on live traffic, including:

Drift Detection

Drift means that application performance changes or degrades over time.

For example, suppose a dashboard tracks faithfulness over the previous 24 hours. If faithfulness suddenly drops during the last several hours, that may indicate a drift event that should trigger investigation and corrective action.

Self-Improving Evaluation Loop

Live InteractionA real user interacts with the chatbot.
→
Failure ExampleA problematic or incorrect response is identified.
→
Add to Offline DataThe example becomes part of the offline/golden evaluation dataset.
→
Future EvaluationFuture versions are tested against the richer dataset.
This creates a feedback loop in which production failures improve the offline evaluation dataset, making future versions easier to evaluate more thoroughly.

12. Roadmap for the Next Four Sessions

Session 1 — Component-Level Evaluation

Build the retriever, evaluate it, build the generator, evaluate it, and introduce DeepEval in practice.

Session 2 — RAG Pipeline + RAG Triad

Connect retriever and generator, create the RAG pipeline, and evaluate context relevance, faithfulness, and answer relevance.

Session 3 — Application-Level Evaluation

Run the complete application evaluation, including quality-oriented metrics and broader application checks.

Session 4 — Regression + Online Evaluation

Cover regression testing and the post-deployment online evaluation/observability process.

13. Interview Master Answer — “How Do You Evaluate Your RAG Chatbot?”

“I would not evaluate a RAG chatbot using only one or two metrics. I would build a complete evaluation suite and evaluate the application at component, pipeline, and application levels. Then I would use the suite for regression testing before deployment and continue with online evaluation after deployment.”

Step-by-Step Interview Structure

  1. Build an Eval Suite. Explain that evaluation happens at multiple levels.
  2. Component level: evaluate the retriever using Recall and Precision.
  3. Generator level: evaluate Faithfulness, Answer Relevance, and Citation Accuracy.
  4. Pipeline level: evaluate the RAG Triad — Context Relevance, Faithfulness, and Answer Relevance.
  5. Application level: evaluate Correctness, Completeness, and Style.
  6. Safety: evaluate Toxicity, PII Leakage, Jailbreak Resistance, and other applicable safety checks.
  7. Operations: monitor Latency, Cost per Query, and Token Usage.
  8. Regression testing: run the complete eval suite whenever a new version/configuration is created and compare it with a baseline.
  9. Automation: use experiment tracking and, where appropriate, CI/CD gates to prevent unacceptable regressions from being deployed.
  10. Deployment: deploy only after the new version passes the required evaluation criteria.
  11. Online evaluation: monitor live traces, latency, cost, tokens, user feedback, quality metrics, and drift after deployment.
  12. Continuous improvement: collect production failures and add useful examples back into the offline/golden evaluation dataset.
Interview differentiator: Merely saying “I check recall, precision, and answer relevance” is incomplete. The stronger answer is a framework: component → pipeline → application → safety/operations → regression → deployment → online evaluation → feedback loop.

14. One-Page Revision Sheet

StageWhat is evaluated?Key Metrics / Concepts
Component — RetrieverQuality of retrieved documentsRecall, Precision
Component — GeneratorLLM generation in isolationFaithfulness, Answer Relevance, Citation Accuracy
PipelineRetriever + Generator togetherRAG Triad: Context Relevance, Faithfulness, Answer Relevance
Application — QualityUser-facing response qualityCorrectness, Completeness, Style
Application — SafetySafe behaviorToxicity, PII Leakage, Jailbreak Resistance
OperationsProduction efficiencyLatency, Cost, Tokens
RegressionNew version vs baselineBaseline comparison, experiment tracking, CI/CD gates
Online EvaluationLive production behaviorTracing, feedback, quality metrics, drift
Continuous ImprovementLearn from failuresProduction failures → offline/golden dataset

15. Mental Model

BuildRetriever → Generator → RAG Pipeline → Application
→
EvaluateComponents → Pipeline → Application → Safety → Operations
→
RegressionCompare new version against baseline
→
DeployRelease only when evaluation criteria are satisfied
→
MonitorTrace → Evaluate → Detect Drift → Collect Failures
→
ImproveAdd valuable failures to offline evaluation data

16. Quick Self-Check

17. Key Takeaways

  • RAG evaluation is a framework, not a list of three or four metrics.
  • Evaluation should happen continuously during development, not only after the entire application is finished.
  • The three major levels are Component → Pipeline → Application/System.
  • Retriever evaluation focuses on retrieval quality; generator evaluation focuses on grounded and relevant generation.
  • The RAG Triad connects question, context, and answer through Context Relevance, Faithfulness, and Answer Relevance.
  • Application evaluation expands beyond RAG quality into correctness, completeness, style, safety, and operations.
  • An Eval Suite collects these tests into one repeatable testing framework.
  • Regression testing compares new application versions against a baseline.
  • Experiment tracking and CI/CD can automate regression detection and deployment gating.
  • Evaluation continues after deployment through online evaluation, tracing, observability, drift detection, and user feedback.
  • Production failures can feed back into the offline/golden dataset, creating a self-improving evaluation loop.