1Foundations
What LLM evaluation is, why it matters, vibe testing, real failures
1What Is LLM Evaluation?
Def: a systematic, repeatable testing setup used to judge an LLM or LLM-powered system against clear criteria.
Not one metric — the entire testing setup: component, dataset, criteria, tools, timing (offline/online).
2Three Core Properties
- Systematic — proper dataset, not a few prompts
- Repeatable — same setup re-run across versions
- Clear criteria — explicit definition of "good"
3Vibe Testing
Def: judging on a handful of prompts by gut feeling — not systematic.
Fine for early prototyping. Never production evidence: informal, not repeatable, doesn't cover real input space.
4Real-World Failures
- Air Canada — wrong policy, airline held liable
- Chevy bot — jailbroken into "$1 car" deal
- Fabricated case law — lawyer used unverified output in court
Common thread: systems trusted without evaluation.
5Traditional vs LLM Testing
| Traditional | LLM Apps |
|---|
| Behavior | Deterministic | Probabilistic |
| Check | Exact match | Semantic validity |
6Two Core Challenges
- Non-determinism — exact-match fails
- Multi-dimensionality — correct ≠ complete ≠ grounded ≠ safe ≠ fast ≠ cheap
7Eval Doesn't End at Deploy
Continuous production monitoring required: latency, TTFT, TPS, behavior under real load.
2Model Eval vs Application Eval
The central split that organizes the whole course
8The Two Types
| Model Eval | App Eval |
|---|
| Tests | The LLM itself | Full app/components |
| Question | "Can the model do this?" | "Will our product work?" |
| Owner | Frontier labs | AI/app engineers |
Not strict terms — both often called "LLM evals"; context disambiguates.
9Smartphone Analogy
A great chip ≠ a great phone — camera, OS, battery, software matter too.
Powerful LLM ≠ Excellent App. Model eval = labs' job. Evaluating the system built around it = your job.
10App Has Many More Parts
UISystem promptTools/APIs
OrchestrationGuardrailsOutput parsers
MemoryRAG stackMonitoring
Myth: "LLM = brain → good LLM = good app." False.
11Component vs System Level
| Component | Retriever, embeddings, reranker, prompt, guardrails, parser, tool selection, memory |
| System | Final response: correctness, groundedness, safety, latency, cost, UX |
12Good Model ≠ Good App
A model can top every benchmark and the app still fails from bad chunking, poor retrieval, tool failures, hallucination, unsafe output, high latency/cost.
13Course Rule of Thumb
Course weight is heavily Application Evals — that's the AI engineer's day-to-day job. Model-eval literacy still needed to read benchmarks and pick models.
3Model Evaluation & Benchmarking
8 capabilities, reference benchmarks, benchmark vs custom eval
148 Capability Areas
Knowledge & ReasoningCoding & SWEMath
Long ContextVision/MultimodalAgentic/Tool Use
Safety & AlignmentInstr. Following
15Reference Benchmarks
| Knowledge/Reasoning | MMLU (57 subjects) |
| Math | GSM8K |
| Coding | SWE-bench, HumanEval |
| Instr. Following | IFEval |
| Long Context | Needle in a Haystack |
| Multimodal | MMMU |
16Long Context ≠ Window Size
Stated context window ≠ effective use of it — test retrieval/summarization within large inputs, not just the max size.
174-Step Eval Process
Select capability→Bring a test→Fixed protocol→Score & interpret
18Benchmark vs Custom Eval
| Benchmark | Custom Eval |
|---|
| What | Standardized, shared | Built from your task/data |
| Answers | "How capable generally?" | "Best for our workload?" |
19Case: Email Router
| Model A | Model B |
|---|
| Cost/1M tok | $15 | $0.50 |
| Class. acc. | 94% | 91% |
| Latency | 4.1s | 9s |
Verdict: B nearly matches A at a fraction of cost — better value despite losing on public leaderboards.
20Model-Eval Mental Model
Don't ask "which model is strongest?" Ask "which performs well on what my app needs, within my cost/latency constraints?"
4Application Eval — Levels & Risk
Why one eval is never enough for a real app
21Why Multiple Pipelines?
- Components/workflows fail independently
- Correct ≠ safe ≠ cheap ≠ fast — each is a separate risk
22Component ✓ + Component ✓ ≠ Workflow ✓
K=5 example: correct doc retrieved in top-5 (retriever "passes") but generator prioritizes higher-ranked wrong docs → wrong final answer.
Fix: add a reranker to reorder before generation.
23Three Failure Levels
| Component | Prompt, retriever, reranker, embeddings, parser, tool selector, memory |
| Workflow | Retrieve+generate, plan+act, multi-turn chat |
| Application | Latency/cost/error rate, load, UX |
24Three Risk Categories
Application QualitySafetyOperations
A correct answer can still fail on safety, or be too slow/expensive to ship.
25RAG-Specific Risks
- Context Relevance — docs relevant?
- Retriever Recall — got needed info?
- Groundedness / Faithfulness — no invented facts
- Citation Accuracy — cites actually support claims
26Agent & Multi-Turn Risks
- Tool selection & parameter correctness
- Task completion, error recovery
- Context retention across turns
- Clarification behavior on ambiguity
27Safety & Operational Risks
SafetyToxicity, bias, PII leak, prompt injection, jailbreak
OpsLatency, TTFT, cost/req, tokens, error rate, load
28Final Mental Model
Find failure points→ID risk categories→Eval components→Eval workflow→Eval app→Monitor continuously
5Evaluation Methods
Who/what executes the judgment: code, human, or judge LLM
29The Three Methods
| Method | Strength | Weakness |
|---|
| Programmatic | Fast, cheap, repeatable | Needs measurable criteria |
| Human | Nuanced judgment | Expensive, slow |
| LLM-as-Judge | Scalable + nuance | Judge must be validated |
30Programmatic — Recall@K
Relevant docs in Top-K ÷ Total relevant. Ex: 1 of 2 found → 50%.
Human builds golden labels; scoring itself stays fully code-driven.
31Human Eval
Used when no formula captures quality reliably (e.g. "helpfulness").
Direct gradingRed teamingA/B testingHuman-in-loop
Disagreement signal: repeated grader disagreement → rubric is ambiguous.
32LLM-as-a-Judge
One LLM evaluates another's output — combines programmatic scale with human-like nuance.
Output→Judge LLM→Score + Reasoning
33Validating the Judge — MAE
MAE = mean(|Human score − LLM score|). MAE=2.3 → judge off by ~2.3 marks avg. Goal: MAE → 0.
Improve via: stronger judge LLM, better prompt/rubric, calibration examples, analyzing disagreements.
34Reference-Based vs Free
| Reference-Based | Ground truth per case exists (Recall@K, graded exams) |
| Reference-Free | No per-case answer; judged vs rubric (chatbot helpfulness) |
35Pick the Method
- Code can measure it? → Programmatic
- Needs nuance? → Human
- Needs scale + nuance? → LLM-as-Judge
6Offline vs Online Evaluation
Pre-deployment gating vs continuous production monitoring
36Offline Evaluation
Def: eval before deployment on a controlled golden dataset + fixed criteria.
Dataset→App→Evaluator→Score→Release?
373 Offline Benefits
- Release gate — threshold ≥95% → deploy, else block
- Version comparison — same dataset across models/prompts/rerankers
- Regression testing — one fix shouldn't quietly break another
38What Changes in Production
- Unexpected inputs — never in golden set (mixed language, prompt injection)
- Emergent failures — only visible at scale (latency spikes, bias patterns)
- Drift — real-world data/env changes vs a frozen eval set
39Online Evaluation
Def: evaluates the deployed system on live traffic, continuously — usually no fixed answer key. Core Q: "Is it behaving normally right now?"
40Offline vs Online
| Offline | Online |
|---|
| When | Pre-deploy | Post-deploy, continuous |
| Data | Fixed golden set | Live traffic |
| Answer key | Usually yes | Usually no |
| Best for | Gate/compare/regress | Drift/anomaly detect |
Complementary, not rivals — Correctness (offline) + Normality (online).
41Captured vs Computed Signals
CapturedLogged directly: thumbs up/down, latency, tokens, cost, errors
ComputedNeeds an evaluator: faithfulness, relevance, hallucination, toxicity, bias
42Logging → Dashboard → Alert
Log→Dashboard→Alert→Action
Logging must be non-blocking, durable/queryable, support late signal attach, and mask PII.
43Sampling for Cost
Running a judge on every conversation gets expensive → sample.
RandomSimple but risky convos under-represented
StratifiedOversample high-risk: thumbs-down, escalated, financial
44Same Evaluator, Two Modes
Evaluator type ≠ mode — the data it runs on decides: on traces = online, on a dataset = offline.
45Self-Improving Loop
Dataset→Offline Eval→Release→Prod→Monitor→Failure→Dataset Update
Real failures get annotated & folded back — next release tested against real + historical cases.
7RAG Retriever Evaluation
Component-level: contextual recall & precision, golden datasets
46Two Retriever Failure Modes
Miss relevant contextNeeded info exists but not retrieved → Recall
Retrieve noiseIrrelevant chunks distract generator → Precision
Trade-off: raising K usually ↑ recall but ↓ precision.
47Ideal-Answer Approach
Instead of brittle chunk labels (break on re-chunking), store an ideal answer per question. LLM judge breaks it into atomic claims and checks context support.
48Contextual Recall & Precision
Recall: supported ideal-answer claims ÷ total claims
Precision: rank-aware relevance — useful chunks retrieved early, without noise
49Failure → Metric → Fix
| Symptom | Metric | Fix |
|---|
| Not grounded | Low recall | Retrieval/chunking/embeddings, K↑ carefully |
| Buried in noise | Low precision | Ranking/filter, K↓, reranker |
50Component-Level Workflow
Prepare docs→Build retriever→Create goldens→Retrieve→Judge claims→Analyze & iterate
51Eval Pseudocode
for row in golden_dataset:
ctx = retriever.retrieve(row.q, k=K)
recall = contextual_recall(q, ideal_answer, ctx)
precision = contextual_precision(q, ideal_answer, ctx)
report(mean(recall), mean(precision))
8Full RAG Evaluation Strategy
RAG Triad, Eval Suite, DeepEval, regression testing, online eval
523 Levels of RAG Eval
| Component | Retriever & generator individually |
| Pipeline | Connected — RAG Triad |
| Application | Full system + safety + ops |
53Generator Metrics
- Faithfulness — grounded, no hallucination
- Answer Relevance — answers the question
- Citation Accuracy — cites the right source
54RAG Triad
| Question + Context | Context Relevance |
| Context + Answer | Faithfulness |
| Question + Answer | Answer Relevance |
55Application-Level Checks
CorrectnessCompletenessStyle
ToxicityPIIJailbreak
LatencyCostTokens
56Eval Suite & DeepEval
One entry point (run_evals.py) triggers component/pipeline/app/safety/ops evals for a repeatable report.
DeepEval: Pytest-style syntax, covers RAG + safety metrics, extends to agents/multi-turn/image apps.
57Regression Testing — 3 Levels
| Simple | Manual compare vs baseline |
| Experiment tracking | MLflow, Confident AI, W&B |
| CI/CD | Auto-run, gate deploy by threshold |
58Regression Test Matrix
| System prompt | Also check pricing accuracy, relevance |
| LLM model | Also check latency, cost, safety |
| Vector DB | Also check answer quality, latency |
59Online Eval & Drift
Tracing tools (LangSmith, Langfuse, Confident AI) capture latency, cost, tokens, feedback, traces on live traffic.
Drift: performance changes over time — e.g. faithfulness dropping over last 24h → investigate.
60Interview Master Answer
"I'd build a complete Eval Suite: component → pipeline (RAG Triad) → application (quality/safety/ops) → regression testing → deploy → online evaluation → feedback loop into the golden dataset."
9LLM Leaderboards
4 types, why they can't be trusted blindly, how to read one
61Benchmark → Result → Leaderboard
Benchmark = Exam→Result = Score→Leaderboard = Ranking
Key idea: leaderboards filter, they don't decide — narrow 100+ models to 3–5 candidates you test yourself.
624 Leaderboard Types
| Benchmark-Specific | Best on one benchmark? (MMLU, HLE) |
| Multi-Benchmark | Best overall combo? (LiveBench, Artificial Analysis) |
| Human-Preference | Humans prefer? (LM Arena) |
| Application-Specific | Best for this domain? (Function Calling) |
637 Reasons Not to Trust Blindly
- Doesn't transfer to messy real apps
- Contamination inflates scores
- Goodhart's Law — measure becomes target, less useful
- Hidden methodology (weights, normalization)
- Small diffs (84.3 vs 84.1) ≈ noise
- Human bias favors length/confidence/style
- Stale or self-reported scores
64Selection Funnel
100+ Models→Leaderboard Filter→3–5 Candidates→Custom Evals→Final Model
Start with "What does my app need?" — not "who is #1?"
10Whole-Series Master Mental Model
The single framework tying every session together
65End-to-End Framework
Pick capability→Read benchmarks/leaderboards→Shortlist model→Custom eval→Build component→Eval component→Eval workflow→Eval app (quality/safety/ops)→Regression vs baseline→Deploy→Online monitor→Detect drift→Feed failures back
66Series Key Takeaways
- Evals = a testing framework, not one metric
- Model eval → capability; App eval → product reliability
- Every real app needs >1 eval pipeline
- Programmatic / Human / LLM-Judge — pick by need for scale vs nuance
- Offline gates releases; Online watches for drift — both required
- RAG Triad = Context Relevance + Faithfulness + Answer Relevance
- Leaderboards filter; custom evals decide
- Evaluation is a permanent loop, not a one-time checkbox
67Core Principle
Build → Evaluate → Improve → Deploy → Monitor → Evaluate Again