1Core Concept
Eval method: the mechanism that executes the judgment — who/what turns an output into a score.
Pipeline = what's checked. Method = who/what checks it: code, human, or judge LLM.
2The Three Methods
| Method | Strength | Weakness |
|---|
| Programmatic | Fast, cheap, repeatable | Needs measurable criteria |
| Human | Nuanced judgment | Expensive, slow |
| LLM-as-Judge | Scalable + nuance | Judge must be validated |
3Programmatic — RAG Retriever
Target: retriever · Task: retrieves relevant docs? · Criterion: Recall@K
Golden dataset ≠ method: human builds labels (50–100 Qs), scoring stays fully programmatic.
4Worked Example
Q: "Prereqs for ML course + duration?" Relevant docs: 1001, 1003.
K=5 returns: 1001,1002,1004,1005,1006 → 1 of 2 found.
5Recall@K
Relevant in Top-K ÷ Total relevant
1 ÷ 2 = 50%
Range: 0%–100%. Avg over dataset e.g. 67% Recall@5.
6Dataset-Level Flow
- Send question → retriever
- Get top-K docs
- Compare vs relevant docs
- Compute Recall@K
- Average across all Qs
7Improve the Retriever
Better embeddings
Query expansion
Raise K
Add reranker
Better chunking
Precision/ranking metrics
8Human Eval — Helpfulness
Target: whole app · Task: rate helpfulness (accuracy, completeness, tone, usefulness)
No single formula captures "helpful" reliably → needs human judgment.
9Helpfulness Rubric (1–5)
| 5 | Accurate, complete, helpful, right tone |
| 3 | Partially helpful, missing aspects |
| 1 | Not helpful / incorrect |
10Human Eval Flow
- 50–100 question dataset
- Send to chatbot
- Collect answer
- Rate w/ rubric
- Average → overall score
11Types of Human Eval
- Direct Grading — score vs rubric
- Red Teaming — attack to find failures
- A/B Testing — compare live variants
- Golden Dataset — define labels/rubric
- Human-in-Loop — route ambiguous cases
12Human Eval Trade-offs
Disagreement signal: repeated grader disagreement → rubric is ambiguous.
+ High nuance/reliability − Costly − Doesn't scale to millions of outputs
13LLM-as-a-Judge
One LLM evaluates another's output — combines programmatic scale with human-like nuance.
Output → Judge LLM → Score + Reasoning
14Case: UPSC Auto-Grading
Target: answer-eval platform · Task: grade like human experts, cheaply, at scale
15Step 1 — Rubric
- Discusses ethical governance & accountability
- Explains the link
- Gives mechanisms
- Uses examples
- Balanced conclusion
16Step 2 — Golden Dataset
SME scores 50–100 student answers with same rubric → becomes reference/golden labels.
17Step 3 — Judge Inputs
Question
Max marks
Rubric
Student answer
Scoring instructions
No credit for verbosity/keyword-stuffing; concise justification required.
18Step 4 — Human vs LLM
| Answer | Human | LLM |
|---|
| A001 | 13 | 12 |
| A002 | 4 | 8 |
| A003 | 8 | 8 |
19MAE — Success Metric
(1/N) × Σ |Human − LLM|
MAE = 2.3 → judge off by ~2.3 marks avg. Goal: MAE → 0.
20Improve the Judge
Stronger judge LLM
Better prompt
Refine rubric
Clearer instructions
Calibration examples
Analyze disagreements
Loop: evaluate → find gaps → improve → re-evaluate.
21Reference-Based vs Free
Test: reference/correct answer exists per case? Yes → based. No → free.
Reference-BasedGround truth per case; compare vs reference. e.g. Recall@K, UPSC scores
Reference-FreeNo per-case answer; judge vs rubric. e.g. Chatbot helpfulness
22Offline vs Online
OfflineBefore production, fixed dataset, controlled. "Does it work on our eval set?"
OnlineIn production, real traffic & feedback. "How's it doing live?"
23Pick the Method
Code can measure?→Programmatic
Needs nuance?→Human
Needs scale + nuance?→LLM-as-Judge
24Mental Model
Pipeline = Target + Task + Criteria + Dataset + Method + Execution + Analysis
25Cheat Table
| Programmatic | Code executes eval |
| Human | Human judgment executes |
| LLM-Judge | LLM scores output |
| Recall@K | Relevant in Top-K ÷ Total |
| MAE | Avg |Human − LLM| score |
| Ref-Based | Ground truth exists |
| Ref-Free | Judge vs rubric only |
| Offline | Fixed dataset, pre-prod |
| Online | Live production signals |