1Core Definition
LLM evals: systematic, repeatable tests used to judge an LLM and LLM-powered systems against clear criteria.
Applied either to the LLM itself, or to LLM-based applications.
2Three Core Characteristics
- Systematic — not "vibe testing" 5 random Qs; use proper datasets covering edge cases (e.g. 100 real user chats → dataset).
- Repeatable — same eval setup re-applied after changing prompt/model/retriever/chunking/embeddings/reranker → v1 vs v2 comparison.
- Clear criteria — define "good" explicitly (correct, simple, from course content, safe, no abuse/threats).
No criteria = vibe testing. Clear criteria = proper evaluation.
3Eval ≠ Just a Metric
ML mindset: eval = Accuracy / Precision / Recall / F1. Incomplete for LLMs.
LLM Eval = the complete testing setup, not one number.
- What's being evaluated / which component?
- What dataset & criteria?
- Offline or post-deployment?
- Which tools (e.g. Ragas for RAG)?
- How are results interpreted?
4Practical Questions Evals Answer
- Model selection — usable for this task?
- Prod readiness — good enough to ship?
- Prompt opt. — did v2 beat v1?
- RAG groundedness — grounded in context?
- Agent perf. — completing tasks correctly?
- Safety — safe for real users?
- Performance — latency under control?
5Two Types of Evals
LLM Evaluations→
Model Evals
App Evals
Model Evals → evaluate the LLM itself. App Evals → evaluate the LLM-based application.
Disclaimer: not strict official terms — industry often says "LLM evals" for both, context disambiguates.
6Model Evals
Def: tests & evaluations of an LLM's capabilities.
On every new release: what can it do, how capable, vs. other models, strong/weak benchmarks — documented & published.
78 Capability Categories
01Reasoning
02Knowledge
03Math
04Coding
05Instr. Following
06Long-Context
07Multimodal
08Tool Use
Knowledge ties to cutoff date — info after cutoff isn't in pretrained knowledge.
8Reference Benchmarks
| Knowledge/Reasoning | MMLU |
| Math | GSM8K |
| Coding/SWE | SWE-bench |
| Instr. Following | IFEval |
| Long Context | Needle in a Haystack |
| Multimodal | MMMU |
Capability→
Benchmark→
Results→
Comparison
9Model-Eval Literacy for Engineers
Frontier-scale benchmarking is done by big labs, not you. Your job:
- Know what model eval / benchmarks are
- Read benchmark results
- Know model strengths/weaknesses
- Use it to pick: OpenAI vs Anthropic vs open-source
10Application Evals
Def: assess behavior/performance of an LLM-powered app — at the system level or a specific component.
Myth: "LLM = brain → app quality = model quality." False — many components sit around the LLM:
UI
System prompt
Tools/APIs
Orchestration
Guardrails
Output parsers
Memory/context
Retrieval/embeddings
Vector DB/reranker
Monitoring
Feedback loops
11Smartphone Analogy
A benchmark-topping chip (Snapdragon/MediaTek) ≠ a great phone — camera, OS, battery, display, audio all matter.
Powerful Processor≠
Excellent Phone
Powerful LLM≠
Excellent App
Model eval = labs' job. Evaluating the system built around it = your job.
12Component vs. System Level
ComponentRetriever, embedding model, reranker, prompt, guardrails, output parser, tool calling, memory.
SystemFinal response quality, correctness, groundedness, safety, latency, cost, UX.
13The Central Question Flips
Model Eval"Can the model do this?" — reason, code, math, follow instructions, long context, tools.
App Eval"Will our product/system work correctly?"
14Worked Example — Campus X Chatbot
Model eval asks: Can the model reason / follow instructions / do math / long-context / use tools?
App eval asks: Correct answer? Right course material used? Grounded? Beginner-friendly? Hallucinated? Safe? Fast? Reliable?
15Why App Evals Matter Most to You
Your job ≠ building the next frontier LLM. It's building RAG systems, chatbots, agents, LLM APIs, doc-QA, assistants — on top of existing models.
Model eval tells you model capability. App eval tells you if the product you built actually works.
16Course Focus & Rule of Thumb
Course weight: heavily Application Evals. Model-eval literacy still needed for benchmark reading & model selection.
Rule: when future content says "LLM Evaluation," check context — it usually means application-level evaluation.
17Four-Line Mental Model
- Evals are systematic, repeatable tests against clear criteria.
- Evals ≠ metrics — they're the complete testing setup.
- Model Evals → how capable the LLM is.
- Application Evals → whether the product actually works.
18What's Next
Why?→
What?→
Types→
How?
Next: how evaluations are actually performed — taught from the Application Eval perspective.