1Core Definition
LLM evals: systematic, repeatable tests used to judge an LLM or LLM-powered system against clear criteria.
Not random prompts — a structured testing system.
2Three Core Properties
- Systematic — proper dataset, not a few manual prompts; covers normal, hard & edge cases.
- Repeatable — same dataset re-run across prompt/model/retriever/chunking changes → comparable scores.
- Clear criteria — explicit definition of "good" (correct, simple, grounded, safe, right tone).
Dataset→
v1→
Score→
v2→
Compare
3Evals ≠ Just Metrics
Classic ML mindset: eval = accuracy/precision/recall/F1/ROC-AUC. Incomplete for LLMs.
An LLM eval = the entire testing setup, not one number.
- What component is tested?
- What dataset / criteria / method?
- What tools?
- When — offline or production?
4Purpose: Answer Real Questions
- Can we use this model?
- Is it ready to ship?
- Did prompt v2 beat v1?
- Is the RAG answer grounded?
- Is the agent completing tasks correctly?
- Is the system safe & fast enough?
5Two Types of Evals
| Model Evals | App Evals |
|---|
| Tests | The LLM itself | Full app / components |
| Main Q | "Can the model do this?" | "Will our product work?" |
| Owner | Frontier labs | AI/app engineers |
| Tools | Benchmarks, leaderboards | Custom datasets, rubrics |
Not strict industry terms — often both just called "LLM evals," context disambiguates.
6Model Evals
Standardized benchmark testing whenever a new LLM ships — published so the industry can compare capability.
New LLM→
Benchmarks→
Scores→
Comparison
78 Capability Areas
01Reasoning
02Knowledge
03Math
04Coding
05Instr. Following
06Long-Context
07Multimodal
08Tool Use
8Reference Benchmarks
| Knowledge/Reasoning | MMLU |
| Math | GSM8K |
| Coding | SWE-bench, HumanEval |
| Instr. Following | IFEval |
| Long Context | Needle in a Haystack |
| Multimodal | MMMU / "Triple-MU" |
Examples only — the real benchmark landscape is much larger.
9Model Evals for AI Engineers
You rarely build frontier benchmarks — but you must be able to read them:
- What benchmarks measure
- How to interpret results
- How results guide model selection (proprietary vs. open-source)
10Application Evals
Def: assess an LLM app's behavior/performance — at the full-system level or per-component.
Myth: "LLM is the brain → good LLM = good app." False — the app has many more parts:
UI
System prompt
Tools/APIs
Orchestration
Guardrails
Output parsers
Memory
RAG stack
Monitoring
Feedback loop
11Smartphone Analogy
A great chip (Snapdragon/MediaTek) ≠ a great phone. Camera, OS, battery, display, audio, software all matter too.
Model Eval+
App Components+
App Eval→
Reliable Product
12System-Level vs Component-Level
SystemFull app — is the final chatbot reply correct, grounded, safe, fast?
ComponentOne part — retriever, embedding model, reranker, etc.
RAG example: eval retriever, embeddings, reranker, context relevance, correctness, groundedness, latency, cost, safety — separately.
13The Key Question Flips
Model Eval asks"Can the model do this?"
App Eval asks"Will our product work correctly?"
14Questions App Evals Answer
Correctness
Groundedness
Completeness
Clarity
Safety
Latency
Cost
Task completion
15Good Model ≠ Good App
A model can top every benchmark and the app can still fail from:
- Bad prompts / chunking / embeddings
- Poor retrieval or orchestration
- Tool failures, hallucinations
- Unsafe output, high latency/cost
16Key Takeaways
- Evals = complete testing setup, not just a metric.
- Must be systematic, repeatable, criteria-driven.
- Model evals → model capability; App evals → product reliability.
- Model evals lean on standardized benchmarks.
- 8 capability areas: reasoning, knowledge, math, coding, instruction following, long-context, multimodal, tool use.
- AI engineers must read benchmarks, rarely build them.
- Application eval = the core day-to-day focus for AI engineers.
- An LLM is just one component of the full application.
- App evals run at both system and component level.
17What's Next
Why?→
What?→
Model Evals→
App Evals→
How?
Next: How application evaluations are actually designed and executed.