1Model Eval vs App Eval
Model EvalTests the LLM's raw capabilities
App EvalTests the LLM-powered app (RAG, agent...)
2Why AI Engineers Need It
- Compare & select the right model
- Track if a new model release is worth migrating to
- Assess safety — hallucination, jailbreaks
- API vs self-host — evidence-based call
Without evals: model selection is guesswork.
3Definition
Model Eval: systematic measurement of a model's capabilities, behavior, reliability & operational traits, under controlled conditions.
No single "LLM IQ" — too many distinct capabilities to reduce to one score.
44-Step Evaluation Process
1. Select capability→2. Bring a test→3. Fixed protocol→4. Score & interpret
5Benchmark vs Custom Eval
| Benchmark | Custom Eval |
|---|
| What | Standardized, shared test | Built from your actual task/data |
| Goal | General capability compare | Application-specific suitability |
| Answers | "How capable generally?" | "Best for our workload?" |
| Ex. | MMLU, SWE-bench | Labeled historical app data |
6Case Study — Email Router
Route emails into Billing/Technical/Refund. Model A: top-leaderboard, $15/1M tok. Model B: mid-table, $0.50/1M tok.
7Case Study — Results
| Model A | Model B |
|---|
| Class. acc. | 94% | 91% |
| Urgency acc. | 88% | 87% |
| Latency | ~4.1s | ~9s |
Verdict: B nearly matches A at a fraction of cost → better value for this workload, despite losing on public benchmarks.
8Golden Dataset for Custom Eval
~200–500 historical emails, labeled by target category → evaluate both models on the real task.
9The 8 Core Capabilities
01Knowledge & ReasoningFacts + multistep logic
02Coding & SWEGen, debug, multi-file, tools
03MathematicsSymbolic + numerical reasoning
04Long ContextEffective use of huge inputs
05Vision & MultimodalImages, video, cross-modal
06Agentic & Tool UseBrowsing, APIs, actions
07Safety & AlignmentHarm resistance, truthfulness
08Instruction FollowingConstraints, format, style
10Cap 1 — Knowledge & Reasoning
Factual recall (biology, physics, history) + multistep logical reasoning.
Benchmark: MMLU — 57 subjects.
11Cap 2 — Coding & SWE
- Gen code from NL requirements
- Write tests, fix from failures
- Find/fix bugs in existing code
- Multi-file, long-horizon tasks
- CLI ops, package install, env setup
- API/function-call use
12Cap 3 — Mathematics
Grade-schoolCompetition-level
UndergraduateResearch-level
Relevant to sci-computing, finance, engineering sims.
13Cap 4 — Long Context
- Retrieve small fact from huge input
- Find entity details in big doc
- Summarize very large inputs
- Hold large codebase context (agent)
Note: stated context window ≠ effective use of it.
14Cap 5 — Vision & Multimodal
Image + video understanding, cross-modal reasoning.
Ex: "what can I cook from what's in this fridge photo?"
15Cap 6 — Agentic & Tool Use
Goal→LLM Reasoning→Tool Call→Action→Result
Web browsing, structured calls, APIs, desktop interaction.
16Cap 7 — Safety & Alignment
- Harmful-content resistance
- Adversarial-attack resistance
- Truthfulness vs sycophancy
- Cybersecurity, crypto, forensics
Why labs care: regulatory + reputational risk.
17Cap 8 — Instruction Following
- Follows requested format (bullets, etc.)
- Respects length/word limits
- Matches requested tone/style
- Asks clarifying Qs when ambiguous
Impact: directly drives user trust in product.
18Final Comparison
| Benchmark | Custom Eval |
|---|
| Design | Standardized/shared | Around your task |
| Comparability | High across models | High in your setup |
| Best Q | "How good generally?" | "Best for our workload?" |
19Quick Facts
2 test types4 eval steps8 capabilitiesMMLU = 57 subjects
20Mental Model
Don't ask "which model is strongest?" Ask: "which performs well on what my app needs, within my cost/latency constraints?"
Rule: Benchmark = general capability. Custom eval = final decision.
21Cheat Table
| Model Eval | Tests the LLM itself |
| App Eval | Tests the full application |
| Benchmark | Standardized, cross-model |
| Custom Eval | Task-specific, your data |
| Fixed protocol | Ensures repeatable comparison |
| Long context ≠ window size | Effective use matters, not just size |
| Instr. following | Format/length/tone/clarify |