1What Is an AI Engineer?
Def: Builds applications/products on top of foundation models (LLMs), rather than training models from scratch.
Core topics: LLM apps, RAG, AI agents, prompt engineering, evaluation, orchestration.
2What Is LLM Evaluation?
Def: Systematic process to measure how well an LLM / LLM app performs against defined expectations.
Not just: "Can I build this?"
Also: "Is it good, safe, reliable & predictable enough to ship?"
3Why It Matters
Default dev loop (risky): build → ask a few Qs → looks good → assume it works → deploy.
- Career edge — less commonly studied than app-building, so it's a differentiator.
- Mindset shift — "I built a project" → "Can this serve millions reliably?"
4Vibe Testing
Def: Judging an LLM app on a handful of prompts using gut feeling, not systematic metrics.
Pros: simple, fast, fine for early experiments.
- Informal & subjective
- Not repeatable
- Different devs → different conclusions
- Doesn't cover the real input space
OK for prototyping — never for production evidence.
5Real-World Failures
Air Canada Chatbot
Gave wrong bereavement-fare policy; airline held legally responsible.
Lesson: a chatbot's mistakes are the company's liability.
Chevrolet Dealer Bot
Jailbroken into "agreeing" to sell a car for $1; went viral.
Lesson: test against adversarial, not just normal, inputs.
Fabricated Legal Cases
Lawyer used unverified LLM output (fake case citations) in court.
Lesson: confident output ≠ correct output.
Common thread: systems trusted without evaluation/safeguards.
6Traditional vs. LLM Testing
| Traditional SW | LLM Apps |
|---|
| Behavior | Deterministic | Probabilistic |
| Same input | Same output | Output can vary |
| Check | Exact match (2+2=4) | Semantic validity, many valid answers |
⚠ Eval can't rely on exact string matching.
7Evaluation Dimensions
| Factuality | Is it factually correct? |
| Completeness | Covers all needed info? |
| Tonality | Right tone / style? |
| Groundedness | Backed by source/context? |
| Latency | Response speed |
| Cost | $ per response |
Relevant dimensions vary by app — support bot ≠ medical assistant ≠ coding agent.
8Two Core Challenges
- Non-determinism — same input, different valid outputs; exact-match fails.
- Multi-dimensionality — correct but incomplete / ungrounded / slow / costly / wrong tone.
910-Step Learning Roadmap
1What are Evals?
2Eval Landscape
3Model Eval
4App Eval
5Eval Pipeline
6RAG Eval
7Agent Eval
8Safety Eval
9Ops Eval
10Prod Monitoring
10Eval Doesn't End at Deploy
Myth: evaluation = pre-launch only. Reality: continuous production monitoring is required.
- Latency — response speed
- TTFT — time to first token
- TPS — tokens per second
- System load — behavior under real traffic
11Mental Model Shift
"Can I build it?"
↓
"Can I prove it behaves correctly & reliably?"
↓
"Can it safely serve millions of real users?"
12Core Principle
Build → Evaluate → Improve → Deploy → Monitor → Evaluate Again
A permanent loop, not a one-time checkbox.
13Key Takeaways
- Building ≠ working reliably — you must evaluate too.
- Vibe testing ≠ production evidence.
- LLMs are probabilistic, not deterministic.
- Evaluation is multidimensional, not just "correctness."
- Real failures → financial / legal / reputational damage.
- No universal metric — criteria depend on the app.
- Evaluation continues after deployment (monitoring).
- Eval literacy = a real AI engineering skill & edge.