1Offline Evaluation
Def: evaluation run before deployment, on a controlled golden dataset.
Dataset→App→Evaluator→Score→Release?
2Benefit: Release Gating
Score above threshold → deploy. Below → block / keep old version. Usable as a CI/CD gate.
3Benefit: Version Comparison
Same dataset + criteria across versions → fair comparison.
LLMsPromptsRerankers
Vector DBsArchitectures
4Benefit: Regression Testing
Checks that a fix elsewhere didn't break existing behavior.
Ex: kinder tone prompt → pricing answers lose precision.
5What Changes in Production
- Unexpected inputs — never in golden set
- Emergent failures — visible only at scale
- Drift — data/env changes over time
- Scale effects — concurrency, latency
6Unexpected Input Examples
- Mixed-language messages
- Ambiguous/incomplete Qs
- Angry msgs w/ hidden question
- Prompt-injection attempts
- Unanticipated edge cases
7Drift
Real-world data/behavior changes; frozen eval set stops representing production.
Risk: offline scores stay "good" while live UX degrades.
8Online Evaluation
Def: evaluates the deployed system using live traffic, continuously — usually with no fixed answer key.
Core question: "Is it behaving normally right now?"
9Offline vs Online
| Offline | Online |
|---|
| When | Pre-deploy | Post-deploy, continuous |
| Data | Fixed golden set | Live traffic |
| Answer key | Usually yes | Usually no |
| Use | Gate, compare, regress | Drift/anomaly detect |
10Complementary, Not Rivals
Offline: Correctness+Online: Normality=Continuous Eval
11Correctness vs Normality
CorrectnessOutput right vs a reference. Hard in prod — no per-case human label exists yet.
NormalityDoes behavior match the healthy baseline distribution?
12Distribution Shift Example
Scores usually cluster 500–700. Sudden shift to 800–900 → investigate — not automatically "wrong," just a trigger.
13Reference-Free Quality Estimates
- Faithfulness — answer supported by retrieved context?
- User feedback — thumbs-down spike = signal
14Captured Signals
Already available — log directly.
Thumbs up/downLatencyTokens
CostErrorsDuration
EscalationRepeated Qs
15Computed Signals
Need an evaluator to compute.
FaithfulnessAnswer relevanceCorrectness*
HallucinationToxicityBias/fairness
16Online Pipeline
Traffic→Logging→Signals→Dashboard/Eval→Alerts→Action
Captured: log → aggregate → alert.
Computed: log → sample → evaluator → aggregate → alert.
17Step 1 — Logging
| Identity | Conv/turn/user/session ID, timestamp |
| Input | User message |
| RAG | Retrieved context |
| Output | Model response |
| Ops | Latency, tokens, cost |
| Errors | Occurrence, status code |
| Behavior | Feedback, escalation, abandonment |
18Logging Properties
- Non-blocking — async, no added latency
- Durable & queryable — retrievable later
- Late signal attach — link via conversation ID
- PII handling — mask sensitive data
19Dashboarding
Aggregates signals over time windows (1h/24h/week). Aggregate trend > single conversation.
Ex: course launch → traffic spike → latency rises → dashboard flags it → scale response.
20Alerting
Automates threshold-violation detection since engineers can't watch dashboards 24/7.
SlackEmailPagerDuty-styleAPI
21Computed Eval: Hallucination
Trace→Sample→Ref-Free Evaluator→Metric→Dashboard→Alert
22Reference-Based vs Free
| Reference-Based | Reference-Free |
|---|
| Known answer key | No fixed answer for this case |
| Typical offline setup | Typical online setup |
| UPSC vs human grading | Answer supported by context? |
23LLM-as-Judge in Prod
Stronger LLM gets question + context + answer + rubric → scores hallucination/quality/safety.
Cost warning: judging every conversation is expensive → sample.
24Random Sampling
Randomly pick a subset to evaluate. Simple, but risky conversations may be under-represented.
25Stratified Sampling
Sample more from high-risk categories, not uniformly.
Thumbs-downAbrupt endEscalated
Repeated QsPricingRefundsAdmissions
Goal: maximize usefulness of limited eval capacity, not just minimize cost.
26Platform Example — LangSmith
Supports tracing, dashboards, alerts, datasets, offline experiments, online evaluators, LLM-judge templates, annotation.
PII LeakPrompt InjectionToxicity
BiasHallucinationCorrectnessAgent Eval
27Same Evaluator, Two Modes
On live traces→ Online evaluation
On dataset/experiment→ Offline evaluation
28Self-Improving Loop
Offline Set→Offline Eval→Release→Prod→Monitor→Failures→Dataset Update
Production failures get annotated & folded back into the golden dataset → next release tested against real + historical cases.
29Mental Model
Build→Offline Eval→Gate→Prod→Log→Monitor→Alert→Learn
30Cheat Table
| Offline | Pre-deploy, controlled, gate/compare/regress |
| Online | Post-deploy, live, drift/anomaly detect |
| Drift | Env/data changed vs frozen eval set |
| Captured | Logged directly (latency, cost...) |
| Computed | Needs evaluator (hallucination...) |
| Correctness | Right vs reference |
| Normality | Matches healthy baseline |
| Stratified sampling | Oversample high-risk convos |
| Self-improving loop | Prod failures → dataset → better releases |