Offline vs Online Evaluation

1Offline Evaluation

Def: eval before deployment, on a controlled golden dataset + predefined criteria.

Dataset→App→Evaluator→Score→Release?

2UPSC Grader Example

Golden dataset + LLM-as-Judge scores compared to human scores, before deployment → classic offline eval.

3Benefit: Release Gate

Threshold e.g. 95%. Score ≥ threshold → deploy. Below → block / keep old version. Automatable in CI/CD.

4Benefit: Version Comparison

Same dataset + same pipeline across versions → fair comparison.

ModelsPromptsRerankers Vector DBsArchitectures

5Benefit: Regression Testing

Improving one behavior shouldn't quietly break another.

Ex: kinder tone prompt → refund success drops 90%→80%.

6Regression Test Matrix

Change	Also Check
System prompt	Pricing accuracy, relevance
LLM model	Latency, cost, safety
Vector DB	Answer quality, latency
Reranker	Faithfulness, prior cases

7What Changes in Prod

Unexpected inputs — never in golden set
Emergent failures — visible only at scale
Drift — data/env changes over time

8Unexpected Input Examples

Hindi-English mixed msgs
Ambiguous/incomplete Qs
Angry msg w/ hidden question
Prompt-injection attempts
Unanticipated edge cases

9Emergent Failures

Course launch → thousands of concurrent users → latency spike never seen offline. Bias patterns visible only across thousands of convos.

10Drift

Real-world data/behavior/env changes; frozen eval set stops representing production.

Risk: stale golden dataset → good offline score, unhappy live users.

11Online Evaluation

Def: evaluates deployed system on live traffic, continuously — usually no fixed answer key.

Core Q: "Is it behaving normally right now?"

12Offline vs Online

	Offline	Online
When	Pre-deploy	Post-deploy, continuous
Data	Fixed golden set	Live traffic
Answer key	Usually yes	Usually no
Best for	Gate/compare/regress	Drift/anomaly detect
Cost	Cheap, repeatable	Can get expensive

13Complementary, Not Rivals

Offline: Correctness+Online: Normality

Production-grade systems need both — online is not a replacement.

14Correctness vs Normality

CorrectnessOutput right vs a reference. Hard in prod — no per-case human label yet.

NormalityDoes behavior match healthy baseline?

15Distribution Shift Example

Scores usually 500–700. Week 5 suddenly 800–900 → not proof of error, but a trigger to investigate.

16Reference-Free Signals

Faithfulness — answer supported by context?
Thumbs-down spike — quality proxy, not proof

17Captured Signals

Already available — log directly.

Thumbs up/downLatencyTokens CostErrors

18Computed Signals

Need an evaluator.

FaithfulnessRelevanceCorrectness HallucinationToxicityBias

19Step 1 — Logging

Why first: no recorded conversations → nothing to evaluate later.

Identity	Conv/turn/user/session ID, ts
Input	User message
RAG	Retrieved context
Output	Model response
Ops	Latency, tokens, cost
Behavior	Feedback, escalation

20Example Log Record

{
"conversation_id": "conv_123",
"latency_ms": 2030,
"prompt_tokens": 850,
"cost": 0.012,
"status": "success",
"feedback": "thumbs_down"
}

Platform example: LangSmith stores traces w/ inputs, outputs, metadata.

21Logging Properties

Non-blocking — async, no added latency
Durable & queryable — retrievable later
Late signal attach — via conversation ID
PII handling — mask sensitive data

22Dashboarding

Aggregates signals over windows (1h/24h/week). Aggregate trend > single conversation.

Log→Dashboard→Alert→Action

23Traffic Spike Example

Course launch→Latency ↑→Dashboard flags→Scale resources

24Alerting

Automates threshold-violation detection since engineers can't watch dashboards 24/7.

SlackEmailPagerDuty-styleAPI

25Computed Eval: Hallucination

Trace→Sample→LLM Judge→Score→Dashboard→Alert

Judge inputs: question, context, answer, rubric.

26Reference-Based vs Free

Reference-Based	Reference-Free
Known answer key	No fixed answer for this case
Typical offline setup	Typical online setup
UPSC vs human grading	Answer supported by context?

27Cost Warning

Judge cost: running an LLM judge on every production conversation gets expensive fast → sample.

28Random Sampling

Randomly pick a subset. Simple, but risky conversations may be under-represented.

29Stratified Sampling

Sample more from high-risk categories, not uniformly.

Thumbs-downAbandonedEscalated Repeated QsFinancial topics

Rule: sample for information value, not convenience.

30Platform — LangSmith

Supports tracing, dashboards, alerts, datasets, offline experiments, online evaluators, judge templates.

PII LeakPrompt Inj.Toxicity BiasHallucinationCorrectnessAgent Eval

31Same Evaluator, Two Modes

On live traces→ Online

On dataset/experiment→ Offline

Evaluator type ≠ mode; the data it runs on decides the mode.

32Evaluator Config Steps

Select application
Select evaluator type
Pick Judge LLM
Configure model/API
Define rubric
Set output format
Run on traces or dataset

33Self-Improving Loop

Dataset→Offline Eval→Release→Prod→Monitor→Failure→Dataset Update

Real failures get annotated & folded back → next release tested against real + historical cases.

34Final Mental Model

Build→Eval→Release→Observe→Detect→Learn→Update→Re-eval

35Cheat Table

Offline	Pre-deploy, controlled, gate/compare/regress
Online	Post-deploy, live, drift/anomaly detect
Drift	Env/data changed vs frozen eval set
Captured	Logged directly (latency, cost…)
Computed	Needs evaluator (hallucination…)
Correctness	Right vs reference
Normality	Matches healthy baseline
Stratified sampling	Oversample high-risk convos
Self-improving loop	Prod failures → dataset → better releases

Offline vs Online Evaluation — Cheat Sheet