Offline vs Online Evaluation
Offline evaluation and online evaluation solve two different but complementary problems. Offline evaluation tests an application in a controlled environment before deployment. Online evaluation observes the application after deployment using real production traffic.
This distinction is essential for production-grade AI engineering. A system can perform extremely well on a golden dataset and still fail when thousands of real users introduce unexpected inputs, new usage patterns, operational load, or changing data.
This page builds the complete mental model: release gates, version comparison, regression testing, production risks, reference-free evaluation, logging, captured and computed signals, dashboards, alerting, sampling, LLM-as-a-Judge, and the feedback loop that continuously improves the offline dataset.
At a Glance
Course Recap
Before introducing offline and online evaluation, the lecture connects the topic to the earlier LLM evaluation framework.
1. What Is Offline Evaluation?
Definition
Offline evaluation is the evaluation of an LLM-based application before it is deployed to production, usually using a controlled evaluation dataset and predefined evaluation criteria.
Simple Explanation
Build the application → prepare test cases → run the evaluation pipeline → inspect the results → decide whether the application is ready for release.
Example: UPSC Answer Grader
The lecture uses an LLM application that evaluates UPSC Mains-style answers like a human evaluator. A golden dataset is created, an LLM-as-a-Judge approach is used, and the automated scores are compared with human evaluation.
Because this happens after building the software but before deployment, it is an example of offline evaluation.
2. Three Major Benefits of Offline Evaluation
Pre-Release Testing
You should not deploy an LLM application without first testing how it behaves against known cases. Offline evaluation gives a controlled way to determine whether the software is ready for production.
Release Gate
The evaluation score can become an automated CI/CD gate. For example, an organization might define a threshold of 95%.
- Score ≥ 95% → deployment pipeline can proceed.
- Score < 95% → deployment is blocked or the previous version is retained.
Version Comparison
When several implementation choices are available, run the same evaluation pipeline and same golden dataset against each version.
Because the evaluation conditions remain the same, the scores provide an objective basis for comparison.
Regression Testing
Regression testing checks whether a change that improves one behavior accidentally damages another existing behavior.
Imagine a CampusX chatbot that sounds too cold when students ask about refunds. You modify the system prompt to make the chatbot kinder and more polite. The new prompt may improve tone—but it might also make the chatbot less precise about prices.
If the old refund success rate was 90%, a healthy change should not unexpectedly reduce it to 80%.
Regression Test Matrix
| Change | Expected Improvement | What Must Also Be Checked? |
|---|---|---|
| System prompt | Better tone | Pricing accuracy, relevance, helpfulness |
| LLM model | Better overall quality | Latency, cost, safety, regression cases |
| Vector database | Better retrieval | Existing answer quality and latency |
| Reranker | Better context selection | Faithfulness and previously strong cases |
3. What Can Go Wrong After Deployment?
Passing offline evaluation does not mean that production risk has disappeared. Production exposes the system to a much larger and less predictable environment.
Users can ask questions that were never included in the golden dataset.
Some failures appear only at scale or after thousands of interactions.
The real-world data distribution and business context can change over time.
3.1 Unexpected Inputs
- Hindi-English mixed conversations even when testing focused mainly on English.
- Ambiguous or incomplete questions.
- Angry messages containing an underlying question.
- Adversarial prompt-injection attempts.
- Unexpected edge cases.
3.2 Emergent and Systematic Failures
Some problems are difficult to reproduce offline because they depend on scale.
For example, a new course launch may suddenly bring thousands of concurrent users. The application may experience a large latency increase that was not visible during offline evaluation.
Another example is a subtle bias that becomes visible only after thousands of conversations reveal a pattern—for instance, noticeably different behavior toward users from different backgrounds.
3.3 Drift
Drift occurs when the real-world data, usage patterns, business information, or operating environment changes over time, making the existing evaluation setup less representative of current production.
Suppose a RAG chatbot is initially evaluated using documents containing course prices, curriculum, and policies. A year later, prices, policies, and curriculum have changed. If the golden dataset is never updated, the evaluation pipeline may continue producing good scores even though users are dissatisfied with the current chatbot.
4. What Is Online Evaluation?
Online evaluation evaluates a deployed LLM application using live production traffic after deployment, while real users interact with the system.
The Biggest Difference
Online evaluation generally operates without a fixed golden answer key. The next user question is unknown, and the correct answer may not already exist in an evaluation dataset.
Core Purpose
Online evaluation asks: “Is the deployed application behaving normally in production right now?”
5. Offline vs Online Evaluation
| Dimension | Offline Evaluation | Online Evaluation |
|---|---|---|
| Timing | Before deployment | After deployment; continuously |
| Data | Fixed golden dataset | Live production traffic |
| Answer key | Usually available for reference-based evaluation | Usually unavailable for newly arriving inputs |
| Inputs | Anticipated and selected cases | Anything real users send |
| Best for | Release gating, version comparison, regression testing | Drift, surprises, emergent bugs, production health |
| Cost | Usually cheaper and repeatable | Can become expensive at scale |
| Speed | Fast and controlled | Continuous and potentially large-scale |
They Are Complementary
“Does my application work correctly on known evaluation cases?”
“Is my deployed application behaving normally in the real world?”
6. Correctness vs Normality
Correctness
Correctness asks whether the system's output is actually right according to an appropriate reference.
For the UPSC grader, correctness means checking whether the marks assigned by the LLM-based grader are close to the marks assigned by a human evaluator.
Why Direct Correctness Is Difficult in Production
After deployment, a new answer can arrive that has never been evaluated by a human. You know what the LLM produced, but you do not yet know what a human would have produced for that exact answer.
Therefore, you cannot directly calculate human-vs-model correctness for every new production interaction.
Alternative: Normality
When direct correctness is unavailable, compare current behavior with a known healthy baseline distribution.
Distribution Example
Suppose a UPSC grader historically produces the following score distribution:
Mostly 500–700
Similar pattern
Suddenly 800–900
The change does not prove that the new scores are incorrect. Students may genuinely have improved. But a large distribution shift is a signal that something changed and should be investigated.
Reference-Free Signals Can Help
Some quality metrics do not require knowing the exact correct answer. For example, faithfulness in a RAG system can be evaluated by checking whether the generated answer is supported by the retrieved context.
User Feedback as a Proxy
Suppose a chatbot suddenly receives a large number of thumbs-down ratings during the last hour. This does not directly prove every answer is incorrect, but it is a strong production-quality signal that something may have gone wrong.
7. Production Signals
The lecture divides online signals into two categories: captured signals and computed signals.
Captured Signals
Values already available from the application, infrastructure, model provider, or user. They can be recorded directly.
Computed Signals
Metrics that require an evaluator or additional computation.
8. Step 1 — Logging
Logging is the first step of online evaluation. If production conversations are not recorded, there is nothing reliable to evaluate later.
What Should Be Logged?
| Category | Examples |
|---|---|
| Conversation identity | Conversation ID, turn ID, user ID, session ID, timestamp |
| User input | The question/message sent by the user |
| RAG context | Retrieved context used to generate the response |
| Model output | The generated answer |
| Operational metrics | Latency, prompt tokens, completion tokens, total cost |
| Errors | Error information and status codes |
| User behavior | Thumbs up/down, escalation, repeated/rephrased questions, abandonment |
Example Logging Record
{
"conversation_id": "conv_123",
"turn_id": "turn_08",
"user_id": "user_456",
"timestamp": "2026-08-13T10:30:00Z",
"user_input": "...",
"retrieved_context": ["..."],
"model_output": "...",
"latency_ms": 2030,
"prompt_tokens": 850,
"completion_tokens": 220,
"cost": 0.012,
"status": "success",
"feedback": "thumbs_down"
}LangSmith as an Example
The lecture uses LangSmith as an example of a tool that can store traces containing user inputs, outputs, metadata, and other production information.
Engineering Properties of Good Logging
Logging should happen without unnecessarily increasing user-facing latency.
Stored traces should remain available and be retrievable for future evaluation and debugging.
Signals that arrive later—such as an email escalation the next day—should be attached to the original conversation using identifiers such as conversation ID.
Phone numbers, addresses, card numbers, dates of birth, Aadhaar numbers, and similar sensitive information should be masked or removed before long-term storage where appropriate.
9. Dashboarding & Alerting
Captured Quantity Flow
Dashboarding
Captured quantities such as latency can be sent directly to a monitoring dashboard. The dashboard aggregates values over time windows such as the last hour, 24 hours, week, or longer.
The important unit is usually not one isolated conversation but the aggregate behavior across many conversations.
Example: Sudden Traffic Spike
A new course launches and hundreds or thousands of users arrive simultaneously. Chatbot latency increases.
An engineering team might allocate additional compute, add instances, adjust load balancing, or otherwise manage traffic.
Alerting
Engineers cannot watch dashboards continuously. Alerts turn abnormal metrics into actionable notifications.
For example, an alert can be configured when a metric crosses a threshold during a defined time window and can notify a team through systems such as Slack, email, PagerDuty-style tools, or an internal API.
10. Computed Evaluation in Production
Computed metrics require an evaluator. A good example is hallucination rate.
Reference-Free Evaluation
A reference-free evaluation does not require a predefined golden answer for the current production input.
Hallucination Evaluation Pipeline
How the Evaluator Works
The evaluator can receive:
- The user's question.
- The retrieved context.
- The generated answer.
- A detailed rubric explaining how hallucination should be detected.
A stronger LLM can then act as a Judge and compute an evaluation score.
Reference-Based vs Reference-Free
| Reference-Based | Reference-Free |
|---|---|
| Known answer/reference exists. | No fixed answer key for the current input. |
| Common in controlled offline datasets. | Useful for many live production cases. |
| Example: automated UPSC marks vs human marks. | Example: checking whether an answer is supported by retrieved context. |
11. Sampling: Controlling Evaluation Cost
If an application receives a large number of production conversations, evaluating every conversation with an LLM-as-a-Judge may be too expensive.
Random Sampling
A basic strategy is to randomly select a subset of conversations and evaluate only that subset.
Why Random Sampling Is Not Always Optimal
Not all conversations are equally informative. Most production conversations may be routine, while a small subset contains the highest-risk behavior.
Stratified Sampling
Stratified sampling divides production conversations into meaningful categories and intentionally samples more heavily from high-risk categories.
Users explicitly indicate dissatisfaction.
Conversation ends abruptly.
User asks for human support.
User rephrases the same question repeatedly.
Pricing, fees, refunds, payments.
Other categories known to be sensitive or failure-prone.
Why Stratification Helps
If a sample contains more conversations that are likely to contain failures, the evaluator has a higher probability of discovering meaningful problems with the same evaluation budget.
12. Evaluation Platform Example — LangSmith
The lecture demonstrates how a platform such as LangSmith can support both online and offline evaluation workflows.
Types of Evaluators
How the Same Evaluator Becomes Online or Offline
| Evaluator Runs On | Mode | Meaning |
|---|---|---|
| Production traces | Online | The evaluator analyzes live logged interactions. |
| Dataset / experiment examples | Offline | The evaluator analyzes controlled evaluation data. |
This is a powerful conceptual point: the evaluator type alone does not determine whether the evaluation is online or offline; the data it runs against matters.
Typical Evaluator Configuration
- Select the application.
- Select the evaluator type, such as hallucination.
- Select an LLM to act as the Judge.
- Configure model/API settings.
- Define the rubric and instructions.
- Specify the required output format.
- Choose whether the evaluator runs on traces or a dataset.
13. The Self-Improving Offline ↔ Online Loop
One of the most important ideas in the lecture is that online and offline evaluation should remain continuously connected.
Step-by-Step
Start with representative evaluation cases.
Check correctness, quality, safety, and regression before release.
Release the version that satisfies the required evaluation gates.
Log traces, collect signals, run online evaluators, and monitor dashboards.
Find conversations where the system behaved incorrectly, unexpectedly, or unsafely.
Annotate the production conversation and make it part of future evaluation.
Run the next release against the enriched dataset and continue the cycle.
Why This Creates a Better System
The offline dataset becomes progressively more realistic because it contains real production failure cases. Instead of relying only on what engineers predicted users might ask, the evaluation suite gradually incorporates what users actually did.
Interview Questions
Evaluation of an LLM application before production deployment, typically using a controlled dataset and predefined evaluation criteria.
Pre-release testing/release gating, version comparison, and regression testing.
An automated condition in the deployment pipeline that allows or blocks a release based on evaluation results.
Testing whether a change improves the intended behavior without degrading previously working behaviors.
Unexpected inputs, emergent/systematic failures, and drift.
Continuous evaluation of a deployed application using live production traffic.
Because new production questions are unknown and generally do not have pre-existing golden answers.
Correctness asks whether the output is actually right; normality asks whether current production behavior is consistent with a healthy baseline.
Signals that can be recorded directly, such as latency, cost, token usage, errors, and user feedback.
Metrics requiring additional evaluation or computation, such as faithfulness, hallucination, toxicity, relevance, and bias.
Because production evaluation requires access to the conversations and signals generated in production.
So observability does not unnecessarily increase user-facing application latency.
Running an additional LLM evaluator over every production conversation can be expensive.
It intentionally prioritizes categories that are more likely to contain failures.
An evaluator running against production traces is an online evaluator; the same type of evaluator running against a controlled dataset is an offline evaluator.
Production failures are added or annotated in the offline dataset, making future offline evaluation more representative.
Quick Revision
Offline Evaluation
- Before deployment
- Golden dataset
- Controlled inputs
- Usually reference-based
- Release gating
- Version comparison
- Regression testing
Online Evaluation
- After deployment
- Live traffic
- Unexpected inputs
- Often reference-free
- Drift detection
- Normality monitoring
- Production risk detection
Captured Signals
- Latency
- Tokens
- Cost
- Errors
- Thumbs up/down
- Escalations
- Repeated questions
Computed Signals
- Faithfulness
- Relevance
- Hallucination
- Correctness estimates
- Toxicity
- Bias & fairness
One-Minute Mental Model
Key Takeaways
- Offline evaluation is mandatory before deploying an LLM application.
- Its three major benefits are release testing, version comparison, and regression testing.
- Production creates risks that a fixed golden dataset cannot fully predict.
- Online evaluation continuously monitors the deployed application using live traffic.
- Online evaluation often has no fixed answer key, so reference-free metrics and production signals become important.
- Correctness and normality are different concepts.
- Captured signals are recorded directly; computed signals require evaluators.
- Logging should be non-blocking, durable, queryable, capable of attaching late signals, and privacy-aware.
- Dashboards reveal aggregate behavior; alerts turn abnormal behavior into action.
- LLM-as-a-Judge can evaluate production traces, but sampling controls evaluation cost.
- Stratified sampling focuses evaluation capacity on high-risk conversations.
- Online and offline evaluation are complementary—not competing approaches.
- Production failures should become future offline evaluation cases.
Final Mental Model
A production-grade AI engineer does not stop at “I built the chatbot.” The complete mindset is:
Build → Evaluate → Release → Observe → Detect → Learn → Update Dataset → Re-evaluate → Release Again.
This is the shift from simply building an LLM application to engineering an LLM system that can operate reliably at real-world scale.