Offline vs Online Evaluation
Introduction. An LLM application does not become “safe and reliable” simply because it passes a test once. Before deployment, we need controlled evaluation against known examples. After deployment, the system is exposed to real users, unexpected inputs, changing data, traffic spikes, and new failure modes.
This page explains why offline evaluation and online evaluation solve different problems, how they work together, and how a production evaluation pipeline can be built using logging, signals, dashboards, alerts, evaluators, sampling, and continuous feedback.
At a Glance
Course Recap
The lecture begins by revisiting the main LLM evaluation topics covered earlier:
- Why do we need evaluations?
- What exactly are evaluations? — including model-based and application-based evaluation.
- What does an LLM evaluation pipeline look like?
- Why does one application need multiple evaluation pipelines? — because failures can occur at component, workflow, and application levels, and because quality, safety, and operational risks are different.
- Evaluation methods: programmatic evaluation, LLM-as-a-Judge, and human evaluation.
1. What Is Offline Evaluation?
Definition
Offline evaluation is evaluation performed on an LLM application before it is deployed to production.
Simple Explanation
You build the application, prepare a controlled evaluation dataset, run the evaluation pipeline, inspect the results, and decide whether the application is ready for release.
All the evaluation examples discussed in earlier sessions—such as the UPSC answer-grading application using a golden dataset and LLM-as-a-Judge—fall under offline evaluation because the system is tested before deployment.
Typical Offline Setup
Why Is It Necessary?
Deploying an untested LLM application is risky because production behavior is unknown. Offline evaluation provides a controlled way to test the application before exposing it to real users.
2. Three Major Benefits of Offline Evaluation
Pre-Release Testing
Offline evaluation lets you test an application before deployment and determine whether it meets the required quality threshold.
Release gate idea: an evaluation score can be used as a CI/CD gate. If the score is above a predefined threshold, deployment proceeds; otherwise, deployment is blocked or the previous version is retained.
Version Comparison
When multiple implementation choices exist, run the same evaluation pipeline on each version. Because the dataset and evaluation criteria remain constant, the scores can be compared fairly.
- Different LLMs
- Different prompts
- Different rerankers
- Different vector databases
- Different application architectures
Regression Testing
A change that improves one behavior can accidentally damage another. Regression testing checks whether existing capabilities remain healthy after a change.
For example, changing a chatbot prompt to make responses kinder may improve tone but could accidentally make exact pricing answers less precise. A golden dataset containing refund, pricing, curriculum, and other question types can expose this regression.
Regression Principle
If a refund-question success rate was approximately 90% before a change, a new version should not unexpectedly drop it to 80% merely because another behavior improved.
3. What Changes After Deployment?
Passing offline evaluation does not eliminate production risk. Once the application is live, it encounters a much larger and less predictable input space.
3.1 Unexpected Inputs
- Mixed Hindi-English conversations.
- Ambiguous or incomplete questions.
- Angry user messages containing a hidden question.
- Adversarial prompt-injection attempts.
- Edge cases that were not anticipated during test-set creation.
3.2 Emergent and Systematic Failures
Some failures appear only when thousands of users interact with the system. For example, a sudden increase in concurrent users may increase latency. A subtle bias may become visible only after a large number of conversations reveals a pattern across user groups.
3.3 Drift
Drift means that the data, behavior, or operating environment changes over time, making an older evaluation setup less representative of the current system.
Consider a RAG chatbot whose source documents contain course prices, curricula, and policies. If those documents change over a year but the golden dataset remains frozen, the evaluation set may no longer represent current production reality.
4. What Is Online Evaluation?
Definition
Online evaluation evaluates an LLM application using live production traffic after deployment, while real users are interacting with the system.
The Defining Characteristic
Online evaluation generally operates without a fixed golden answer key. Production users can ask questions that were never anticipated, so the evaluator often has to estimate quality using signals, reference-free evaluators, user feedback, distributions, and operational measurements.
5. Offline vs Online Evaluation
| Dimension | Offline Evaluation | Online Evaluation |
|---|---|---|
| When? | Before deployment | After deployment; continuously |
| Data | Fixed, controlled golden dataset | Live production traffic |
| Answer key | Usually available in reference-based evaluation | Usually unavailable for new production inputs |
| Inputs | Anticipated/selected cases | Anything real users send |
| Main use | Release gating, correctness, version comparison, regression testing | Drift detection, anomaly detection, production health |
| Speed & cost | Fast, relatively cheap, repeatable | Potentially expensive at production scale; sampling can reduce cost |
| Nature | Controlled experiment | Continuous observation |
They Are Complementary, Not Rivals
Online evaluation is not a replacement for offline evaluation. Both are required.
Correctness / Readiness
Production Normality
Offline: Is the application correct enough to release?
Online: Is the released application continuing to behave normally in production?
6. Correctness vs Normality
Correctness
Correctness asks whether the system's output is actually right according to an appropriate reference.
In the UPSC grading example, correctness could be measured by comparing the marks assigned by the automated grader with marks assigned by a human evaluator.
Why Correctness Is Hard in Production
Suppose a new answer arrives in production. The system evaluates it, but the human evaluator has not independently graded that same answer. Therefore, there is no human reference against which to directly compare the current output.
Normality
When direct correctness cannot be measured, production monitoring can ask whether the system's behavior resembles its established baseline.
For the UPSC example, store the scores produced over time and examine their distribution. If the distribution suddenly changes, that is a signal that something has changed and deserves investigation.
Distribution Example
Imagine previous weeks usually produced a score distribution concentrated around 500–700. If a new week suddenly produces many scores around 800–900, the system may be behaving differently.
This does not prove that the new scores are wrong. Students may genuinely have performed better. The distribution is an investigation trigger, not a correctness guarantee.
Other Ways to Estimate Quality Without a Golden Answer
Some metrics can be evaluated without knowing the exact correct answer. For a RAG chatbot, faithfulness can be estimated by checking whether the generated answer is supported by the retrieved context.
Another useful production signal is user feedback. If thumbs-down feedback suddenly increases, it can indicate a quality problem even though a complete answer key is unavailable.
7. Production Signals
The lecture divides production signals into two broad categories.
Captured Signals
- Thumbs-up / thumbs-down feedback
- Latency
- Token usage
- Cost
- Error information and status codes
- Conversation duration
- Escalation events
- Repeated/rephrased questions
Computed Signals
- Faithfulness
- Answer relevance
- Correctness, where an evaluation strategy permits it
- Hallucination
- Toxicity
- Bias and fairness
- Other quality, safety, and conversational metrics
8. Online Evaluation Pipeline
The core production pipeline discussed in the lecture can be represented as:
Two Branches
Captured metric: Log it → aggregate/display it → alert on abnormal behavior.
Computed metric: Log it → sample relevant traffic → run evaluator → aggregate metric → dashboard → alert.
9. Step 1 — Logging
Why Logging Comes First
If production conversations are not recorded, there is nothing to evaluate later. Logging creates a structured, retrievable record of production activity.
What Should Be Logged?
| Category | Examples |
|---|---|
| Identity & context | Conversation ID, turn ID, user ID, session ID, timestamp |
| User input | User question/message |
| RAG information | Retrieved context used to generate the answer |
| Output | Model-generated response |
| Operations | Latency, prompt tokens, completion tokens, total cost |
| Errors | Error occurrence and status code |
| Behavioral signals | Thumbs-up/down, escalation, repeated questions, conversation abandonment |
Engineering Properties of Logging
Logging should not unnecessarily increase user-facing latency. The conversation should continue while logging occurs asynchronously.
Stored traces should be reliably retained and retrievable for future investigation and evaluation.
Some signals arrive after the conversation ends. They should be linked back to the original conversation using identifiers such as conversation ID.
Personal Identifiable Information such as phone numbers, addresses, card numbers, dates of birth, or Aadhaar numbers should be masked or removed before long-term storage where appropriate.
10. Dashboarding and Alerting
Captured Quantity Flow
Dashboarding
A dashboard aggregates production signals over time windows such as the last hour, 24 hours, week, or longer periods.
The important point is that a single conversation in isolation is often less useful than the behavior of many conversations together. For example, one request may have unusually high latency, but if the average latency across thousands of recent conversations rises, that suggests a system-level issue.
Example: Traffic Spike
A new course launch causes hundreds or thousands of concurrent users to interact with the chatbot. Latency rises. The dashboard shows a sudden change from the previous baseline. The engineering team can respond by allocating additional compute resources, scaling instances, or adjusting traffic management.
Alerting
Engineers cannot watch dashboards continuously. Alerts automate the detection of threshold violations.
Example: If aggregated latency exceeds a chosen threshold for a defined time window, trigger an alert through a notification or incident-management channel.
The lecture references tools and integrations such as Slack, email, PagerDuty-style systems, and APIs as possible notification mechanisms.
11. Computed Evaluation in Production
Example: Hallucination Rate
Suppose we want to measure whether a production RAG chatbot is hallucinating.
Unlike latency, hallucination is not simply a value that can always be copied from the trace. An evaluator must inspect the conversation and compute a score.
Reference-Based vs Reference-Free
| Reference-Based | Reference-Free |
|---|---|
| Has a known reference/answer key. | No fixed answer key for the current production example. |
| Typical controlled offline evaluation. | Useful for many live production evaluations. |
| Example: compare automated UPSC grading with human grading. | Example: judge whether a generated answer is supported by retrieved context. |
LLM-as-a-Judge for Production Evaluation
A stronger LLM can act as an evaluator. It can receive the user's question, retrieved context, generated answer, and a detailed rubric, then estimate whether the response contains hallucination or other quality/safety issues.
12. Sampling to Control Evaluation Cost
When production volume is high, evaluating every conversation with an LLM-as-a-Judge can become expensive. Sampling reduces evaluation cost while still providing a useful view of production behavior.
Random Sampling
A simple approach is to randomly select a subset of conversations and evaluate only those.
Why Random Sampling May Not Be Enough
Not all conversations have equal risk. A random sample may contain many normal conversations and too few problematic ones.
Stratified Sampling
Stratified sampling divides conversations into meaningful categories and intentionally samples more from categories that are more likely to contain failures.
Potential high-value strata from the lecture
- Thumbs-down conversations
- Conversations that ended abruptly
- Escalated conversations
- Repeated or rephrased questions
- Pricing discussions
- Refund discussions
- Admission or fee discussions
- Other categories identified as high-risk
This increases the chance that the sampled set contains useful failure cases rather than mostly routine interactions.
13. Evaluation Platform Example — LangSmith
The lecture uses LangSmith as an example of an evaluation and observability platform that can support multiple parts of the workflow.
What the Platform Can Support
- Conversation tracing and logging
- Monitoring dashboards
- Alerts
- Datasets
- Offline experiments
- Online evaluators
- LLM-as-a-Judge evaluation templates
- Annotation and failure analysis
Different Evaluation Categories
The lecture highlights evaluators for areas such as:
Online vs Offline Evaluator Configuration
A key observation from the platform example is that the same type of evaluator can serve different purposes depending on what it runs against:
| Evaluator Input | Evaluation Mode |
|---|---|
| Production traces / live logged conversations | Online evaluation |
| Controlled dataset / experiment dataset | Offline evaluation |
This illustrates that a platform can support both offline and online evaluation workflows within the same overall evaluation system.
14. The Self-Improving Evaluation Loop
The most important system-level idea is that online and offline evaluation should continuously feed each other.
When a production conversation contains a meaningful failure, it can be added to the offline evaluation dataset. It may also be annotated with what went wrong and what the expected behavior should be.
Why This Matters
The offline dataset becomes more representative over time because it incorporates real production failures. Future releases are then tested against both historical cases and newly discovered failure modes.
Interview Preparation
Offline evaluation tests an LLM application before production deployment, usually against a controlled dataset and predefined evaluation criteria.
Pre-release testing/release gating, version comparison, and regression testing.
Online evaluation evaluates a deployed system using live production traffic and continuously monitors its behavior.
Offline tests use a controlled set of anticipated examples. Production contains unexpected inputs, scale-dependent failures, systematic patterns, and changing distributions.
Drift is a change in the real-world data, behavior, or environment that can make an older evaluation setup less representative of current production behavior.
No. They are complementary. Offline evaluation focuses on controlled correctness/readiness; online evaluation focuses on production behavior and normality.
A signal that is already available and can be recorded directly, such as latency, token usage, cost, or thumbs-up/down feedback.
A metric that requires an evaluator or additional computation, such as faithfulness, hallucination, toxicity, relevance, or bias.
Because evaluation requires production data. Without durable traces of conversations, inputs, outputs, context, and operational signals, there is nothing reliable to analyze later.
Logging should not unnecessarily increase the latency of the user-facing application.
Correctness asks whether an output is actually right according to a reference. Normality asks whether current production behavior resembles an established healthy baseline.
Because evaluating every production interaction with another LLM can be expensive. Sampling reduces cost while retaining a useful estimate of production behavior.
Because it deliberately samples more from high-risk categories, increasing the probability of finding meaningful failures.
Production failures discovered online are added or annotated in the offline dataset, making future offline tests more representative and improving subsequent releases.
Quick Revision
Offline
- Before deployment
- Controlled dataset
- Often reference-based
- Release gating
- Version comparison
- Regression testing
Online
- After deployment
- Live production traffic
- Often reference-free
- Drift detection
- Anomaly detection
- Production health
Captured Signals
- Latency
- Cost
- Tokens
- Errors
- User feedback
- Escalation
Computed Signals
- Faithfulness
- Hallucination
- Relevance
- Toxicity
- Bias/Fairness
- Other quality metrics
One-Minute Mental Model
Key Takeaways
- Offline evaluation is necessary before deployment.
- Offline evaluation supports release gates, version comparison, and regression testing.
- Production introduces risks that controlled offline tests cannot fully anticipate.
- Online evaluation observes live behavior after deployment.
- Online evaluation often works without a fixed answer key.
- Captured signals can be logged directly; computed signals require evaluators.
- Logging should be non-blocking, durable, queryable, support late signals, and protect PII.
- Dashboards reveal trends; alerts turn abnormal trends into actionable events.
- LLM-as-a-Judge can compute reference-free production metrics, but cost requires sampling.
- Stratified sampling can prioritize risky conversations.
- Online and offline evaluation should operate together in a continuous self-improving loop.