Why LLM Applications Need Multiple Evaluation Pipelines

1. Previous Session Recap

Before starting the current topic, let us quickly recap what was covered in the previous LLM Evaluation session.

Model Evaluation vs Application Evaluation

2. Why Do We Need Multiple Evaluation Pipelines?

A common misconception is that if we evaluate the complete application once, that should be enough.

In reality, an LLM application is usually composed of multiple components that interact with each other. Each component can fail, the interaction between components can fail, and the complete application can fail to satisfy production requirements.

3. RAG Example: Understanding Failure Points

Consider a Retrieval-Augmented Generation (RAG) chatbot built for a company, school, or college.

Basic RAG Architecture

How the RAG Flow Works

1. A user submits a query.
2. The retriever receives the query.
3. The retriever searches the vector database.
4. Relevant documents are retrieved.
5. The query and retrieved context are passed to the generator.
6. The generator, usually an LLM, produces the final answer.

Two Obvious Failure Points

4. Component-Level Evaluation

Because individual components can fail, we need evaluation pipelines for individual components.

Retriever Evaluation

The primary responsibility of a retriever is:

Therefore, a retriever evaluation pipeline should check whether the documents returned for a given query are relevant.

Generator Evaluation

The generator receives a query and context and generates an answer.

One important quality dimension is faithfulness, also called groundedness.

Example of Groundedness

Suppose the user asks:

What is the duration of the Machine Learning course?

The retrieved document says:

The Machine Learning course duration is 3 weeks.

A grounded answer would be:

The Machine Learning course duration is 3 weeks.

The model should not add unrelated information such as:

It is a great course.
You can also purchase the Python course.
The Python course lasts 4 weeks.
        

Those additional statements are not supported by the provided context.

5. Workflow-Level Evaluation

Now comes the important question:

No.

The interaction between independently correct components can still produce an incorrect result.

The Key Idea

We therefore need to evaluate not only individual components, but also the workflow formed by those components.

6. RAG Failure Example: K = 5

Assume the retriever uses:

K = 5

Here, K means that the retriever returns the top 5 retrieved documents.

Retrieved Documents

Did the Retriever Do Its Job?

Yes.

The correct document was included within the top-5 results.

The retriever successfully retrieved the required information within its configured K results.

What Happens Next?

The query and all five retrieved documents are passed to the generator.

Suppose the system prompt tells the generator to prioritize higher-ranked documents.

In that case, the generator may focus more heavily on D1, D2, D3, and D4.

Imagine one of those documents says:

The Python course duration is 6 weeks.

The generator might incorrectly combine information and produce:

The Machine Learning course duration is 6 weeks.

Is the Final Answer Correct?

No.

Did the Generator Necessarily Fail?

Not necessarily.

The generator followed its instructions and used information provided in the retrieved context. It did not necessarily hallucinate the number from nowhere.

The problem was that the correct information was ranked too low, while incorrect or irrelevant information had higher priority.

Workflow-Level Evaluation Detects the Problem

A workflow-level evaluator can identify that the final answer is incorrect even though the individual component evaluations passed.

Possible Solution: Reranking

A reranker can reorder the retrieved documents after the initial retrieval step.

In this example, the reranker could recognize that D5 is the most relevant document and move it to the top.

7. Application-Level Evaluation

Now suppose we have three successful evaluations:

1. Retriever evaluation passes.
2. Generator evaluation passes.
3. Retriever + Generator workflow evaluation passes.

Does this guarantee that the application is production-ready?

No.

Example: Latency

Suppose the complete RAG pipeline takes:

10 seconds per user query

Technically, the system may produce correct answers.

However, if users have to wait 10 seconds for every response, the application may not provide an acceptable production experience.

Application-Level Metrics

• Overall latency
• Time to First Token (TTFT)
• Cost per request
• Token efficiency
• Error/failure rate
• Latency under load
• Overall reliability

8. Three Levels Where Failure Can Occur

An LLM-based application can have failure points at three broad levels.

9. Second Reason for Multiple Evals: Risk Categories

Multiple failure points are only one reason for having multiple evaluation pipelines.

The second major reason is:

For example, consider a RAG chatbot.

A generated answer should not only be correct. It should also be safe and operationally efficient.

10. Application Quality Risk Categories

Application quality focuses on whether the application actually performs its intended task well.

General LLM Application

Consider a text summarization application.

The user provides a long piece of text and the system generates a concise summary.

11. RAG-Specific Risk Categories

RAG systems introduce additional risks because they contain retrieval and context-generation workflows.

12. Agent-Specific Risk Categories

Agentic AI systems introduce additional evaluation dimensions because agents must decide what actions to take.

13. Multi-Turn Chatbot Risk Categories

A multi-turn chatbot has additional risks because conversations extend across multiple user interactions.

14. Safety Risk Categories

Correctness and helpfulness are not enough.

An LLM application must also be safe.

Example: Private Information Leakage

Imagine a chatbot gives a user another user's phone number, email address, credit-card information, or other private data.

Even if the information is factually correct, the application has failed its safety requirements.

Examples of Harmful Content Categories

• Self-harm related content
• Weapons-related content
• Illegal activity assistance
• Privacy violations
• Other unsafe or restricted content

15. Operational Risk Categories

Operational evaluation focuses on whether the system can run effectively in production.

16. Overall Evaluation Framework

We can now combine the entire discussion into one framework.

Level 1 — Components

Evaluate individual components.

Retriever
Reranker
Query Rewriter
Embedding Model
Vector Database
Generator
Output Parser
Tool Selector
Memory
Guardrails
System Prompt
        

Level 2 — Workflows

Evaluate how multiple components interact.

RAG Workflow
Retriever → Reranker → Generator

Agent Workflow
Planner → Tool Selection → Tool Execution → Observation → Next Action

Multi-Turn Workflow
User Turn → Memory → Model → Response → Next Turn
        

Level 3 — Complete Application

Evaluate the entire product from a production perspective.

User Request
      ↓
Complete Application
      ↓
Final Response
      ↓
Quality + Safety + Operations
        

Risk Categories Across the System

The Evaluation Pipeline Concept

Example: One RAG Application Can Have Multiple Evals

• Retriever relevance evaluation
• Retriever recall evaluation
• Reranker evaluation
• Generator groundedness evaluation
• Faithfulness evaluation
• Citation accuracy evaluation
• End-to-end RAG workflow evaluation
• Safety evaluation
• Latency evaluation
• Cost evaluation
• Error-rate evaluation

17. Final Summary

The Two Main Reasons

Most Important Concept

Think in Three Layers

┌─────────────────────────────────────────────┐
│             APPLICATION LEVEL               │
│                                             │
│ Quality + Safety + Operations               │
│                                             │
├─────────────────────────────────────────────┤
│              WORKFLOW LEVEL                 │
│                                             │
│ Component Interaction & End-to-End Behavior │
│                                             │
├─────────────────────────────────────────────┤
│             COMPONENT LEVEL                 │
│                                             │
│ Retriever / LLM / Tool / Memory / Parser   │
│                                             │
└─────────────────────────────────────────────┘
        

Think in Three Risk Categories

APPLICATION QUALITY
        ↓
Correctness
Accuracy
Relevance
Completeness
Instruction Following

SAFETY
        ↓
Toxicity
Harmful Content
Bias
Privacy / PII
Prompt Injection
Jailbreak Resistance

OPERATIONS
        ↓
Latency
Cost
Token Efficiency
Error Rate
Reliability
Performance Under Load
        

RAG Example in One Sentence

Final Mental Model

1. Find the failure points.
2. Identify the risk categories.
3. Create evaluations for individual components.
4. Evaluate the interaction between components.
5. Evaluate the complete application.
6. Continuously monitor quality, safety, and operations.