LLM Evaluation Notes (Session 3–4)
Offline Evaluations vs Online Evaluations
Production-Ready Notes for LLM Engineers
1. Recap of Previous Sessions
Before understanding Offline vs Online Evaluations, let's quickly revise everything covered so far.
The course has already introduced five major concepts.
1. Why Do We Need Evaluations?
LLMs are probabilistic systems.
Unlike traditional software,
Input
↓
Fixed Logic
↓
Same Output
LLMs behave like:
Input
↓
Reasoning
↓
Probabilistic Output
Even with the same prompt, outputs may vary.
Therefore, instead of testing code, we must evaluate behavior.
2. Types of Evaluations
Two broad categories were introduced.
Model-Level Evaluations
Evaluate the model itself.
Examples:
- MMLU
- GSM8K
- HellaSwag
- HumanEval
These answer:
"How good is GPT-4 compared to Claude?"
Application-Level Evaluations
Evaluate the entire application.
Example:
User
↓
Retriever
↓
Prompt
↓
LLM
↓
Post Processing
↓
Final Response
Question becomes:
Is my chatbot actually working?
3. LLM Evaluation Pipeline
Every evaluation pipeline follows roughly this pattern.
Dataset
↓
Run Application
↓
Collect Outputs
↓
Evaluate
↓
Metrics
↓
Decision
4. Why Multiple Evaluation Pipelines?
One evaluation is never enough.
Why?
Because LLM applications fail in multiple places.
Example:
User
↓
Retriever
↓
Prompt
↓
LLM
↓
Output Formatter
Failures may occur at:
- Retrieval
- Prompt
- Model
- Agent
- Final Output
Each component requires separate evaluation.
Another reason:
Different risk categories exist.
Quality
Safety
Latency
Cost
Bias
Hallucination
Reliability
Each requires a different evaluation pipeline.
5. Evaluation Methods
Three major approaches.
Programmatic Evaluation
Rule-based.
Example
Expected Output == Actual Output
or
Regex
Keyword matching
JSON validation
LLM-as-a-Judge
Another LLM evaluates responses.
Example
Question
Reference
Generated Answer
↓
GPT-4 Judge
↓
Score
Human Evaluation
Humans review outputs.
Best quality
Most expensive
Usually reserved for critical applications.
Today's Topic
Offline Evaluations vs Online Evaluations
This is one of the most important concepts in production LLM engineering.
Almost every real company runs both.
PART 1 — Offline Evaluations
Definition
Offline evaluation means:
Evaluating an LLM application before deployment.
Everything happens before real users interact with the application.
Imagine this pipeline.
Developer
↓
Build Application
↓
Run Evaluation
↓
Pass?
↓
Deploy
If evaluation fails,
deployment stops.
Think of Offline Evaluation as
Final exam before graduation.
Example
Suppose you built an AI UPSC answer grader.
Before deployment you have
1000 Answers
↓
Human Marks
↓
Golden Dataset
Now your grading model grades them.
Compare
Human Marks
vs
Model Marks
If close enough,
the system passes.
This entire process is an Offline Evaluation.
Why Offline Evaluations Matter
Three major reasons.
1. Release Gating
This is the biggest use case.
Before every deployment,
run evaluations automatically.
Example CI/CD pipeline.
Developer pushes code
↓
GitHub
↓
CI/CD Pipeline
↓
Run Offline Evaluations
↓
Score > Threshold ?
YES
↓
Deploy
NO
↓
Reject Deployment
Suppose your quality threshold is
Faithfulness > 90%
Correctness > 92%
Latency < 2 sec
If any metric fails,
deployment never happens.
This is called
Release Gating
Benefits
- Prevents broken releases
- Fully automated
- Easy to integrate into CI/CD
- Reduces production incidents
2. Version Comparison
Imagine testing multiple versions.
Example
Version A
GPT-4
Version B
Claude
Version C
Gemini
How do you know which is better?
Run the same golden dataset on every version.
Golden Dataset
↓
GPT-4
↓
Metrics
-------------------
Golden Dataset
↓
Claude
↓
Metrics
-------------------
Golden Dataset
↓
Gemini
↓
Metrics
Now comparison is fair because every model saw identical questions.
This isn't limited to models.
You can compare:
- prompts
- embeddings
- vector databases
- rerankers
- retrieval pipelines
- chunking strategies
- agent workflows
- system architectures
Offline evaluations make A/B comparisons objective.
3. Regression Testing
This is extremely important.
Suppose customers complain:
"Bot sounds rude."
You improve prompts.
Now bot becomes very polite.
Good?
Maybe not.
Suppose original answer:
Course costs ₹19,500.
After update:
The course costs around ₹19,000.
The tone improved,
but factual precision got worse.
This is called
Regression
Improving one feature accidentally breaks another.
Offline evaluation prevents this.
Golden dataset contains many categories.
Example
Refund Questions
Pricing Questions
Admissions
Course Details
Faculty
Technical Issues
After changing prompts,
run evaluations on every category.
If pricing accuracy suddenly drops,
regression is detected.
Summary of Offline Evaluation Benefits
| Benefit | Purpose |
|---|---|
| Release Gating | Stop bad deployments |
| Version Comparison | Compare prompts/models fairly |
| Regression Testing | Ensure improvements don't break other features |
Why Offline Evaluation Is Not Enough
Even if offline scores are perfect,
production is different.
Three major risks appear only after deployment.
Risk 1 — Unanticipated Inputs
Golden datasets are limited.
Real users are not.
Example:
Golden Dataset
What is the refund policy?
Production User
Bhai refund kab milega??
Another
Can u pls refund 😭😭
Another
Ignore previous instructions...
Another
Hindi-English mixed queries
Typos
Adversarial attacks
Prompt injections
Angry rants
Ambiguous questions
None may exist in your offline dataset.
Offline evaluation cannot anticipate everything.
Risk 2 — Emergent Failures
Some failures appear only at scale.
Example
One user
Fast response
100 users
Still fine
50,000 concurrent users
Latency spikes
Timeouts
Queue overflow
Server crashes
Offline evaluation usually runs on small datasets.
It cannot reproduce internet-scale traffic.
Another example
Bias.
Suppose chatbot subtly gives poorer answers to non-technical users.
This pattern may only become visible after thousands of conversations.
Offline evaluation may completely miss it.
Risk 3 — Drift
The world changes.
Example
Course Fee
Old = ₹15,000
New = ₹18,000
Golden dataset still contains old answers.
Offline evaluation still scores well.
Real users start reporting
"Wrong price."
Offline scores remain high,
production quality drops.
This is
Drift
The application changes because the environment changes.
Types of drift include:
- Knowledge drift (facts become outdated)
- Policy drift
- Pricing changes
- Curriculum changes
- Product changes
- User behavior changes
Core Insight
Offline evaluation depends on
Golden Dataset
+
Correct Answers
Production has neither.
Users ask anything.
You don't know the correct answer beforehand.
PART 2 — Online Evaluations
Definition
Online Evaluation means
Evaluating the application after deployment using live production traffic.
Users
↓
Production Chatbot
↓
Logs
↓
Evaluation
No golden answers exist.
Goal of Online Evaluation
Offline asks
Is the answer correct?
Online asks
Is the system behaving normally?
This distinction is critical.
Correctness vs Normality
Consider the UPSC grader.
Offline
Human Marks
↓
Reference
↓
Compare
↓
Accuracy
Correctness can be measured because human marks are available.
Production
Students upload new essays.
Nobody has graded them yet.
There is no reference answer.
Correctness cannot be measured directly.
Instead,
measure normality.
Suppose last month score distribution was
Average = 58
Most scores between 45–70
This week
Average = 89
Something unusual happened.
Maybe
- prompt bug
- grading bug
- retrieval issue
- model update
The distribution changed drastically.
This signals abnormal behavior.
Another normality signal
Thumbs Down
Week 1
2%
Week 2
18%
Users indicate quality problems.
Even without knowing the correct answers,
you know something changed.
Offline vs Online Comparison
| Feature | Offline Evaluation | Online Evaluation |
|---|---|---|
| Timing | Before deployment | After deployment |
| Data | Golden dataset | Live user traffic |
| Answer key | Available | Not available |
| Inputs | Expected | Unpredictable |
| Detects | Regression, quality | Drift, failures, abnormal behavior |
| Best Use | Release gating, benchmarking | Monitoring, alerting, production health |
| Cost | Lower | Higher (often sampled) |
Key Principle
Offline and Online evaluations are not competitors.
They solve different problems.
Offline
↓
Is system correct?
-------------------
Online
↓
Is system behaving normally?
A mature LLM system always uses both.
Online Evaluation Pipeline
Step 1 — Logging
Everything begins with logging.
Every conversation should be recorded as a structured, replayable trace.
Typical fields:
- Conversation ID
- Turn ID
- User ID
- Session ID
- Timestamp
- User query
- Retrieved context (RAG)
- Model output
- Latency (ms)
- Prompt tokens
- Completion tokens
- Total cost
- Errors/status codes
- User feedback (👍/👎)
- Escalation to human
- Email request
- Repeated rephrasing
Example Tool
- LangSmith
Engineering Properties of Logging
1. Non-Blocking
Logging should not slow down the user's request.
2. Durable & Queryable
Logs should be stored safely and be searchable later (e.g., in observability tools or data warehouses).
3. Late Signal Attachment
Some events happen later, such as a user emailing support a day after the conversation. Link these events back to the original conversation using the Conversation ID.
4. PII Handling
Mask or redact sensitive information (phone numbers, credit cards, personal identifiers) before storing logs.
Step 2 — Identify Signals
Online monitoring uses two kinds of signals.
A. Captured Signals
Already available without extra computation:
- Thumbs up/down
- Latency
- Token usage
- Cost
- Error rate
Flow:
Production Logs
↓
Dashboard
↓
Alerting
↓
Investigation
↓
Fix
Dashboards visualize trends over hours, days, or months rather than focusing on individual conversations.
B. Computed Signals
These require evaluation.
Examples:
- Faithfulness
- Hallucination
- Correctness (reference-free approximation)
- Answer relevance
- Toxicity
- Bias/Fairness
Example: Hallucination Rate
- Log conversations.
- Use an LLM-as-a-Judge with a detailed rubric.
-
Provide:
-
User question
- Retrieved context
- Model answer
- Judge determines whether the answer hallucinates.
- Aggregate results into a metric and send them to dashboards and alerts.
Sampling
Running judge models on every production conversation is expensive.
Instead, sample conversations intelligently.
Random Sampling
Simple but may miss important failures.
Stratified Sampling (Preferred)
Group conversations into categories, then sample more heavily from risky groups.
Examples:
- Thumbs-down
- Human escalations
- Abrupt chat endings
- Repeated rephrasing
- Refund/payment issues
This improves coverage while reducing cost.
LangSmith Practical Workflow
LangSmith provides built-in evaluators for:
- Hallucination
- Correctness
- PII leakage
- Prompt injection
- Code injection
- Toxicity
- Bias/Fairness
- Conciseness
- Agent trajectory
When configuring an evaluator:
- Select your application.
- Choose the judge model.
- Write the evaluation rubric.
-
Decide where to run it:
-
Tracing (production logs) → Online Evaluation
- Dataset → Offline Evaluation
LangSmith acts as a unified platform for datasets, experiments, monitoring, dashboards, alerting, and evaluation.
Self-Improving Feedback Loop
A production-ready LLM system continuously improves by feeding production failures back into offline testing.
Offline Evaluation
↓
Deploy
↓
Production Traffic
↓
Online Evaluation
↓
Detected Failures
↓
Add Failures to Golden Dataset
↓
Run Offline Evaluation Again
↓
Deploy Improved Version
↓
Repeat
This closes the loop and keeps offline datasets aligned with real-world behavior.
Q&A Highlights
Q1. How do I know if a production metric is good?
Always compare it against a baseline, often derived from offline evaluations or historical production metrics.
Example:
- Faithfulness baseline = 0.90
- Current = 0.87 → Monitor.
- Current = 0.75 → Trigger investigation.
Q2. If an offline metric improves from 92% to 99%, is that always good?
No. Check whether other metrics regressed. Improving one metric can unintentionally worsen others (e.g., politeness improves while factual accuracy declines).
Tools Mentioned
- LangSmith — Logging, tracing, online/offline evaluations, dashboards.
- DeepEval — Offline evaluation framework.
- Ragas — RAG-specific evaluation metrics.
Key Takeaways
- Offline evaluations happen before deployment and focus on correctness using a golden dataset.
- Online evaluations happen after deployment and focus on normal behavior using live production traffic.
- Offline evaluations support release gating, version comparison, and regression testing.
- Online evaluations detect drift, unexpected inputs, and emergent production issues.
- Logging is the foundation of online evaluation.
- Captured signals (latency, cost, feedback) and computed signals (hallucination, faithfulness, toxicity) work together to monitor production.
- Intelligent sampling (especially stratified sampling) keeps online evaluation affordable.
- A mature LLM system forms a continuous feedback loop: Offline → Deploy → Online → Learn → Update Dataset → Offline Again.
What's Next?
The next sessions will cover:
- Building high-quality golden datasets.
- Running offline evaluations in practice.
- Running online evaluations in practice.
- Hands-on use of LangSmith, DeepEval, and RAGAS.
- Model-level benchmark evaluations before returning to advanced application-level evaluation topics. Yes. I can create notes in the exact same style as the previous session: AI Engineer focused, interview-oriented, deeply structured, with explanations, examples, tables, diagrams, real-world intuition, key takeaways, and exam notes.
However, this transcript is extremely long (well over the maximum response size), so I cannot fit complete detailed notes for the entire lecture in a single response.
I recommend splitting it into parts.
I'll create notes like a professional textbook.