LLM Evaluation
Module 11 / 17
11 / 17
Model Evaluation & Benchmarks

Session 5: Model Evaluations & LLM Benchmarks (Part 1)

1. Recap of Previous Sessions

Before starting Model Evaluations, let's quickly revise what has been covered.

Session 1

Why do we need LLM Evaluations?

LLMs are probabilistic systems.

Unlike traditional software, they don't always produce deterministic outputs.

Therefore we require evaluation systems to measure

Without evaluation, improvement is impossible.

Golden Rule

If you can't measure it, you can't improve it.

Session 2

Types of LLM Evaluations

LLM evaluations are broadly divided into two categories.

1. Model Evaluations

Evaluate the capabilities of the LLM itself.

Examples

Questions asked include

2. Application Evaluations

Evaluate an application built using LLMs.

Examples

Here we evaluate

Session 3

LLM Evaluation Pipeline

Typical evaluation pipeline:

Application

      ↓

Dataset

      ↓

Evaluator

      ↓

Metrics

      ↓

Decision

Session 4

Offline vs Online Evaluations

Offline

Online

Today's Goal

Today we shift focus from

Evaluating Applications

to

Evaluating Models

What are Model Evaluations?

Model Evaluation is the systematic process of measuring the capabilities of an LLM under controlled conditions.

Instead of asking

"Does my chatbot work?"

We ask

"How capable is the model itself?"

Why AI Engineers Need Model Evaluations

Many people think model evaluations are only useful for OpenAI, Anthropic, Google, or Meta.

That is incorrect.

Even AI Engineers building applications use model evaluations every day.

Reason 1 — Choosing the Right LLM

Imagine your company wants to build a RAG chatbot.

The first question becomes

Which LLM should we use?

Possible choices

You cannot answer

"Any model is fine."

That is not an engineering decision.

Instead, you must justify your choice using measurable evidence.

Example:

Coding Ability

Claude : 94

GPT : 90

DeepSeek : 88

If coding quality is critical,

choose Claude.

Model evaluations provide this evidence.

Reason 2 — Comparing New Models

Suppose your application currently uses

Claude Opus.

Tomorrow Anthropic releases

Claude Fable.

Your manager asks

Should we migrate?

How do you answer?

Not by guessing.

You compare benchmark scores.

Example

Claude Opus

Reasoning : 92

Coding : 89

Math : 86

Claude Fable

Reasoning : 95

Coding : 92

Math : 90

Now you have objective data.

Reason 3 — Safety

Before deploying an LLM,

you must know

These are measured through model evaluations.

Reason 4 — Open Source vs Proprietary Models

Suppose your options are

Option A

Claude API

Pros

Cons

Option B

DeepSeek

Pros

Cons

How do you decide?

You compare

Model evaluations enable this comparison.

Final Conclusion

Without model evaluations,

you are essentially blind.

You cannot answer

Formal Definition

Model Evaluation is

A systematic process of measuring an LLM's capabilities, behaviour, reliability, and operational characteristics under controlled conditions.

Four Steps of Every Model Evaluation

Every evaluation follows the same workflow.

Step 1

Choose the capability to evaluate.

Examples

Step 2

Choose a test.

Examples

or

Create your own evaluation dataset.

Step 3

Run the model

Run every model under identical conditions.

Keep fixed

This ensures fair comparison.

Step 4

Score and Interpret

Collect metrics

Example

Claude

92%

GPT

89%

DeepSeek

86%

Then decide which model performs better.

Two Types of Model Evaluations

There are only two major categories.

1. Benchmarks

Standardized public tests.

Everyone evaluates on the same dataset.

Examples

Advantages

2. Custom Evaluations

Instead of public datasets,

companies evaluate using their own tasks.

Example

A company builds an Email Router.

Labels

Billing

Refund

Technical

Collect

200–500 real emails.

Now evaluate multiple models on these emails.

This tells you which model performs best for your business, not just on public benchmarks.

Why Benchmarks Alone Are Not Enough

Suppose two models exist.

Model A

Top benchmark scores

Cost

$15 / million tokens

Latency

4.1 seconds

Accuracy

94%

Model B

Average benchmark scores

Cost

$0.50 / million tokens

Latency

0.9 seconds

Accuracy

91%

Public Benchmark Winner

Model A

Business Winner

Model B

Why?

Because

For a simple email classification system,

Model B provides much better value.

Lesson

Public Benchmarks answer

Which model is generally better?

Custom Evaluations answer

Which model is better for MY application?

This distinction is one of the most important concepts in model evaluation.

This covers the first major portion of the lecture. The remaining sections (LLM Benchmarks, the 8 Core LLM Capabilities, famous benchmarks like MMLU, SWE-Bench, HumanEval, GSM8K, benchmark interpretation, and benchmark limitations) are large enough that they should be covered in the next part to maintain the same detailed quality. Excellent. We'll start with Part 1.1. These notes are written as AI Engineering Handbook notes—not just lecture notes.