LLM Evaluation
Module 14 / 17
14 / 17
Model Evaluation & Benchmarks

Evaluation Execution Pipeline (How a Benchmark is Actually Run)

At first glance, evaluating a model seems simple:

  1. Ask the model a question.
  2. Check whether the answer is correct.
  3. Repeat for all questions.
  4. Calculate the final score.

In reality, evaluation is a carefully controlled pipeline where every step must be identical across models to ensure a fair comparison.

What is an Evaluation Loop?

An evaluation loop is the process of running a model on every example in a benchmark dataset and measuring its performance.

For every question in the dataset, the same sequence of operations is performed.

Dataset
   │
   ▼
Load Question
   │
   ▼
Build Prompt
   │
   ▼
Run Model
   │
   ▼
Capture Output
   │
   ▼
Extract Answer
   │
   ▼
Compare with Gold Answer
   │
   ▼
Store Result
   │
   ▼
Repeat for Every Question
   │
   ▼
Aggregate Scores
   │
   ▼
Final Benchmark Score

Step 1 — Load the Dataset Item

The evaluation starts by loading one example from the benchmark.

Each dataset item contains:

Example

Question:
Natalia sold clips to 48 friends in April.
She sold half as many in May.
How many clips did she sell altogether?

Gold Answer:
72

At this stage, nothing has been sent to the model.

Step 2 — Build the Prompt

The raw question is converted into the exact prompt the model will receive.

This step may include:

Example (Few-Shot Prompt)

You are a helpful math assistant.

Example 1:
Question:
2 + 3

Answer:
5

Question:
Natalia sold clips...

The benchmark controls this prompt so every model receives the exact same input.

Why this matters:

Even small prompt changes can significantly affect benchmark scores.

Step 3 — Call the Model

The prompt is sent to the model using a fixed decoding configuration.

Typical settings include:

API Model

HTTP Request
↓

OpenAI API
↓

Response

Local Model

Prompt
↓

GPU

↓

Generated Output

The goal is to ensure every model is evaluated under identical conditions.

Step 4 — Capture the Raw Output

The model generates a response.

Example:

Natalia sold 48 clips in April.

Half of 48 is 24.

Therefore,

48 + 24 = 72 clips altogether.

This raw output is stored exactly as generated.

Why logging raw output is important

Suppose the benchmark reports an incorrect answer.

Without the raw output, it's impossible to know whether:

Raw outputs are essential for debugging evaluation errors.

Step 5 — Extract the Final Answer

Models often generate explanations rather than only the final answer.

Example output:

Natalia sold 48 clips in April.

Half is 24.

Total = 72 clips.

The benchmark must extract only the final answer.

Typical methods include:

Example

Raw Output

↓

Regex

↓

72

Without extraction, the evaluator cannot compare the prediction to the gold answer.

Step 6 — Grade the Answer

The extracted prediction is compared against the correct answer.

Example:

Prediction = 72

Gold = 72

Result:

Correct

Score = 1

Another example:

Prediction = 74

Gold = 72

Result:

Incorrect

Score = 0

For closed-ended tasks, grading is automatic because there is a single correct answer.

Step 7 — Store the Result

The evaluator records all important information for each question.

Typical information stored:

Example record:

{
    Prompt,
    Output,
    Prediction,
    Gold Answer,
    Score
}

Why store everything?

If benchmark scores appear suspicious, engineers can inspect:

This makes evaluation reproducible and debuggable.

Repeat for Every Question

The evaluator repeats the same process for every item in the dataset.

Question 1
↓

Question 2
↓

Question 3
↓

...

↓

Question N

Every example follows the exact same evaluation pipeline.

Aggregate the Results

After evaluating all questions, individual scores are combined into a single benchmark metric.

Example:

1000 Questions

↓

920 Correct

↓

Accuracy = 92%

Formula:

Accuracy = Total Correct Answers / Total Questions

This aggregated score is the final benchmark result reported in papers and leaderboards.

Evaluation Pipeline Summary

Dataset
   │
   ▼
Load Question
   │
   ▼
Build Prompt
   │
   ▼
Run Model
   │
   ▼
Capture Raw Output
   │
   ▼
Extract Final Answer
   │
   ▼
Compare with Gold Answer
   │
   ▼
Assign Score (0 or 1)
   │
   ▼
Store Result
   │
   ▼
Repeat for All Questions
   │
   ▼
Aggregate Scores
   │
   ▼
Final Benchmark Score

Pseudocode of an Evaluation Loop

results = []

for item in dataset:

    # Step 1: Build the prompt
    prompt = build_prompt(
        item,
        few_shot=8,
        template=chat_template
    )

    # Step 2: Run the model
    output = model.generate(
        prompt,
        temperature=0,
        max_tokens=512,
        stop=["\n\n"]
    )

    # Step 3: Extract the final answer
    prediction = extract_answer(output)

    # Step 4: Grade the answer
    score = (prediction == item.gold)

    # Step 5: Store the result
    results.append({
        "prompt": prompt,
        "output": output,
        "prediction": prediction,
        "gold": item.gold,
        "score": score
    })

# Step 6: Aggregate scores
accuracy = mean(r["score"] for r in results)

Why This Pipeline Matters

A benchmark score is only meaningful if every model is evaluated using the same pipeline.

Changing any component can change the final score:

For fair and reproducible comparisons, the entire evaluation pipeline—not just the dataset—must remain consistent.

Key Takeaways

Evaluation Harness (Automating Model Evaluation)

When you first learn about model evaluation, it appears to be a very simple process:

  1. Take a question.
  2. Send it to the model.
  3. Check whether the answer is correct.
  4. Repeat for every question.

In reality, running a benchmark on thousands of questions involves many engineering challenges. Instead of writing all this logic yourself, AI engineers use an evaluation harness.

Why Do We Need an Evaluation Harness?

Imagine you want to evaluate a model on 10,000 benchmark questions.

Simply sending prompts to the model is not enough.

You also need to handle:

Implementing all of this from scratch is time-consuming and error-prone.

An evaluation harness solves these problems automatically.

What is an Evaluation Harness?

An evaluation harness is a software framework that automates the entire evaluation process.

Instead of manually writing code for every benchmark, you simply tell the harness:

The harness takes care of everything else.

Definition

Evaluation Harness: A ready-made system that automatically runs benchmark evaluations by managing datasets, prompts, model execution, scoring, and result aggregation.

Problems an Evaluation Harness Solves

Without a harness, an engineer must manually implement many repetitive tasks.

1. Answer Extraction

Models often produce long explanations instead of just the final answer.

Example:

After solving the problem carefully, I conclude that the answer is Option C.

or

The correct answer is C.

or

C

All three responses mean the same thing.

The evaluation harness extracts the actual prediction (C) before grading.

2. Official Benchmark Scoring

Different benchmarks use different scoring rules.

For example:

The harness already knows the official scoring method for each benchmark.

3. Efficient Batch Processing

Large benchmarks may contain:

of evaluation examples.

Instead of sending one API request at a time, the harness can:

This makes evaluations much faster.

4. Automatic Retry of Failed Requests

API requests sometimes fail because of:

Without a harness:

Question 5387 failed

↓

Evaluation crashes

A harness automatically retries failed requests until they succeed or reach a retry limit.

5. Rate Limit Handling

Most model APIs have request limits.

Example:

Only 60 requests per minute

If too many requests are sent, the API returns an error.

An evaluation harness automatically:

This prevents unnecessary failures.

What Does an Evaluation Harness Already Know?

A good evaluation harness already contains all the logic required to run standard benchmarks.

It knows:

As a result, engineers can focus on evaluating models instead of building evaluation infrastructure.

Workflow of an Evaluation Harness

Choose Benchmark
        │
        ▼
Load Dataset
        │
        ▼
Format Questions
        │
        ▼
Send Questions to Model
        │
        ▼
Collect Model Outputs
        │
        ▼
Extract Final Answers
        │
        ▼
Score Predictions
        │
        ▼
Aggregate Results
        │
        ▼
Generate Final Benchmark Report

Benchmark vs Evaluation Harness

Many beginners confuse these two concepts.

Benchmark Evaluation Harness
The test itself The system that runs the test
Contains questions Runs the entire evaluation
Defines what to measure Defines how to measure it
Example: GSM8K Example: lm-evaluation-harness
Similar to an exam paper Similar to the exam administration system

Easy Analogy

Benchmark = Exam Paper

Contains:

Example:

Math Exam

Question 1
Question 2
Question 3
...

Evaluation Harness = Complete Examination System

The harness manages everything needed to conduct the exam.

It handles:

The benchmark provides what to test, while the harness provides how to run the test.

Popular Evaluation Harnesses

Several open-source frameworks are widely used in industry and research.

1. lm-evaluation-harness

Best for: General LLM benchmarking.

2. Inspect

Best for: Flexible and customizable evaluations.

3. HELM (Holistic Evaluation of Language Models)

Best for: Broad, multi-dimensional evaluation of LLMs.

Complete Picture

                Benchmark
          (Questions + Answers)
                    │
                    ▼
         Evaluation Harness
     ┌────────────────────────┐
     │ Load Dataset           │
     │ Build Prompt           │
     │ Call Model             │
     │ Retry Failures         │
     │ Handle Rate Limits     │
     │ Extract Answers        │
     │ Score Predictions      │
     │ Aggregate Results      │
     └────────────────────────┘
                    │
                    ▼
          Final Benchmark Score

Key Takeaways