13 / 17

Model Evaluation & Benchmarks

Model Evaluation Notes (Part 3)

Benchmarks: How LLMs Are Evaluated

What is a Benchmark?

A benchmark is a standardized test used to measure a specific capability of an AI model.

Just like students take exams to measure their knowledge, AI models take benchmarks to measure capabilities such as:

Knowledge & Reasoning
Coding
Mathematics
Long Context
Safety
Instruction Following
Vision
Agentic Tool Use

A benchmark allows researchers to compare different models under the same conditions.

Definition

Benchmark = Standardized Dataset + Fixed Evaluation Procedure + Scoring Method

A benchmark ensures that every model is tested fairly.

Why Do We Need Benchmarks?

Benchmarks help answer questions like:

Which model is better?
Is the new model actually improving?
Is the model ready for production?
Which model should our company choose?

Without benchmarks, every company would evaluate models differently, making comparisons impossible.

Four Components of Every Benchmark

Every benchmark consists of four major parts.

                Benchmark
                    │
      ┌─────────────┼─────────────┐
      │             │             │
   Dataset     Run Config    Scoring
                    │
               Aggregation

Part 1 — Dataset

What is a Dataset?

A dataset is simply:

Questions
Correct Answers (Ground Truth)

Every benchmark begins with a collection of carefully designed evaluation questions.

Example: GSM8K

Question

Natalia sold clips to 48 friends in April. In May she sold half as many. How many clips did she sell altogether?

Correct Answer

Each question already has a verified answer.

During evaluation, the model must produce the same answer.

Dataset = Input + Gold Answer

Question
↓

Model

↓

Prediction

↓

Compare with Gold Answer

Example

Question	Gold Answer
Natalia sold clips...	72
Solve 8 × 7	56
Capital of India	New Delhi

Why is the Dataset Important?

A good dataset should be:

Diverse
High quality
Unbiased
Representative of the capability being measured

Poor datasets produce misleading benchmark scores.

Part 2 — Run Configuration

What is Run Configuration?

The Run Configuration contains all the settings used when running the benchmark.

Even the same model can produce different scores if the configuration changes.

A benchmark score is only meaningful if the configuration is clearly specified.

Run Configuration has three layers.

Run Configuration
       │
 ┌─────┼─────┐
 │     │     │
Prompt Decoding Environment

Layer 1 — Prompt Construction

Prompt construction defines how the question is presented to the model.

Even small prompt changes can significantly affect benchmark scores.

Zero-Shot Prompting

The model receives:

Only the question
No examples

Example

Question:

What is 12 × 9?

The model solves it directly.

Advantages

Simple
Fast
Easy to compare

Few-Shot Prompting

Few-shot prompting provides solved examples before asking the actual question.

Example

Example 1
2+2=4

Example 2
3+5=8

Now solve:

12 × 9

These examples teach the model the expected reasoning style and output format.

Why Few-Shot Helps

The model learns:

Expected format
Reasoning style
Output pattern

As a result, accuracy often increases.

Example:

GSM8K is traditionally reported using 8-shot prompting.

Chain-of-Thought vs Direct Answer

Another important prompt choice is whether to encourage reasoning.

Direct Prompt

What is the answer?

The model immediately predicts a number.

Sometimes it guesses incorrectly.

Chain-of-Thought Prompt

Think step by step.

Example

48 sold in April

Half = 24

48 + 24 = 72

Reasoning before answering often improves performance on complex tasks.

Why Chain-of-Thought Improves Accuracy

Instead of guessing,

the model:

Breaks the problem into smaller steps
Solves each step
Produces the final answer

This usually leads to higher reasoning accuracy.

Layer 2 — Decoding / Sampling Configuration

Once the prompt is ready, the model generates an answer.

The decoding configuration controls how the answer is generated.

Temperature

Temperature controls randomness.

Temperature = 0

Greedy Decoding

The model always chooses the most probable token.

Properties:

Deterministic
Reproducible
Same prompt → Same answer

Most benchmarks use Temperature = 0 for fairness.

Temperature > 0

The model samples from multiple possible tokens.

Result:

Same prompt

↓

Different outputs on different runs.

Useful for creativity, but not for benchmark reproducibility.

max_tokens

This limits the maximum length of the model's response.

Example

Suppose the reasoning requires:

150 tokens

But

max_tokens = 50

The response is cut off before the final answer.

Example

48...

Half...

24...

48+

No final answer appears.

The benchmark records this as incorrect, even though the model was reasoning correctly.

Important Interview Point

A low benchmark score may reflect a bad configuration, not a weak model.

Layer 3 — Scoring Strategy & Environment

Even after generating answers, evaluation settings still matter.

Pass@1

The model gets one attempt.

Correct on first attempt?

Yes → Correct

No → Wrong

Pass@k

The model gets k attempts.

If any one attempt is correct,

the question counts as correct.

Example

Attempt 1 ❌

Attempt 2 ❌

Attempt 3 ✅

Pass@3 = Correct

Majority Voting (maj@k)

Also called Self-Consistency.

The model answers multiple times.

The most common answer becomes the final prediction.

Example

72

72

70

72

71

Majority = 72

Why This Matters

The same model can have very different scores.

Example

Pass@1 = 85%

maj@8 = 92%

If a benchmark report doesn't specify the evaluation strategy, the score is difficult to interpret.

Tools Enabled vs Tools Disabled

Some benchmarks allow the model to use external tools.

Examples:

Python Interpreter
Calculator
Search Engine

Example

Without Python

Solve a difficult math problem.

The model reasons mentally.

With Python

Write Python code.

Execute it.

Return result.

Performance can improve significantly.

Therefore:

The same benchmark with tools enabled is not directly comparable to the same benchmark without tools.

Part 3 — Scoring Method

What is Scoring?

Scoring converts the model's generated text into a numerical score.

This happens in two stages.

Model Output

↓

Extract Answer

↓

Compare with Gold Answer

↓

Correct / Incorrect

Stage 1 — Extraction

Models usually generate explanations, not just answers.

Example Output

Natalia sold 48 clips in April.

Half of 48 is 24.

48 + 24 = 72 clips altogether.

The benchmark must extract:

How is Extraction Done?

Common methods include:

Regular Expressions (Regex)
Parsing after markers (e.g., ####)
JSON parsing
Structured output parsing

Why Extraction is Fragile

Suppose the model outputs:

72 clips

$72

Seventy-two

If the parser only accepts plain integers, a correct answer may be incorrectly marked as wrong.

Stage 2 — Comparison

Once the answer is extracted, it is compared with the ground truth.

Example

Prediction = 72

Gold Answer = 72

Result

Correct

Closed-Ended Evaluation

There is exactly one correct answer.

Examples:

Mathematics
Multiple-choice questions
Classification

Evaluation is automatic and objective.

Open-Ended Evaluation

Example

Write a good summary.

There is no single correct answer.

Evaluation often requires:

Human judges
LLM-as-a-Judge
Preference models

This introduces subjectivity and potential bias.

Part 4 — Aggregation

What is Aggregation?

After evaluating every question, we combine all individual scores into one final benchmark score.

Question 1 ✔

Question 2 ✔

Question 3 ✖

...

↓

Overall Accuracy

Example

Suppose:

1000 questions

920 correct

80 incorrect

Final score:

Accuracy = 920 / 1000

= 92%

This is the headline benchmark score reported in papers.

Aggregation Methods

Unweighted Mean

Every question contributes equally.

Final Score

=

Average of all question scores

Weighted Mean

Some categories receive more weight than others.

Example:

MMLU contains 57 subjects with different numbers of questions.

Researchers may weight subjects differently to avoid overrepresenting larger categories.

End-to-End Benchmark Pipeline

Dataset
     │
     ▼
Build Prompt
     │
     ▼
Run Model
     │
     ▼
Generate Output
     │
     ▼
Extract Answer
     │
     ▼
Compare with Gold Answer
     │
     ▼
Per-Question Score
     │
     ▼
Aggregate Scores
     │
     ▼
Final Benchmark Score

Interview Cheat Sheet

Component	Purpose	Key Points
Dataset	Questions + Gold Answers	Defines what the model is tested on.
Run Configuration	Evaluation settings	Includes prompt construction, decoding parameters (temperature, max_tokens), scoring strategy (pass@1, pass@k, maj@k), and tool availability.
Scoring Method	Converts outputs into scores	Extract the final answer, then compare it with the gold answer. Closed-ended tasks are objective; open-ended tasks may require human or LLM judges.
Aggregation	Combines individual results	Produces the final benchmark metric (e.g., 92% accuracy), using methods such as unweighted or weighted averages.

Key Takeaways

A benchmark is a standardized evaluation designed to measure a specific model capability.
Every benchmark has four essential components: Dataset → Run Configuration → Scoring Method → Aggregation.
Run configuration matters: prompt design, temperature, max_tokens, scoring strategy, and tool usage can significantly change results.
Scoring is a two-step process: first extract the model's answer, then compare it with the correct answer.
Aggregation converts thousands of individual evaluations into a single metric, such as 92% accuracy.
When comparing benchmark results, always verify that the dataset, configuration, scoring method, and aggregation strategy are identical; otherwise, the numbers are not directly comparable.