LLM Evaluation
Module 15 / 17
15 / 17
Model Evaluation & Benchmarks

Who Performs Model Evaluations?

Model evaluations are performed by three major groups, each with different goals and evaluation methodologies.

Understanding who is evaluating a model helps you correctly interpret benchmark results and choose the right model for your application.

1. Frontier Labs (Model Builders)

Who are Frontier Labs?

Frontier labs are organizations that build and release large language models.

Examples include:

These organizations develop foundation models and continuously evaluate them throughout the development lifecycle.

Why Do Frontier Labs Evaluate Models?

Frontier labs perform evaluations for three main reasons.

1. Internal Development

The vast majority of evaluation happens before the model is released.

Engineers use evaluations to guide every stage of model development.

Evaluations help answer questions such as:

Without evaluation, model development becomes guesswork.

You can't improve what you can't measure.

2. Release Gating

Before releasing a new model, labs evaluate whether it is:

If evaluation results are poor, the model is not released.

This process is called release gating.

New Checkpoint
      │
      ▼
Run Evaluations
      │
      ▼
Pass?
 ┌────┴────┐
 │         │
Yes       No
 │         │
Ship    Continue Training

3. Marketing

The benchmark tables shown in launch blogs are also produced through evaluations.

Example:

Model MMLU SWE-bench GPQA
Model A 90% 72% 81%
Model B 87% 65% 76%

These benchmark numbers help demonstrate improvements over competing models.

This is the evaluation work most users see publicly.

How Should You Interpret Frontier Lab Results?

Frontier lab benchmark scores should be viewed as a best-case performance estimate.

These results are often measured under carefully optimized conditions, such as:

As a result:

Think of frontier lab numbers as a performance ceiling rather than an average real-world outcome.

2. Third-Party Evaluators

Who Are They?

Third-party evaluators are independent organizations whose primary purpose is to evaluate AI models.

Examples include:

Unlike frontier labs, they do not build foundation models.

Their goal is to evaluate models fairly and independently.

Why Do They Evaluate Models?

Their business depends on being trusted as neutral evaluators.

They:

Their focus is on fair comparisons rather than marketing.

Why Are Third-Party Evaluations Valuable?

For AI engineers, third-party evaluations are often the most useful because:

This makes it easier to compare competing models objectively.

What Do Third-Party Evaluators Measure?

In addition to benchmark accuracy, they often evaluate practical deployment metrics that model builders may not emphasize.

These include:

These operational metrics are crucial when selecting a model for production systems.

How Should You Interpret Third-Party Results?

Because every model is tested under the same conditions:

For engineering decisions, third-party evaluations are often more informative than vendor-reported benchmark tables.

3. Companies and AI Engineering Teams

Who Are They?

This group includes organizations that use AI models to build real-world applications rather than creating foundation models.

Examples include:

Their goal is not to determine the world's best model but to identify the model that works best for their specific use case.

Why Do Companies Evaluate Models?

Public benchmarks measure general capabilities, but businesses need answers to application-specific questions.

For example:

These questions cannot be answered by generic benchmarks like MMLU or GSM8K.

Example: Customer Support Email Routing

Suppose a company is building an AI system to categorize incoming customer emails into:

Public benchmarks might show that Model A is the most capable model overall.

However, after evaluating both models on the company's own dataset:

Metric Model A Model B
Category Accuracy 94% 91%
Cost / 1,000 Emails $6.20 $0.21
p95 Latency 4.1 s 0.9 s

Although Model A is slightly more accurate, Model B is dramatically cheaper and faster.

For this business, Model B may provide the better overall trade-off.

This illustrates why companies perform custom evaluations instead of relying solely on public benchmarks.

Comparing the Three Evaluators

Evaluator Primary Goal Measures Best For
Frontier Labs Improve and release models Capability, safety, benchmark performance Tracking model development and launch performance
Third-Party Evaluators Fair comparison across models Accuracy, cost, latency, reliability Independent model comparison
Companies / AI Teams Solve business-specific problems Task accuracy, cost, latency, business KPIs Selecting the best model for production

Complete Picture

                  Who Evaluates Models?

        ┌────────────────────────────────┐
        │ 1. Frontier Labs               │
        │ Build and improve models       │
        └────────────────────────────────┘
                    │
                    ▼
        ┌────────────────────────────────┐
        │ 2. Third-Party Evaluators      │
        │ Compare models fairly          │
        └────────────────────────────────┘
                    │
                    ▼
        ┌────────────────────────────────┐
        │ 3. Companies & AI Teams        │
        │ Test models for their own use  │
        └────────────────────────────────┘

Key Takeaways