LLM Evaluation — Cheat Sheet

1 · Core Idea

LLM Evaluation = the systematic, repeatable measurement of an LLM/app's quality, correctness, reliability, safety, and production-readiness — using datasets, rubrics, and metrics instead of guesswork.

Build → Evaluate → Fix → Deploy → Monitor (continuous loop)

Vibe Testing

Judging quality from a handful of manual prompts + gut feel. No metrics, not reproducible, misses edge cases. OK for toy projects; unsafe at production scale.

2 · Why It Matters: Case Studies

Lesson: hallucination + jailbreak + zero evaluation = legal, financial, and reputational risk.

3 · Common Failure Types

• Hallucination — confidently wrong/fabricated info
• Jailbreak — bypassing safety rules via crafted prompts
• Prompt Injection — hidden instructions in input/content hijack behavior
• Prompt Leakage — system prompt/internal instructions exposed
• Toxic Output — offensive, biased, unsafe content
• Privacy Leakage — PII or training data exposed

4 · Deterministic vs Probabilistic

5 · Model Eval vs Application Eval

Model Evaluation

Tests the raw foundation model via public benchmarks (MMLU, GSM8K, HumanEval…). Done by frontier labs, third parties, and teams comparing models.

Application Evaluation

Tests the system built on top (chatbot, RAG, agent) for a specific use case: task success, safety, latency, cost.

6 · Evaluation Dimensions (15)

Correctness · Factuality · Groundedness · Faithfulness · Relevance · Completeness · Conciseness · Fluency · Tone · Safety · Bias · Robustness · Latency · Cost · User Satisfaction

Groundedness = supported by retrieved context. Faithfulness = doesn't contradict the source. Factuality = objectively true regardless of source.

7 · Three Evaluation Methods

Programmatic (deterministic)

Exact-match, regex, Recall@K / Precision@K. Fast & cheap; only works for verifiable outputs.

Human Evaluation

Direct rating, red-teaming, A/B testing, golden-dataset creation, human-in-the-loop. Gold standard, but slow/expensive — use a rubric + multiple raters.

LLM-as-a-Judge

A strong LLM scores outputs vs a rubric. Scalable for open-ended text; the judge itself must be validated (MAE vs human scores) — cheaper than humans, can drift or be biased.

Reference-based needs a gold answer   Reference-free judges quality without one

8 · Application-Specific Evaluation

• RAG — Retriever: Recall@K, Precision@K, MRR. Generator: faithfulness, groundedness, answer relevance.
• Agents — task completion, tool-call correctness, step efficiency, planning quality
• Safety — jailbreak resistance, red-teaming, toxicity, PII leakage
• Operations — latency, cost, uptime, logging/tracing, drift detection

9 · 12-Step Application Eval Workflow

Define task & target → Define success criteria → Build golden dataset → Choose eval method → Run model → Evaluate results → Analyze results → Improve system → Repeat evaluation → Deploy → Monitor → Update golden dataset ↺

10 · Offline vs Online Evaluation

Offline — pre-deployment, golden dataset. Used for release gating, version comparison, regression testing. Misses: unanticipated inputs, emergent failures, drift.

Online — post-deployment on real traffic. Captured signals (👍/👎, retries) + computed signals (latency, refusal rate). Goal: detect abnormality, not just incorrectness.

11 · Why Multiple Eval Pipelines per App

Reason 1 — Multiple failure points: retriever ≠ generator ≠ workflow ≠ whole app; each layer needs its own eval.

Reason 2 — Multiple risk categories: Quality, Safety, and Operations each need a dedicated pipeline.

12 · Model Eval: 8 Core Capabilities

Knowledge & Reasoning · Coding & Software Engineering · Mathematics · Long Context · Vision & Multimodal · Agentic & Tool Use · Safety & Alignment · Instruction Following

13 · Anatomy of a Benchmark

• Dataset = input + gold answer
• Run Config = zero/few-shot, CoT vs direct, temperature, pass@k, tools on/off
• Scoring = extract answer → compare (exact-match or judge/rubric)
• Aggregation = mean, macro vs micro average, weighted by category

14 · Who Performs Model Evaluation

• Frontier Labs — self-report at model release (read cautiously — marketing incentive)
• Third-Party Evaluators — independent, standardized, comparable across labs
• Companies / Eng Teams — custom evals for their exact use case (build vs buy)

15 · Knowledge & Reasoning Benchmarks

16 · Common Benchmark Problems

• Contamination — test data leaks into training data
• Saturation — models max out, benchmark stops discriminating
• Configuration Gaming — cherry-picked settings inflate scores
• Aggregation Bias — one average hides per-category weaknesses

17 · Key Takeaway

Building an LLM application is only half the job. Evaluating, securing, monitoring, and continuously improving it is the other half — that's what separates a prototype from a production AI system.