1What Is a Leaderboard?
Simple idea: Benchmark = the exam. Leaderboard = where results get published & compared.
Benchmark→Result→Leaderboard
2Why Leaderboards Exist
- Compare models across labs
- Trust — 3rd-party vs self-reported
- Filter 100+ models → shortlist
- Detect saturation — top scores clustering
- Discover smaller/cheaper models
3Filtering, Not Deciding
Key idea: leaderboards narrow 100+ models to ~3–5 candidates you then test yourself.
4Who Uses Leaderboards
| AI Engineers | Shortlist → custom evals |
| Frontier Labs | Track competitive position |
| Researchers | Saturation, trends, techniques |
| Policy/Safety | Monitor capabilities |
| OSS Community | Discovery, visibility |
5"Nano Banana" Example
A model appeared under a stealth identity on an image leaderboard, performed extremely well, then was publicly identified — shows leaderboards as discovery tools for labs too.
64 Leaderboard Types
| Type | Question |
|---|
| Benchmark-Specific | Best on this one benchmark? |
| Multi-Benchmark | Best overall combination? |
| Human-Preference | Which do humans prefer? |
| Application-Specific | Best for this domain/task? |
7Type 1 — Benchmark-Specific
Single benchmark (MMLU, HumanEval, GSM8K, GPQA). Simple, but narrow.
Limit: excellent on one test ≠ best overall model.
Ex: Humanity's Last Exam (HLE).
8Type 2 — Multi-Benchmark
Combines dimensions: knowledge, reasoning, math, coding, instr. following.
Cost/tokenLatencyOutput speedContext window
Ex: LiveBench (contamination-free), Artificial Analysis (intelligence/speed/cost/coding/speech/image/audio).
9Type 3 — Human-Preference
2 models answer→User votes better→Votes aggregate→Ranking
Ex: LM Arena — chat, coding, image, video categories.
Bias: humans favor longer/confident/well-formatted/entertaining answers — not necessarily "correct."
10Type 4 — Application-Specific
CodingAgenticSQLMedicalTool calling
Ex: Berkeley Function Calling Leaderboard.
11Core Warning
Rule: a leaderboard score is not a prediction of real-world app performance.
12Limit 1 — Doesn't Transfer
Benchmarks = clean data, clear problems. Real apps are messy:
Ambiguous requestsMissing infoCompany dataTool failuresEdge cases
13Limit 2 — Contamination
If benchmark data leaks into training data, score gets inflated — not proof of real generalization.
14Limit 3 — Goodhart's Law
"When a measure becomes a target, it becomes less useful as a measure."
Models can get over-optimized for leaderboard-specific reward patterns instead of real capability.
15Limit 4 — Hidden Methodology
- Which benchmarks included/excluded?
- How are scores normalized?
- What weight per capability?
- How is composite computed?
16Limit 5 — Small Diffs Don't Matter
84.3 vs 84.1 ≈ noise, not meaningful superiority. Don't fixate on rank position alone.
17Limit 6 & 7
Human biasPrefers style over correctness
Stale/self-reportedOld versions, unverified numbers
18How to Read a Leaderboard
Start with: "What does my app need?" — not "who is #1?"
19Step 1 — Define Constraints
App typeLatencyCostContext needsDeployment (API/on-prem)
20Step 2 — Match Leaderboard
| App | Leaderboard |
|---|
| Chatbot | Human-preference (LM Arena) |
| Agent | Agent-specific |
| RAG | Embedding/model leaderboards |
| Cost-sensitive | Cost + speed oriented |
| Tool calling | Function-calling leaderboard |
21Step 3 — Read Methodology
- What & how is scored, by whom?
- Inference/reasoning budget used?
- Dataset age, update frequency
- Private test set? Saturated?
- Confidence intervals, dataset size
- Composite weighting per capability
22Step 4 & 5 — Shortlist & Test
- Filter to 3–5 candidates
- Run your own custom evals
- Best on YOUR eval = production choice
23Full Selection Funnel
100+ Models→Leaderboard Filter→3–5 Candidates→Custom Evals→Final Model
24Session Flow
Benchmarks→Leaderboards→Custom Evals→App Evals (RAG/Agent)
25Cheat Table
| Benchmark | The test/exam |
| Leaderboard | Ranking/comparison table |
| LM Arena | Human-preference example |
| Goodhart's Law | Measure → target → less useful |
| Benchmark score → prod? | Not necessarily |
| Engineer's role | Filter → shortlist → custom eval |
26Memory Trick
Benchmark = Exam→Score = Result→Leaderboard = Ranking→Shortlist = Filter→Custom Eval = Decision