LLM Leaderboards

1. What Is an LLM Leaderboard?

After different LLMs take the same benchmark, their scores have to be displayed somewhere so we can compare them. That comparison table is the leaderboard.

A leaderboard therefore gives us a single place where different models can be compared on the same benchmark or set of evaluations.

2. Why Do LLM Leaderboards Exist?

1. Compare Models Across Labs

Different labs can be compared using a common reference. If every model takes the same exam, the leaderboard tells us who performed first, second, last, etc.

This gives users an immediate overview of which models perform best on a particular benchmark.

2. Provide Trust Through Third-Party Evaluation

Third-party leaderboards can be more trustworthy than a company simply publishing its own score. If a model provider says “our model scored very high,” users may naturally question the claim because the provider has an incentive to present its model positively.

A third party evaluating multiple providers on the same benchmark creates a more independent comparison.

3. Guide Model Selection Without Running Every Evaluation

There are hundreds of models. Testing every model yourself can require enormous amounts of money, time, and effort.

A practical strategy is to use a leaderboard to narrow a huge candidate pool to a small shortlist, and then run your own evaluations on those shortlisted models.

4. Detect Benchmark Saturation

If many top models begin clustering around almost identical scores—for example, the top 10 models all scoring around 92–94—it may indicate that the benchmark is becoming saturated.

Once a benchmark stops clearly separating strong models, it becomes less useful for distinguishing newer capabilities.

5. Discover New Models

Leaderboards are useful for discovering models beyond the usual top three or four frontier labs. Scrolling further down can reveal newer or smaller models that may not be the absolute best but could be useful for a particular application.

These models can also be cheaper, making them interesting candidates for production applications.

3. Who Uses LLM Leaderboards?

Frontier-Lab Strategy

Leaderboards can reveal whether a new model iteration is actually competitive. If a new internal model cannot beat an existing competitor, a lab may choose not to release that iteration and continue improving it.

The lecture also uses the “Nano Banana” example to illustrate how a model can initially appear under a hidden/stealth identity on an image leaderboard, attract attention by performing extremely well, and later be publicly identified.

4. Four Types of LLM Leaderboards

Type 1 — Benchmark-Specific Leaderboards

These rankings are based on a single benchmark such as MMLU, HumanEval, GSM8K, or GPQA.

The main advantage is simplicity: you can immediately see which model performs best on that benchmark.

Example from the lecture: The Humanity’s Last Exam (HLE) leaderboard, which displays model performance on that particular benchmark.

Type 2 — Multi-Benchmark Leaderboards

These combine multiple benchmarks and evaluation dimensions to create a more comprehensive view.

Possible dimensions can include knowledge, reasoning, mathematics, coding, instruction following, and data analysis.

Example discussed: LiveBench, described in the lecture as a contamination-free LLM benchmark with objective tasks across multiple categories, including reasoning, coding, and agentic coding.

Such leaderboards can also provide operational information such as:

The lecture also discusses Artificial Analysis as an example of a platform providing many types of model leaderboards, including intelligence, speed, cost, coding agents, speech, image, audio, and hardware-related comparisons.

Type 3 — Human-Preference Leaderboards

These do not primarily rank models by a fixed benchmark score. Instead, users compare responses from two models and vote for the response they prefer.

Famous example discussed: LM Arena, where users compare model responses and vote for the better one. The lecture notes that categories can include normal chat, coding, image, and video.

Type 4 — Application-Specific Leaderboards

These are built around a particular domain, task, or application.

• Coding
• Agentic tasks
• SQL/query generation
• Medical questions
• Tool/function calling

Example discussed: Berkeley Function Calling Leaderboard, which focuses specifically on a model’s tool/function-calling capability.

It can combine multiple benchmarks while remaining focused on one domain.

5. Why You Cannot Blindly Trust Leaderboards

1. Benchmark Performance May Not Transfer to Real Applications

A model scoring 80–90 on a benchmark does not mean it will deliver the same level of performance in your application.

Benchmarks generally have cleaner data and clearer problem statements. Real applications are messy.

• Ambiguous requests
• Missing information
• Company-specific data
• Tool failures
• Unusual edge cases

A model may perform extremely well on a clean benchmark while behaving differently in these real-world conditions.

2. Benchmark Contamination

If benchmark data becomes part of a model’s training data or is otherwise memorized, its benchmark score can become inflated.

Therefore, a high leaderboard score is not automatically proof of genuine generalization.

3. Models Can Be Over-Optimized for the Leaderboard

When a leaderboard becomes a major target, organizations may optimize models specifically for that evaluation.

For example, if a human-preference leaderboard rewards certain response styles, a model could be trained to produce answers that are more likely to receive votes without necessarily improving its underlying capability.

The lecture refers to this as Goodhart’s Law.

4. Composite Leaderboards Hide Methodology

When a leaderboard combines multiple benchmarks, you need to understand:

• Which benchmarks are included?
• Which benchmarks are excluded?
• How are scores normalized?
• How much weight does each capability receive?
• How is the final composite score calculated?

The more transparent the methodology, the more useful the leaderboard becomes.

5. Small Score Differences Do Not Necessarily Mean Meaningful Differences

Suppose one model scores 84.3 and another scores 84.1. A difference of 0.2 does not necessarily mean the first model is meaningfully better.

Do not become overly focused on rank numbers. If two models are statistically or practically very close, the lower-ranked model could still be better for your application.

6. Human-Preference Leaderboards Have Human Bias

Human-preference rankings do not necessarily identify the objectively strongest model.

Humans may favor:

• Longer answers
• More confident answers
• Better formatting
• More entertaining responses
• Answers that personally feel appealing

7. Leaderboard Scores Can Be Stale, Incomplete, or Self-Reported

Some leaderboards may not be updated immediately when new model versions are released. They may still show older model versions or discontinued models.

Some results may also be self-reported by model providers. That makes it important to understand where the numbers came from and whether the evaluation process is independent.

6. How an AI Engineer Should Read a Leaderboard

Step 1 — Define Your Application Constraints

Writing these constraints first prevents you from automatically choosing the #1 model. For example, an on-premise requirement may eliminate models that are only available through a public API.

Step 2 — Choose the Relevant Leaderboard

Use a leaderboard that matches your application.

Step 3 — Read the Leaderboard Correctly

Do not only look at the score. Investigate the evaluation methodology.

• What exactly is being scored?
• How is it scored?
• Who evaluated it?
• What inference/reasoning budget was used?
• Was reasoning enabled?
• When was the evaluation dataset created?
• How frequently is the leaderboard updated?
• Is there a private test set?
• Has the benchmark become saturated?
• Are confidence intervals reported?
• How large is the evaluation dataset?
• For composite leaderboards, what weights are assigned to each capability?

Step 4 — Shortlist 3–5 Candidate Models

Use the leaderboard as a filtering mechanism. Based on your application constraints and the leaderboard methodology, select a small set of candidate models.

Step 5 — Run Your Own Custom Evaluations

This is the most important step. Take the 3–5 shortlisted models and run them on your own evaluation set and application-specific requirements.

The model that performs best on your own evaluation becomes your production candidate.

7. Most Important Takeaway

Do not use a leaderboard to directly decide which model your application will use.

8. Session Flow

9. Quick Revision