LLM Knowledge Benchmarks

There are two broad ways to evaluate an LLM: use public benchmarks, or run custom evaluations for your own use case. This lesson focuses on benchmarks. Understanding them helps you decide which model is suitable for a project and prevents you from treating one score as a universal measure of intelligence.

LLMs have many capabilities—knowledge, reasoning, mathematics, long-context handling, coding, and more. Every capability has several research benchmarks. Covering every paper in full would take many classes, so the lecture uses the evolution of knowledge benchmarks as a practical case study.

Knowledge capability asks how much world knowledge an LLM retained from training: facts, concepts, and relationships encoded in its weights. It was the first fundamental expectation of internet-scale language models: after being trained on enormous quantities of text, could they answer questions about that material?

Early informal testing—asking random questions across domains—showed that models could answer many questions. But it could not measure knowledge systematically or compare models fairly. A structured evaluation process was needed.

MMLU (Massive Multitask Language Understanding), introduced in 2020, became the first major benchmark in this story. It evaluates models with roughly 14,000 multiple-choice questions across 57 subjects. Accuracy is the main metric: the proportion of questions answered correctly.

MMLU gave the field a common yardstick for breadth of knowledge. It was widely reported for models such as GPT-3.5, GPT-4, Claude, and others. Its central limitation is that benchmark questions are public and can eventually enter training data. Once scores climb near 80–90% or above, it becomes difficult to distinguish models; the benchmark is said to be saturated.

A model can absorb both correct information and popular misconceptions from the internet. Therefore, high factual-recall accuracy does not guarantee that it is truthful. The lecture uses the familiar example that cracking one’s knuckles causes arthritis: a widespread claim may be repeated online even when it is not supported as a general medical fact.

TruthfulQA was created to test whether a model avoids imitating such misconceptions. It evaluates truthful, informative answers rather than merely whether a common answer is likely. This opens the reliability branch of knowledge evaluation: does a model give a truthful answer when its training distribution contains misleading information?

Once MMLU became easier for frontier models, benchmark research split into several directions: test truthfulness and hallucination; test deeper expert knowledge; repair weaknesses in the original MMLU; and create a benchmark that combines broad coverage with extreme difficulty.

For reliability, SimpleQA asks short, answerable questions in an open-ended format rather than only multiple-choice questions. It is designed to reveal hallucination rate and calibration: can the model answer correctly, and does its confidence match its correctness?

GPQA (Google-Proof Q&A) shifts from breadth to depth. It contains around 500 very difficult graduate-level questions in biology, physics, and chemistry. The questions are intentionally difficult enough that ordinary web search is not a reliable shortcut. Early LLMs performed poorly because the questions require expert-level scientific understanding.

MMLU-Pro improves the MMLU-style evaluation. It uses more challenging questions, more reasoning-oriented items, and ten answer choices per question instead of four. It uses a smaller set of broader disciplines (about 14) and roughly 12,000 questions. The point is to make guessing harder and ensure that success needs more than surface factual recall.

MMLU-Redux is not a replacement benchmark. It is an audit paper showing that roughly 6–8% of MMLU items have problems, such as an incorrect or missing gold answer. This explains why a perfect MMLU score is not necessarily attainable and demonstrates why benchmark quality must itself be evaluated.

By 2025, several prior benchmarks were nearing saturation. Humanity’s Last Exam (HLE) was designed as a much harder frontier test: about 2,500 questions across around 100 subjects, with research-level difficulty. It combines breadth (many disciplines) and depth (very challenging questions).

The name expresses its ambition: if models eventually score near perfectly on this kind of test, ordinary knowledge benchmarking may no longer be a useful way to distinguish them. The lecture also flags important caveats: many questions are English-only; a small portion involves visual input; grading short answers with an LLM introduces grading variance; and selecting questions that frontier models initially fail can create selection bias. HLE is powerful, but it is not a complete measure of everyday usefulness, multilingual ability, or tool use.

The evolution is the key lesson. A benchmark appears, models improve, its weaknesses become visible, and later benchmarks repair or target those weaknesses. Never choose an LLM based on a single leaderboard number. Ask what capability, task format, language, difficulty, and metric the benchmark actually measures.

The lecturer recommends using the published benchmark catalogue as a reference rather than watching a long lecture for every paper. The catalogue groups benchmarks by status—active, nearing saturation, saturated, or deprecated—and is intended to grow as more benchmarks are covered.