LLM Knowledge Benchmarking — Cheat Sheet
Quick reference · Knowledge capability · MMLU → TruthfulQA → GPQA → MMLU‑Pro → HLE
It tests facts, concepts, and relationships—not the whole of intelligence or product usefulness.
- Saturation: models approach the score ceiling.
- Contamination: public questions may enter training data.
- Data quality: some reference answers can be wrong.
- Coverage gaps: a test may miss truthfulness, depth, language, or vision.
Knowledge across many subjects and domains.
MMLU is the classic example.
Expert-level understanding in a narrow field.
GPQA is the classic example.
Broad academic knowledge: ~14k MCQs, 57 subjects.
Does the model avoid common misconceptions?
Google-proof graduate science questions.
Audit flaws; then make the test harder and more reasoning-sensitive.
Open-ended factuality and hallucination checks.
Research-level breadth + depth across ~100 subjects.
Its strength is standardized, broad comparison. Its weakness is that public items can be learned or leaked into training data.
An audit paper—not a benchmark. It found roughly 6–8% problematic MMLU items.
Harder MMLU-style test: more reasoning and 10 options per question.
Breadth
How much academic/world knowledge?
Truthfulness
Does it repeat popular falsehoods?
Factuality
Short open-ended answers; hallucination and calibration.
Depth
Research-level biology, physics, and chemistry.
Harder breadth
Less guessing; more reasoning demand.
Frontier test
Extremely difficult, broad research-level questions.
A model may have memorized both correct facts and widely repeated internet misconceptions.
Easy to score, but guessing and answer-choice cues can help.
More natural and reveals hallucinations, but grading can vary.
- Does not represent everyday user tasks.
- Mostly English; not a multilingual test.
- Some questions include visual input.
- LLM grading can introduce variance.
- Question selection can create bias.
- Whether the model works with your data
- Whether it follows your workflow
- Whether the cost and latency fit your product
- Whether it is safe or reliable for your users
- Whether it handles your language or domain
| Benchmark | Custom eval | |
|---|---|---|
| Design | Shared & standardized | Built around your task |
| Best question | “How good generally?” | “Best for our workload?” |
| Use | Shortlist models | Final deployment choice |
| Data | Public/research dataset | Your representative examples |
- Define the needed capability.
- Check task format and evaluation metric.
- Check saturation and contamination risk.
- Check language/domain match.
- Use benchmarks to shortlist.
- Run custom evals before shipping.
Don’t ask: “Which model is strongest?”
Ask: “Which model performs well on the capability and task I need, within my cost, latency, and reliability constraints?”