AI Benchmarks Explained: MMLU, HumanEval, and More

A benchmark is a standardised exam

When a lab announces a new model, the launch post is usually a wall of bar charts. MMLU. GPQA. HumanEval. SWE-bench. Each of those is a fixed set of questions with known answers. The model is fed the questions, its answers are marked automatically, and the percentage it gets right becomes the number on the chart. That is genuinely all a benchmark is: a standardised exam, sat by a machine.

That framing is useful because it tells you what the score can and cannot mean. An exam result tells you how someone did on that exam. It does not tell you whether they will be good at your job.

The names you keep seeing

MMLU and MMLU-Pro

Massive Multitask Language Understanding: multiple choice questions across dozens of subjects, from school history to professional law. It is the closest thing the field has to a general knowledge test. MMLU-Pro is the harder rewrite, created after the leading models started scoring so highly on the original that it stopped telling them apart.

GPQA

Graduate-level questions in biology, physics and chemistry, written so that a clever person with a search engine still struggles. It tests reasoning through unfamiliar technical material rather than recall.

HumanEval and SWE-bench

Both are coding tests, but they measure different things. HumanEval hands the model a small self-contained programming problem and checks whether the code passes hidden tests. SWE-bench is closer to real work: it takes genuine issues from open source projects and asks for a patch that fixes them. A good HumanEval score means the model can write a function. A good SWE-bench score means it can find its way around somebody else's codebase.

LMArena and preference rankings

Instead of marking against an answer key, people are shown two anonymous responses to the same prompt and asked which they prefer. Votes are aggregated into a ranking. This captures something the other tests cannot, namely whether humans actually like the output, though it also rewards answers that look confident and are neatly formatted.

Why the scores climb while your day stays the same

Three things break the link between a launch chart and your experience of the tool.

• Saturation. Once the leading models all sit in the same narrow band near the top, the gaps between them are largely noise. A couple of points at that level is not something you will notice in a customer email.
• Contamination. These test sets are published on the internet, and models are trained on the internet. When questions leak into training data, the model is being marked on homework it has already seen. Labs try to filter for this with mixed success.
• Distance from your work. Nothing on the standard list measures whether a model can write a decent reply to a two-star review, or read a year of quotes and flag the ones nobody chased.

Build your own five-prompt exam instead

The genuinely useful move is a small private benchmark. It takes an afternoon and it will outlive every model release.

• Pick five tasks you would actually hand to an assistant: a service page rewrite, a review reply, a summary of a long email thread, a call list built from messy notes, a monthly report.
• Write down what a good answer looks like before you run anything. Length, tone, what must never appear.
• Run the same five prompts through each model you are considering, unchanged.
• Mark them yourself, against your own standard, and save the results in a file with the date.

When the next model arrives, rerun the same five prompts. If the output is better on your work, switch. If it is not, the chart in the announcement is irrelevant to you.

When published benchmarks are still worth reading

They are not useless. They are good for ruling options out rather than picking a winner. A model that scores poorly on reasoning tests will probably struggle with anything multi-step, and that is worth knowing before you spend a week building around it. They are also a fair record of direction over time: the distance between what these systems could do two years ago and what they do now is real, and the benchmarks did track it.

What they will not do is tell you which of the two or three leading models belongs in your business. That comes down to price, how it handles your particular tasks, and whether the tooling fits the way you already work. If the money side is the sticking point, our note on what AI marketing actually costs a small business covers it in plain terms.

No. Above a certain level the leading models cluster so tightly that differences fall inside the margin of error, and the tests may overlap with training data. Treat a very low score as a warning sign, treat a very high score as unremarkable, and judge anything in the top group on your own tasks instead.

It means test questions ended up in the model's training data, so it has effectively seen the exam paper in advance. Because benchmarks are published openly and training sets are scraped from the web, this is hard to avoid completely. It inflates scores without improving real capability, which is why harder replacement benchmarks keep appearing.

Not really. What you need is a way to tell whether a tool does your work well, and five of your own prompts do that better than any published chart. Benchmarks are useful background if you follow the field, but no business owner has ever won a customer because they knew what GPQA measured.

SWE-bench and human preference rankings such as LMArena come nearest, because one uses genuine tasks from real projects and the other measures whether people prefer the result. Even then, neither tests customer emails, local marketing copy, or reading your own records, which is where most small business use actually sits.