An AI benchmark is a fixed exam, with questions or tasks whose answers are known, given to several models to compare them by the same yardstick. The result is usually a percentage of correct answers. It gives a quick idea, but it measures those tasks, not necessarily yours.
A question from the app. Pick an answer.
A benchmark table says A is better. How do you choose for your task?
That table measures other tasks: it may not reflect yours.
Price does not guarantee it does better on your task. You check.
Your cases tell you how many attempts each needs and what each solved task costs.
The word is used elsewhere too. In business, benchmarking means comparing yourself with competitors to copy what works. In computers and phones, a benchmark is a speed test for the processor or the graphics card. This page is about the artificial intelligence kind: the test given to an AI model.
Someone builds hundreds or thousands of questions with their correct answer: math problems, medical questions, code bugs to fix, tasks in a terminal. Each model answers the same ones, under the same conditions, and the correct answers are counted. That is why every launch comes with a chart: "our model scores 80% where the other scores 74%".
First, it measures other tasks: being the best at olympiad problems says nothing about how it reads your invoices. Second, the questions of an old benchmark may have ended up in the training data, and then the model remembers instead of reasoning. Third, the launch figure is measured by the provider itself, with its own setup: more effort, several attempts or tailored instructions.
And when every model is close to the ceiling, the benchmark no longer tells anyone apart. That is why new benchmarks appear every few months.
Use it as a filter, not a verdict: it tells you which two or three models are worth trying. Then build twenty or thirty real cases of your task, with the answer you expect, and run them on each one. Count hits, attempts and what each correctly solved task costs. That is your benchmark, and it is the only one that measures what matters to you.