Benchmark

A benchmark is a standardized test used to compare models, such as a set of maths, coding or knowledge questions; scores are useful but easy to over-read.

  • Using models
  • Updated

Developers publish benchmark scores in their announcements, and independent groups run their own tests or collect human votes in arenas. A score reported by the developer is self-reported, and the settings behind it are often chosen to show the model at its best.

Benchmarks can be contaminated if test questions leak into the training data, and a high score does not guarantee that a model fits your task. This site labels developer-reported numbers as such, and recommends testing a shortlist of models on your own examples.

Related terms

  • Reasoning model A reasoning model works through a problem step by step in a 'thinking' phase, visible or hidden, before it gives its final answer.
  • AI agent An AI agent is a system in which a model plans and carries out multi-step tasks by calling tools, such as running code, searching the web or editing files.
  • Hallucination A hallucination is a confident but false or invented statement produced by an AI model.

Keep reading

All terms