Benchmark
A benchmark is a standardized test used to compare models, such as a set of maths, coding or knowledge questions; scores are useful but easy to over-read.
Developers publish benchmark scores in their announcements, and independent groups run their own tests or collect human votes in arenas. A score reported by the developer is self-reported, and the settings behind it are often chosen to show the model at its best.
Benchmarks can be contaminated if test questions leak into the training data, and a high score does not guarantee that a model fits your task. This site labels developer-reported numbers as such, and recommends testing a shortlist of models on your own examples.
Related terms
- Reasoning model A reasoning model works through a problem step by step in a 'thinking' phase, visible or hidden, before it gives its final answer.
- AI agent An AI agent is a system in which a model plans and carries out multi-step tasks by calling tools, such as running code, searching the web or editing files.
- Hallucination A hallucination is a confident but false or invented statement produced by an AI model.
Keep reading
- How to choose an open AI model: a practical decision guide
- How to choose an open-source speech-to-text model
- How to choose an open-source text-to-speech model
- How to choose an open-source translation model
- How to choose an open-source OCR model for PDFs and scans
- How to evaluate AI models yourself: look past the benchmark charts
- How to pick an open model for a coding agent