How to evaluate AI models yourself: look past the benchmark charts

Benchmark scores in model announcements are mostly self-reported. Learn how to build a small test set, compare models fairly and judge quality, speed, memory and license for your task.

  • Intermediate
  • 4 min read
  • Published
  • Updated

Short answer. Treat published benchmark scores as hints, not as answers. Build 20 to 50 examples from your real work, run every candidate model with the same settings, and score the results with a simple rubric. Then compare speed, memory and license alongside quality.

Why benchmark charts can mislead

Developers publish benchmark scores in their announcements, and this site labels those numbers as self-reported wherever they appear. There are good reasons to be careful:

  • The developer picks the conditions. Our model pages repeat the same warning: for example, Moonshot’s results for Kimi K3.1 Pro were measured with harnesses it chose, and some of the gains for DeepSeek V4.1 Flash come from tuning for specific harnesses.
  • Test questions can leak. If benchmark questions end up in the training data, a high score may reflect memory rather than ability. This is called contamination.
  • Popular tests saturate. When many models score near the top, the test no longer separates them.
  • Your task is different. A model that excels at competition math may be mediocre at drafting your support emails.
  • Names move. Hosted model names can be re-pointed to newer models. DeepSeek’s old API names such as deepseek-v4-flash are now served by the new model, so a test you pinned to that name may change. Running a model yourself, with a fixed file, avoids this problem. Benchmark explains more.

Independent sources, and how to use them

Some groups run many models under identical settings and publish combined indexes, and arenas collect human votes on blind side-by-side answers. Developers themselves often cite such indexes, for example the Artificial Analysis Intelligence Index. These sources are more trustworthy than a single developer’s chart, but they still measure general tasks. Use them to build a shortlist of two or three models, then decide with your own tests.

Build a small test set

  1. Collect 20 to 50 real tasks from your actual work: real questions, real documents, real audio or real prompts.
  2. Cover the range. Include easy, typical and hard cases, and a few odd ones.
  3. Write down what good looks like for each: the correct answer, or a short list of criteria.
  4. Include cases where the right answer is “I don’t know” or a refusal, to see whether the model makes things up. Hallucinations are a main risk.
  5. Keep the set private and versioned, so published models cannot train on it and you can compare results over time.

Run a fair comparison

  • Use identical prompts, context and tools for every model.
  • Use the settings the developer recommends, such as the thinking mode or temperature on the model card, and the same settings for each run.
  • Test the build you will deploy. A 4-bit version can behave differently from the original, so evaluate the one you will actually use. Quantization explained shows why.
  • Run more than once. Models are not deterministic, so run important cases several times and look at the spread.
  • Keep the software fixed. A different serving engine or version can change speed and sometimes results.

Score the answers

Task How to score
Chat and writing Blind side-by-side reading by more than one person, with a rubric for accuracy, completeness, tone and format
Code Run unit tests or the program itself and record the pass rate
Extraction and structured output Field-by-field accuracy, and whether the output validates against your schema
Transcription Word error rate on your own audio, with special attention to names and numbers. Whisper can invent words that were never spoken
Search and retrieval Whether the right passage appears in the top results for your real queries
Images and video Blind side-by-side votes from several people, using the same prompts and seeds

If you use an AI model as a judge, give it a clear rubric and check a sample of its scores by hand. Judge models tend to prefer longer answers and answers that look like their own.

Measure more than quality

A model that is slightly better but ten times slower or too big for your hardware is the wrong choice. Record for each candidate:

  • Speed: time to the first token and tokens per second, on your hardware and with your usual context length.
  • Memory: the peak memory use, including the KV cache for your longest inputs.
  • Cost: hardware, electricity or hosting.
  • License: whether it allows your use. See AI licenses explained.
  • Reliability: how much results vary between runs and whether it follows your format.

A simple scorecard

Give each criterion a weight that reflects your priorities, then score every candidate from 1 to 5. For example:

Criterion Weight
Quality on your examples 50%
Speed on your hardware 20%
Memory fit and cost 15%
License fit 10%
Reliability 5%

The weights are yours to change. What matters is that you decide them before you see the results, so the scorecard does not simply confirm a favorite.

Keep it going

Save your test set and your results. When a promising new model appears, or you change the quantization, the prompt or the serving software, run the set again. A good test set is a small asset that pays off every time you choose a model, and how to choose an open AI model shows where it fits in the process.

Frequently asked questions

How many test examples do I need?
Twenty to fifty real examples already reveal large differences between candidates. Use a hundred or more when the models are close or the stakes are high.
Can I use an AI model to grade another model's answers?
Yes, as a first pass, with care. Judge models tend to favor longer answers and answers in their own style, so give them a clear rubric and check a sample of their grades yourself.
Which benchmark should I trust?
No single one. Prefer independent tests that run every model under the same conditions, use them to build a shortlist, and make the final decision on your own examples.
Why do my results differ from the model card?
Developers choose the prompts, sampling settings, tools and software that show the model at its best. Different quantization, a different serving engine or a different prompt can change the result.
How often should I re-test?
Whenever you change the model version, the quantization, the prompt or the serving software, and when a promising new release appears.

Key terms in this guide

  • Benchmark A benchmark is a standardized test used to compare models, such as a set of maths, coding or knowledge questions; scores are useful but easy to over-read.
  • Embedding model An embedding model turns text, images or other data into a list of numbers that captures its meaning, so that similar items end up close together.
  • Hallucination A hallucination is a confident but false or invented statement produced by an AI model.
  • Inference Inference is the act of running a trained model to get an output, such as answering a prompt, as opposed to training it.
  • Quantization Quantization stores a model's weights with fewer bits, for example 4 instead of 16, to cut memory use and speed up inference at a small cost in quality.

Models mentioned in this guide

Model Released Parameters Context License Smallest variant needs
Kimi K3.1 Pro Moonshot AI 2.8T 104B active 1M Kimi K3 License (Conditional) 1.72 TB Multi-node cluster
DeepSeek V4.1 Flash DeepSeek 552B 16B active 1M MIT (Permissive) 561 GB GPU server
Ling-3.1-flash inclusionAI (Ant Group) 560B 25B active 1M MIT (Permissive) 386 GB Big-memory desktop
Gemma 4 Google DeepMind 5.1B – 30.7B 256K Apache 2.0 (Permissive) 3.52 GB Laptop or phone
Whisper large-v3-turbo OpenAI 810M – 1.54B – MIT (Permissive) 6 GB Laptop or phone
Qwen3.8-27B Alibaba Qwen 27.8B 256K Apache 2.0 (Permissive) 19 GB Gaming GPU