How to evaluate AI models yourself: look past the benchmark charts
Benchmark scores in model announcements are mostly self-reported. Learn how to build a small test set, compare models fairly and judge quality, speed, memory and license for your task.
Short answer. Treat published benchmark scores as hints, not as answers. Build 20 to 50 examples from your real work, run every candidate model with the same settings, and score the results with a simple rubric. Then compare speed, memory and license alongside quality.
Why benchmark charts can mislead
Developers publish benchmark scores in their announcements, and this site labels those numbers as self-reported wherever they appear. There are good reasons to be careful:
- The developer picks the conditions. Our model pages repeat the same warning: for example, Moonshot’s results for Kimi K3.1 Pro were measured with harnesses it chose, and some of the gains for DeepSeek V4.1 Flash come from tuning for specific harnesses.
- Test questions can leak. If benchmark questions end up in the training data, a high score may reflect memory rather than ability. This is called contamination.
- Popular tests saturate. When many models score near the top, the test no longer separates them.
- Your task is different. A model that excels at competition math may be mediocre at drafting your support emails.
- Names move. Hosted model names can be re-pointed to newer models. DeepSeek’s old API names such as deepseek-v4-flash are now served by the new model, so a test you pinned to that name may change. Running a model yourself, with a fixed file, avoids this problem. Benchmark explains more.
Independent sources, and how to use them
Some groups run many models under identical settings and publish combined indexes, and arenas collect human votes on blind side-by-side answers. Developers themselves often cite such indexes, for example the Artificial Analysis Intelligence Index. These sources are more trustworthy than a single developer’s chart, but they still measure general tasks. Use them to build a shortlist of two or three models, then decide with your own tests.
Build a small test set
- Collect 20 to 50 real tasks from your actual work: real questions, real documents, real audio or real prompts.
- Cover the range. Include easy, typical and hard cases, and a few odd ones.
- Write down what good looks like for each: the correct answer, or a short list of criteria.
- Include cases where the right answer is “I don’t know” or a refusal, to see whether the model makes things up. Hallucinations are a main risk.
- Keep the set private and versioned, so published models cannot train on it and you can compare results over time.
Run a fair comparison
- Use identical prompts, context and tools for every model.
- Use the settings the developer recommends, such as the thinking mode or temperature on the model card, and the same settings for each run.
- Test the build you will deploy. A 4-bit version can behave differently from the original, so evaluate the one you will actually use. Quantization explained shows why.
- Run more than once. Models are not deterministic, so run important cases several times and look at the spread.
- Keep the software fixed. A different serving engine or version can change speed and sometimes results.
Score the answers
| Task | How to score |
|---|---|
| Chat and writing | Blind side-by-side reading by more than one person, with a rubric for accuracy, completeness, tone and format |
| Code | Run unit tests or the program itself and record the pass rate |
| Extraction and structured output | Field-by-field accuracy, and whether the output validates against your schema |
| Transcription | Word error rate on your own audio, with special attention to names and numbers. Whisper can invent words that were never spoken |
| Search and retrieval | Whether the right passage appears in the top results for your real queries |
| Images and video | Blind side-by-side votes from several people, using the same prompts and seeds |
If you use an AI model as a judge, give it a clear rubric and check a sample of its scores by hand. Judge models tend to prefer longer answers and answers that look like their own.
Measure more than quality
A model that is slightly better but ten times slower or too big for your hardware is the wrong choice. Record for each candidate:
- Speed: time to the first token and tokens per second, on your hardware and with your usual context length.
- Memory: the peak memory use, including the KV cache for your longest inputs.
- Cost: hardware, electricity or hosting.
- License: whether it allows your use. See AI licenses explained.
- Reliability: how much results vary between runs and whether it follows your format.
A simple scorecard
Give each criterion a weight that reflects your priorities, then score every candidate from 1 to 5. For example:
| Criterion | Weight |
|---|---|
| Quality on your examples | 50% |
| Speed on your hardware | 20% |
| Memory fit and cost | 15% |
| License fit | 10% |
| Reliability | 5% |
The weights are yours to change. What matters is that you decide them before you see the results, so the scorecard does not simply confirm a favorite.
Keep it going
Save your test set and your results. When a promising new model appears, or you change the quantization, the prompt or the serving software, run the set again. A good test set is a small asset that pays off every time you choose a model, and how to choose an open AI model shows where it fits in the process.
Frequently asked questions
- How many test examples do I need?
- Twenty to fifty real examples already reveal large differences between candidates. Use a hundred or more when the models are close or the stakes are high.
- Can I use an AI model to grade another model's answers?
- Yes, as a first pass, with care. Judge models tend to favor longer answers and answers in their own style, so give them a clear rubric and check a sample of their grades yourself.
- Which benchmark should I trust?
- No single one. Prefer independent tests that run every model under the same conditions, use them to build a shortlist, and make the final decision on your own examples.
- Why do my results differ from the model card?
- Developers choose the prompts, sampling settings, tools and software that show the model at its best. Different quantization, a different serving engine or a different prompt can change the result.
- How often should I re-test?
- Whenever you change the model version, the quantization, the prompt or the serving software, and when a promising new release appears.
Key terms in this guide
- Benchmark A benchmark is a standardized test used to compare models, such as a set of maths, coding or knowledge questions; scores are useful but easy to over-read.
- Embedding model An embedding model turns text, images or other data into a list of numbers that captures its meaning, so that similar items end up close together.
- Hallucination A hallucination is a confident but false or invented statement produced by an AI model.
- Inference Inference is the act of running a trained model to get an output, such as answering a prompt, as opposed to training it.
- Quantization Quantization stores a model's weights with fewer bits, for example 4 instead of 16, to cut memory use and speed up inference at a small cost in quality.
Models mentioned in this guide
| Model | Released | Parameters | Context | License | Smallest variant needs |
|---|---|---|---|---|---|
| Kimi K3.1 Pro Moonshot AI | 2.8T 104B active | 1M | Kimi K3 License (Conditional) | 1.72 TB Multi-node cluster | |
| DeepSeek V4.1 Flash DeepSeek | 552B 16B active | 1M | MIT (Permissive) | 561 GB GPU server | |
| Ling-3.1-flash inclusionAI (Ant Group) | 560B 25B active | 1M | MIT (Permissive) | 386 GB Big-memory desktop | |
| Gemma 4 Google DeepMind | 5.1B – 30.7B | 256K | Apache 2.0 (Permissive) | 3.52 GB Laptop or phone | |
| Whisper large-v3-turbo OpenAI | 810M – 1.54B | – | MIT (Permissive) | 6 GB Laptop or phone | |
| Qwen3.8-27B Alibaba Qwen | 27.8B | 256K | Apache 2.0 (Permissive) | 19 GB Gaming GPU |