Translation metrics
Translation metrics are the scores that model cards use to rate a translation. Some count overlap with a human reference, others use a trained model or human reviewers, and their scales cannot be compared.
Older metrics count overlap. BLEU compares word sequences with a human reference and chrF++ compares character sequences, so a correct translation worded differently loses points. Newer metrics use a neural model trained on human ratings. COMET, and its XCOMET variant, scores the translation against the source and a reference. Google's MetricX works the same way, but lower is better. Quality-estimation versions such as MetricX-QE need no reference, and Google used MetricX-QE and AutoMQM, which asks a language model to mark errors like a human reviewer, as reward models when training TranslateGemma. MQM is the human version: reviewers mark errors and their penalties add up, so lower is better. Some cards also use a large language model as the judge, as the Index-Translate report does with GPT-5.6-Sol for its WMT26 judge score and its instruction-following tests.
Compare numbers with care. TranslateGemma's card gives COMET on a 0 to 100 scale, 83.5 for the 12B model on WMT24++ across 55 languages. The Index-Translate report uses a 0 to 1 scale and tests English into 20 languages for WMT24++, so its 0.8601 for the 9B model is a different measurement. PLaMo 3 Translate reports chrF++ and an internal Japanese-English set. A developer's own tables also choose which rival models and settings to include. Because we found no independent leaderboard that covers these models, translate 50 to 100 sentences of your own text in your own language pairs and have a speaker of the target language read them.
Related terms
- Benchmark A benchmark is a standardized test used to compare models, such as a set of maths, coding or knowledge questions; scores are useful but easy to over-read.
- Machine translation Machine translation is software that translates text, speech or text inside images from one language to another. The open models of today usually start from a general model and are then trained only to translate.
- Word error rate (WER) Word error rate is the standard score for speech recognition: the share of words in a transcript that are wrong, missing or extra compared with a correct reference.