Four open translation models, four different yardsticks

TranslateGemma, Hy-MT2, Index-Translate and PLaMo 3 Translate each report quality on different test sets, metrics and rivals, so their claims cannot be compared directly.

  • Analysis
  • 3 min read
  • Published

Four families of open translation models arrived within ten months: TranslateGemma from Google in January 2026, Hy-MT2 from Tencent in May, Index-Translate from Bilibili in September and PLaMo 3 Translate from Preferred Networks on 5 November. Each came with a report in which it does well. Read side by side, the reports show something else: no two of them measure quality the same way.

What each report measures

Model Languages Main scores Compared with
TranslateGemma 55 MetricX and COMET on WMT24++, human MQM on WMT25, MetricX on Vistra images Its own Gemma 3 baselines
Hy-MT2 33 General, business, domain and instruction-following tests, plus its own IFMTBench DeepSeek-V4-Pro, Kimi K2.6, Microsoft and Doubao APIs
Index-Translate 150 COMET-22 on FLORES and WMT24++, an LLM judge on WMT26, its own instTrans and MEME sets Hy-MT2, TranslateGemma, Qwen3.5, DeepSeek-V4.1-Flash, GPT-5.6-Sol, Gemini 3.5 Flash Lite
PLaMo 3 Translate 53 chrF++ on FLORES+, COMET-22 on WMT24++ English to Japanese, its own PFTB and SaaS sets TranslateGemma and Hy-MT2

The glossary entry on translation metrics explains MetricX, COMET, chrF++ and MQM. Every row is the developer’s own work, and most of the sets built in-house are not public.

One benchmark, three meanings

WMT24++ is the one test that appears in more than one report, and each report uses it differently. Google covers 55 languages and reports MetricX, where lower is better, and COMET on a 0 to 100 scale. Bilibili tests English into 20 languages with COMET-22 on a 0 to 1 scale. PFN uses only the English-to-Japanese track, and says the benchmark has short sentences and references it had concerns about, so it relied mainly on two sets of its own. PFTB has references written with several LLMs, and the SaaS set translates web pages that carry HTML tags. Both are scored by character overlap with the reference, which PFN argues works when the references are good.

Bilibili also built its own instTrans and MEME sets, and uses GPT-5.6-Sol as the judge for several of its scores. The benchmark entry explains why scores that developers report themselves need care, since the settings behind them are often chosen to show the model at its best.

Where the claims meet

Two reports include rivals from this group, so there is something to read. Bilibili’s table puts its 9B model ahead of Hy-MT2-7B and TranslateGemma-12B on eight of nine columns, by the widest margin on instruction following, which TranslateGemma’s documented template does not offer. Both rivals are slightly ahead on one reference-based IFMTBench score. TranslateGemma-12B also has a lower off-target rate in the report’s low-resource FLORES test, 1.7% against 4.0% for the 9B model.

IFMTBench is worth a note because Tencent built it. It is the only instruction-following test here that a rival also reports, and on it the Index-Translate 9B model scores 0.8760 against 0.8741 for Hy-MT2-7B, a gap too small to read as a win. PFN’s charts, in turn, place every PLaMo size ahead of both TranslateGemma and Hy-MT2 on Japanese-English and multilingual tests, but they leave out Index-Translate. Google’s report compares TranslateGemma only with its Gemma 3 baselines, and Tencent’s abstract names DeepSeek-V4-Pro, Kimi K2.6 and commercial APIs as its rivals. In the two tables that include rivals from this group, the rivals rank behind the developer’s own model, and no table includes all four. We found no independent comparison.

What to take from it

The differences you can check without a benchmark matter more:

  • Languages: 33 for Hy-MT2, 53 for PLaMo 3 Translate, 55 for TranslateGemma and 150 for Index-Translate.
  • Licenses: Apache 2.0 for Hy-MT2 and Index-Translate, the Gemma Terms of Use with a use policy for TranslateGemma, and the revenue-limited PLaMo Community License.
  • Limits and controls: TranslateGemma takes 2K tokens of input and a strict language-code template, PLaMo takes 32,768 tokens for input and output together, and Hy-MT2 and Index-Translate accept instructions for tags and glossaries, with tone control in Index-Translate’s case.
  • Small builds: a 0.57 GB 4-bit GGUF for PLaMo 0.8B, a 1.31 GB 4-bit GGUF for Index-Translate-2B and a 440 MB 1.25-bit build of Hy-MT2-1.8B.

Then translate 50 to 100 sentences of your own text with two or three candidates and have a speaker of the target language read them. Our translation models list and guide to choosing a translation model set out the steps, and the Index-Translate vs TranslateGemma page compares the two translators that accept images.

Sources

Models in this story

Model Released Parameters Context License Smallest variant needs
TranslateGemma Google DeepMind 4.3B – 27.4B 2K Gemma Terms (Conditional) 2.97 GB Laptop or phone
Hy-MT2 Tencent 2.04B – 30.1B 3B active 256K Apache 2.0 (Permissive) 1.41 GB Laptop or phone
Index-Translate Bilibili 2.27B – 36B 3B active 256K Apache 2.0 (Permissive) 1.44 GB Laptop or phone
PLaMo 3 Translate Preferred Networks 890M – 31.8B 32K PLaMo Community (Conditional) 0.614 GB Laptop or phone