Four open translation models, four different yardsticks
TranslateGemma, Hy-MT2, Index-Translate and PLaMo 3 Translate each report quality on different test sets, metrics and rivals, so their claims cannot be compared directly.
Four families of open translation models arrived within ten months: TranslateGemma from Google in January 2026, Hy-MT2 from Tencent in May, Index-Translate from Bilibili in September and PLaMo 3 Translate from Preferred Networks on 5 November. Each came with a report in which it does well. Read side by side, the reports show something else: no two of them measure quality the same way.
What each report measures
| Model | Languages | Main scores | Compared with |
|---|---|---|---|
| TranslateGemma | 55 | MetricX and COMET on WMT24++, human MQM on WMT25, MetricX on Vistra images | Its own Gemma 3 baselines |
| Hy-MT2 | 33 | General, business, domain and instruction-following tests, plus its own IFMTBench | DeepSeek-V4-Pro, Kimi K2.6, Microsoft and Doubao APIs |
| Index-Translate | 150 | COMET-22 on FLORES and WMT24++, an LLM judge on WMT26, its own instTrans and MEME sets | Hy-MT2, TranslateGemma, Qwen3.5, DeepSeek-V4.1-Flash, GPT-5.6-Sol, Gemini 3.5 Flash Lite |
| PLaMo 3 Translate | 53 | chrF++ on FLORES+, COMET-22 on WMT24++ English to Japanese, its own PFTB and SaaS sets | TranslateGemma and Hy-MT2 |
The glossary entry on translation metrics explains MetricX, COMET, chrF++ and MQM. Every row is the developer’s own work, and most of the sets built in-house are not public.
One benchmark, three meanings
WMT24++ is the one test that appears in more than one report, and each report uses it differently. Google covers 55 languages and reports MetricX, where lower is better, and COMET on a 0 to 100 scale. Bilibili tests English into 20 languages with COMET-22 on a 0 to 1 scale. PFN uses only the English-to-Japanese track, and says the benchmark has short sentences and references it had concerns about, so it relied mainly on two sets of its own. PFTB has references written with several LLMs, and the SaaS set translates web pages that carry HTML tags. Both are scored by character overlap with the reference, which PFN argues works when the references are good.
Bilibili also built its own instTrans and MEME sets, and uses GPT-5.6-Sol as the judge for several of its scores. The benchmark entry explains why scores that developers report themselves need care, since the settings behind them are often chosen to show the model at its best.
Where the claims meet
Two reports include rivals from this group, so there is something to read. Bilibili’s table puts its 9B model ahead of Hy-MT2-7B and TranslateGemma-12B on eight of nine columns, by the widest margin on instruction following, which TranslateGemma’s documented template does not offer. Both rivals are slightly ahead on one reference-based IFMTBench score. TranslateGemma-12B also has a lower off-target rate in the report’s low-resource FLORES test, 1.7% against 4.0% for the 9B model.
IFMTBench is worth a note because Tencent built it. It is the only instruction-following test here that a rival also reports, and on it the Index-Translate 9B model scores 0.8760 against 0.8741 for Hy-MT2-7B, a gap too small to read as a win. PFN’s charts, in turn, place every PLaMo size ahead of both TranslateGemma and Hy-MT2 on Japanese-English and multilingual tests, but they leave out Index-Translate. Google’s report compares TranslateGemma only with its Gemma 3 baselines, and Tencent’s abstract names DeepSeek-V4-Pro, Kimi K2.6 and commercial APIs as its rivals. In the two tables that include rivals from this group, the rivals rank behind the developer’s own model, and no table includes all four. We found no independent comparison.
What to take from it
The differences you can check without a benchmark matter more:
- Languages: 33 for Hy-MT2, 53 for PLaMo 3 Translate, 55 for TranslateGemma and 150 for Index-Translate.
- Licenses: Apache 2.0 for Hy-MT2 and Index-Translate, the Gemma Terms of Use with a use policy for TranslateGemma, and the revenue-limited PLaMo Community License.
- Limits and controls: TranslateGemma takes 2K tokens of input and a strict language-code template, PLaMo takes 32,768 tokens for input and output together, and Hy-MT2 and Index-Translate accept instructions for tags and glossaries, with tone control in Index-Translate’s case.
- Small builds: a 0.57 GB 4-bit GGUF for PLaMo 0.8B, a 1.31 GB 4-bit GGUF for Index-Translate-2B and a 440 MB 1.25-bit build of Hy-MT2-1.8B.
Then translate 50 to 100 sentences of your own text with two or three candidates and have a speaker of the target language read them. Our translation models list and guide to choosing a translation model set out the steps, and the Index-Translate vs TranslateGemma page compares the two translators that accept images.
Sources
- TranslateGemma Technical Report (arXiv 2601.09012) arxiv.org
- Hy-MT2: A Family of Fast, Efficient and Powerful Multilingual Translation Models in the Wild (arXiv 2605.22064) arxiv.org
- Index-Translate-9B model card (Hugging Face) huggingface.co
- Index-Translate: A Multilingual Translation Model Family (arXiv 2609.40181) arxiv.org
- PLaMo 3 Translate: World-Class Translation with Open Weights (PFN blog, 5 November 2026) preferred.jp
Models in this story
| Model | Released | Parameters | Context | License | Smallest variant needs |
|---|---|---|---|---|---|
| TranslateGemma Google DeepMind | 4.3B – 27.4B | 2K | Gemma Terms (Conditional) | 2.97 GB Laptop or phone | |
| Hy-MT2 Tencent | 2.04B – 30.1B 3B active | 256K | Apache 2.0 (Permissive) | 1.41 GB Laptop or phone | |
| Index-Translate Bilibili | 2.27B – 36B 3B active | 256K | Apache 2.0 (Permissive) | 1.44 GB Laptop or phone | |
| PLaMo 3 Translate Preferred Networks | 890M – 31.8B | 32K | PLaMo Community (Conditional) | 0.614 GB Laptop or phone |