We benchmarked TranslateGemma-12b against 5 frontier LLMs on subtitle translation - it won across the board, with one significant catch

A study benchmarked TranslateGemma-12b against five frontier LLMs on subtitle translation for six language pairs, showing the task-specific model consistently outperformed general-purpose models. While initial numbers indicated a clear win, human QA added a significant catch which will be detailed in the full report.