Transformer models are good at translation, given appropriate training data (i.e., "this text but in multiple languages") – though you still have to watch out f

Transformer models are good at translation, given appropriate training data (i.e., "this text but in multiple languages") – though you still have to watch out for them translating "English" as "Deutsch", "Français", etc. Asking them to do repeated next token prediction isn't asking them to translate, though, especially not after the RLHF passes that OpenAI does to their ChatGPT models. When you test it, you get exactly the failure modes you'd expect: translations that start off okay, but go off on tangents; translations that "correct" the original text, so the translations aren't faithful; attempts (usually successful) to cover up a gap in "knowledge" that prevents the model from translating correctly. When considering these failure modes, which have come up every single time I've seen ChatGPT used for translation, it's clear that the simplistic system I described would work better. It'll output gibberish often (just as LibreTranslate does when given decontextualised Chinese and asked to translate to English), but that's better than a GPT model, which will just confabulate something in the same circumstance. The goal isn't "maximise the amount of successful translatedness": it's "reduce the language barrier as much as possible", something the benchmarks don't test.