would pre-training on non-English text (using a joint dictionary) improve translation performance? I'm not sure if the information on http://nlpprogress.com/en

would pre-training on non-English text (using a joint dictionary) improve translation performance? I'm not sure if the information on http://nlpprogress.com/english/machine_translation.html http://nlpprogress.com/english/machine_translation.html is accurate, but it appears that the top translation results rely on backtranslation, boosting and other data augmentation techniques with a vanilla transformer model. It would be interesting to see the Bleu scores for T5 that's more optimized for translation specifically.