That's true, but specifically for translation I think all SOTA methods use task-specific embeddings (i.e., learn their own instead of using BERT or Elmo, etc.).

That's true, but specifically for translation I think all SOTA methods use task-specific embeddings (i.e., learn their own instead of using BERT or Elmo, etc.). See https://arxiv.org/pdf/1808.09381.pdf https://arxiv.org/pdf/1808.09381.pdf and note that the BERT paper doesn't mention translation as a metric.