4 ms·
Didn't "Attention Is All you Need" bill transformers primarily as a translation model?
by herculity275 2y ago
Didn't "Attention Is All you Need" bill transformers primarily as a translation model?
- Alifatisk 2y agoYes, it was primarly for translation. I don’t know how OP come to the conclusion that it was accidental
- magoghm 2y agoI guess that what OP meant is that LLMs are not specifically trained to do translation, they learn to do it as a side effect.
- hackmaxim 2y agoYes, but the translation model in the “Attention is All You Need” was trained on parallel sentences (input: English sentence, output: French). LLMs such as Claude are not trained on any labeled data. Yet, they still surpass the special-case models.
- londons_explore 2y agoThey surpass because there is faaaaaaar more language data generally than there is data that's paired english-french.
- authorfly 2y agoAnd the key takeaway of the T5 paper was also how "instructions" could vary massively. But the main instructions were two types - summarise x, and translate y to z. Translation in many ways was the root of seeing that multiple tasks/instructions could fit a model and not just in the context of multiple training loss methods, like NSP vs Gap filling(a modus of training difference, not actual task itself). And the special tokens for BERT etc trace their origins back to enabling the task above and positional embeddings(and encodings). Which largely trace their origin to translation work.
- xanderlewis 2y agoAs far as I know, it wasn’t just primarily for translation — it was entirely intended for machine translation. It only because clear later on that it was a more generally applicable architecture.