3 ms·
The original Transformer, indeed for translation, was an encoder-decoder architecture. Most current LLMs, like Claude 3, are "decoder only". So in a sense the d
by jampekka 3y ago
The original Transformer, indeed for translation, was an encoder-decoder architecture. Most current LLMs, like Claude 3, are "decoder only". So in a sense the decoder-only translation is kinda accidental feature too.
- _giorgio_ 3y agoSo anything except encoder decoder translation is accidental? Crazy way to define research.
- jampekka 3y agoResearch, especially in academia, isn't usually that interested in the single concrete task that is under study. Translation is just one case where you need to find some good functions to map sequences to other sequences. The "Attention Is All You Need" paper frames the problem and contribution as: "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely." Translation is "just an experiment" for the general architecture and they study other tasks in the paper too. In a sense, all applications of basic research are "accidental".