3 ms·
This is basically how RNN encoder/decoder architectures worked. The encoder encoded the input as a single vector and the decoder would decode this into text (eg
by microtonal 3y ago
This is basically how RNN encoder/decoder architectures worked. The encoder encoded the input as a single vector and the decoder would decode this into text (eg. for machine translation) [1]. However, fixed-length vectors generally required too much 'compression' to represent variable-length text, so people started adding attention mechanisms so that the decoder could also attend to the input text. And the seminal Transformer paper by Vaswani and others showed that you only need an attention mechanism and you could ditch the RNN (hence the title 'Attention is all you need') and here we are.
So, this has been possible already for quite a long time.
[1] https://arxiv.org/pdf/1409.3215.pdf https://arxiv.org/pdf/1409.3215.pdf
- sdenton4 3y agoTo be sure, the seq2seq style already allowed multi length embeddings before transformers were a thing. RNN vs Transformer is entirely an implementation choice; you can build a seq2seq model with any combination of transformer, RNN, and conventional layers. Each of these layer types have different computational costs for training and inference, and encode different inductive biases, which may be more or less appropriate to a given problem.