3 ms·
Post starts with wrong statement right away: "The Transformer architecture revolutionized sequence modeling with its introduction of attention" Attention was
by riku_iki 11mo ago
Post starts with wrong statement right away:
"The Transformer architecture revolutionized sequence modeling with its introduction of attention"
Attention was developed before transformers.
- Alifatisk 11mo ago> Attention was developed before transformers. I just looked this up and it’s true, this changes the timeline I had in my mind completely! I thought the paper on Transformers is what also introduced the attention mechanism, but it existed before too and was applied on RNN encoder-decoder. Wow
- logicchains 11mo agoKnowing how such things go, it was probably invented by Schmidhuber in the 90s.