3 ms·
The attention mechanism started as a simple trick to not use recurrent neural networks. Read the intro in the original paper "Attention is all you need" (https
by qigtyofhp 4y ago
The attention mechanism started as a simple trick to not use recurrent neural networks.
Read the intro in the original paper "Attention is all you need"
(https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762)
This video explains the drawbacks to RNNs and how transformers solve that: https://youtu.be/S27pHKBEp30?t=394 https://youtu.be/S27pHKBEp30?t=394
Andrej Karpathy explains attention here:
https://youtu.be/kCc8FmEb1nY?t=3719 https://youtu.be/kCc8FmEb1nY?t=3719
He explains how attention is seen as a communication network:
https://youtu.be/kCc8FmEb1nY?t=4298 https://youtu.be/kCc8FmEb1nY?t=4298
- godelski 4y ago> Read the intro in the original paper "Attention is all you need" I wouldn't call this the original "attention" paper. Definitely not the first paper to use the phrase. If you want clear proof of this, let's read the paper > Attention mechanisms have become an integral part of compelling sequence modeling and transduction models in various tasks, allowing modeling of dependencies without regard to their distance in the input or output sequences. I do think a lot of people's lack of understanding of attention is because they are so focused on DP(S)A that they miss a lot of the broader picture. And math. Not enough people dig into the math.