4 ms·
What made attention mechanism click to me was the video https://www.youtube.com/watch?v=0PjHri8tc1c https://www.youtube.com/watch?v=0PjHri8tc1c by Sebastian Ras
by metanonsense 3y ago
What made attention mechanism click to me was the video https://www.youtube.com/watch?v=0PjHri8tc1c https://www.youtube.com/watch?v=0PjHri8tc1c by Sebastian Raschka. Another good video on attention that tries to give a bit more intuition on the query/key/value stuff is https://www.youtube.com/watch?v=OyFJWRnt_AY https://www.youtube.com/watch?v=OyFJWRnt_AY
In hindsight, what helped me most is understanding the dimensionality at each step of the attention computation (explained well in Raschka's video). Another think I simply did not get from the original Transformer paper is that the learning of self-attention happens in the linear layers. (At least, that's my current understanding. If that's wrong, I appreciate any correction :)
- PartiallyTyped 3y ago> Another think I simply did not get from the original Transformer paper is that the learning of self-attention happens in the linear layers. You can replace the KVQ kernels with any parametric computation that allows you to pass gradients through, and you will have learning. I think some newer language model architectures use residual blocks here, and some vision transformers use FeedForward networks for the kernels.
- api 3y agoThanks. One thought I’ve had is to wonder if attention isn’t kind of like a way to unroll RNNs to make training more efficient. Any truth in that?
- opyate 3y agoI thoroughly recommend Sebastian's newsletter "Ahead Of AI". Link on his website: https://sebastianraschka.com/ https://sebastianraschka.com/