3 ms·
Attention is just a weighted sum over a set of vectors, where the weights sum to one. Attention weights are usually created by neural nets. The word "attention"
by visarga 9y ago
Attention is just a weighted sum over a set of vectors, where the weights sum to one. Attention weights are usually created by neural nets. The word "attention" might seem more grandiose than what it actually does.
- speedplane 9y agoThat may be true on a mathematical level, but that's also the answer to just about any neural net question... it's all just a a weighted sum. My understanding of "attention" on a higher level is the ability to concentrate more neurons on "important" areas of an image than less important ones. An imperfect analogy is how the human visual system has better resolution at your eye-line's center than at it's edges. In this analogy, your brain should not waste effort processing image details in your peripheral vision.
- visarga 9y agoThe key element is that we use neural nets to compute the attention weights, so attention itself is learnable.