3 ms·

I'm seconding this. I could not find a good resource to understand what "attention" actually _is_. (The next step for me would be to follow the citation trail

I'm seconding this. I could not find a good resource to understand what "attention" actually _is_. (The next step for me would be to follow the citation trail to the original paper, but that might not be the best place to come to an understanding of the thing.)

visarga 9y ago
Attention is just a weighted sum over a set of vectors, where the weights sum to one. Attention weights are usually created by neural nets. The word "attention" might seem more grandiose than what it actually does.
- speedplane 9y ago
  That may be true on a mathematical level, but that's also the answer to just about any neural net question... it's all just a a weighted sum. My understanding of "attention" on a higher level is the ability to concentrate more neurons on "important" areas of an image than less important ones. An imperfect analogy is how the human visual system has better resolution at your eye-line's center than at it's edges. In this analogy, your brain should not waste effort processing image details in your peripheral vision.
  - visarga 9y ago
    The key element is that we use neural nets to compute the attention weights, so attention itself is learnable.
mattj 9y ago
The other answers cover the math well, but I think the “why do you need attention?” statement is worth making (and answers the more engineering-y question of “how/when?”): DNNs typically operate on fixed-size tensors (often with a variable batch size, which you can safely ignore). In order to incorporate a non-fixed size tensor, you need some way of converting it into a fixed size. For example, processing a sentence of variable length into a single prediction value. You have many choices for methods of combining the tensors from each token in the sentence - max, min, mean, median, sum, etc etc. Attention is a weighted mean, where the weights are computed based on a query, key, and value. The query might represent something you know about the sentence or the context (“this is a sentence from a toaster review”), the key represents something you know about each token (“this is the word embedding tensor”), and the value is the tensor you want to use for the weighted mean.

visarga 9y ago

Attention is just a weighted sum over a set of vectors, where the weights sum to one. Attention weights are usually created by neural nets. The word "attention" might seem more grandiose than what it actually does.

speedplane 9y ago
That may be true on a mathematical level, but that's also the answer to just about any neural net question... it's all just a a weighted sum. My understanding of "attention" on a higher level is the ability to concentrate more neurons on "important" areas of an image than less important ones. An imperfect analogy is how the human visual system has better resolution at your eye-line's center than at it's edges. In this analogy, your brain should not waste effort processing image details in your peripheral vision.
- visarga 9y ago
  The key element is that we use neural nets to compute the attention weights, so attention itself is learnable.

mattj 9y ago

The other answers cover the math well, but I think the “why do you need attention?” statement is worth making (and answers the more engineering-y question of “how/when?”): DNNs typically operate on fixed-size tensors (often with a variable batch size, which you can safely ignore). In order to incorporate a non-fixed size tensor, you need some way of converting it into a fixed size. For example, processing a sentence of variable length into a single prediction value. You have many choices for methods of combining the tensors from each token in the sentence - max, min, mean, median, sum, etc etc. Attention is a weighted mean, where the weights are computed based on a query, key, and value. The query might represent something you know about the sentence or the context (“this is a sentence from a toaster review”), the key represents something you know about each token (“this is the word embedding tensor”), and the value is the tensor you want to use for the weighted mean.