5 ms·

Can somebody assist in breaking this down?

by ferros 9y ago

RangerScience 9y ago
I'm seconding this. I could not find a good resource to understand what "attention" actually _is_. (The next step for me would be to follow the citation trail to the original paper, but that might not be the best place to come to an understanding of the thing.)
- visarga 9y ago
  Attention is just a weighted sum over a set of vectors, where the weights sum to one. Attention weights are usually created by neural nets. The word "attention" might seem more grandiose than what it actually does.
  - speedplane 9y ago
    That may be true on a mathematical level, but that's also the answer to just about any neural net question... it's all just a a weighted sum. My understanding of "attention" on a higher level is the ability to concentrate more neurons on "important" areas of an image than less important ones. An imperfect analogy is how the human visual system has better resolution at your eye-line's center than at it's edges. In this analogy, your brain should not waste effort processing image details in your peripheral vision.
    - visarga 9y ago
      The key element is that we use neural nets to compute the attention weights, so attention itself is learnable.
- mattj 9y ago
  The other answers cover the math well, but I think the “why do you need attention?” statement is worth making (and answers the more engineering-y question of “how/when?”): DNNs typically operate on fixed-size tensors (often with a variable batch size, which you can safely ignore). In order to incorporate a non-fixed size tensor, you need some way of converting it into a fixed size. For example, processing a sentence of variable length into a single prediction value. You have many choices for methods of combining the tensors from each token in the sentence - max, min, mean, median, sum, etc etc. Attention is a weighted mean, where the weights are computed based on a query, key, and value. The query might represent something you know about the sentence or the context (“this is a sentence from a toaster review”), the key represents something you know about each token (“this is the word embedding tensor”), and the value is the tensor you want to use for the weighted mean.
visarga 9y ago
It shows that multiple attention heads can replace CNN's and RNN's, creating a powerful new paradigm for neural nets.
- speedplane 9y ago
  It just seems so random. I understand that CNNs are better suited to process pixels in a 2D grid, it sort of makes sense b/c images are in a 2D grid and the operations emulate convolutions used in lots of image processing. I also understand that RNNs used to process symbols, and their feedback mechanisms can emulate short-term memory. Still... even though I understand what works and what doesn't, I don't really understand why RNNs are better at processing symbols and CNNs are better at processing images. They hold the same information, and work the same way, they are just organized differently. It makes little sense.
  - candiodari 9y ago
    Think of it this way: Neural networks are a particular kind of formula. For instance a 2x2 neural network takes 2 inputs, x, and y, has 2 biases, bx and by, and 4 weights w1 ... w4 and calculates 2 outputs ox, and oy. The formula it uses is: ox = w1 * x + w3 * y + bx oy = w2 * x + w4 * y + by And then learns how to change w1 ... w4, bx and by to get ox and oy to be more useful (less error). Is it so weird that a 2 layer network (which is the same formula, but it uses ox and oy as inputs to a second identical calculation to produce the final outputs) will produce better results, even with learning ? Even in a neural network so absurdly large as the human mind, it isn't just a bunch of wires connecting everything to everything. There are clear patterns, clear formulas that are dictated not by learning, but by the architecture itself (e.g. CNNs in the eyes and optical cortex, and something much more like LSTMs in memory heavy regions). In some sense one might even say it doesn't make a difference. A fully connected network with the same fanout as a CNN can do everything a CNN can, and more. Likewise, a network that is simply presented with the last 50 timesteps can do strictly better than an LSTM or GRU RNN would. But such a network would have much, much greater computational complexity (by a factor in the thousands at least, in the case of those CNNs by a factor that is roughly the number of pixels in the image). So they don't work better per se. In fact they are "worse" by the most important metric (error). But they are a good trade off : LSTMs are thousands of times faster, with only slightly worse results for sequence labeling or production (ie. text and audio comprehension). CNNs are millions of times faster at image comprehension than a fully connected network, and are only slightly worse at it.
nl 9y ago
The distill.pub explanation is pretty good https://distill.pub/2016/augmented-rnns/ https://distill.pub/2016/augmented-rnns/
xgb84j 9y ago
When you want to process a sequence of vectors while incorporating information from a second sequence, you use attention. Attention means you create a weighted sum of all vectors in the second sequence for each vector in the first sequence. You basically add a customized summary vector of the second sequence to each vector of the first.

Can somebody assist in breaking this down?

by ferros 9y ago

Can somebody assist in breaking this down?

I'm seconding this. I could not find a good resource to understand what "attention" actually _is_. (The next step for me would be to follow the citation trail to the original paper, but that might not be the best place to come to an understanding of the thing.)

visarga 9y ago
Attention is just a weighted sum over a set of vectors, where the weights sum to one. Attention weights are usually created by neural nets. The word "attention" might seem more grandiose than what it actually does.
- speedplane 9y ago
  That may be true on a mathematical level, but that's also the answer to just about any neural net question... it's all just a a weighted sum. My understanding of "attention" on a higher level is the ability to concentrate more neurons on "important" areas of an image than less important ones. An imperfect analogy is how the human visual system has better resolution at your eye-line's center than at it's edges. In this analogy, your brain should not waste effort processing image details in your peripheral vision.
  - visarga 9y ago
    The key element is that we use neural nets to compute the attention weights, so attention itself is learnable.
mattj 9y ago
The other answers cover the math well, but I think the “why do you need attention?” statement is worth making (and answers the more engineering-y question of “how/when?”): DNNs typically operate on fixed-size tensors (often with a variable batch size, which you can safely ignore). In order to incorporate a non-fixed size tensor, you need some way of converting it into a fixed size. For example, processing a sentence of variable length into a single prediction value. You have many choices for methods of combining the tensors from each token in the sentence - max, min, mean, median, sum, etc etc. Attention is a weighted mean, where the weights are computed based on a query, key, and value. The query might represent something you know about the sentence or the context (“this is a sentence from a toaster review”), the key represents something you know about each token (“this is the word embedding tensor”), and the value is the tensor you want to use for the weighted mean.

visarga 9y ago

It shows that multiple attention heads can replace CNN's and RNN's, creating a powerful new paradigm for neural nets.

speedplane 9y ago
It just seems so random. I understand that CNNs are better suited to process pixels in a 2D grid, it sort of makes sense b/c images are in a 2D grid and the operations emulate convolutions used in lots of image processing. I also understand that RNNs used to process symbols, and their feedback mechanisms can emulate short-term memory. Still... even though I understand what works and what doesn't, I don't really understand why RNNs are better at processing symbols and CNNs are better at processing images. They hold the same information, and work the same way, they are just organized differently. It makes little sense.
- candiodari 9y ago
  Think of it this way: Neural networks are a particular kind of formula. For instance a 2x2 neural network takes 2 inputs, x, and y, has 2 biases, bx and by, and 4 weights w1 ... w4 and calculates 2 outputs ox, and oy. The formula it uses is: ox = w1 * x + w3 * y + bx oy = w2 * x + w4 * y + by And then learns how to change w1 ... w4, bx and by to get ox and oy to be more useful (less error). Is it so weird that a 2 layer network (which is the same formula, but it uses ox and oy as inputs to a second identical calculation to produce the final outputs) will produce better results, even with learning ? Even in a neural network so absurdly large as the human mind, it isn't just a bunch of wires connecting everything to everything. There are clear patterns, clear formulas that are dictated not by learning, but by the architecture itself (e.g. CNNs in the eyes and optical cortex, and something much more like LSTMs in memory heavy regions). In some sense one might even say it doesn't make a difference. A fully connected network with the same fanout as a CNN can do everything a CNN can, and more. Likewise, a network that is simply presented with the last 50 timesteps can do strictly better than an LSTM or GRU RNN would. But such a network would have much, much greater computational complexity (by a factor in the thousands at least, in the case of those CNNs by a factor that is roughly the number of pixels in the image). So they don't work better per se. In fact they are "worse" by the most important metric (error). But they are a good trade off : LSTMs are thousands of times faster, with only slightly worse results for sequence labeling or production (ie. text and audio comprehension). CNNs are millions of times faster at image comprehension than a fully connected network, and are only slightly worse at it.

nl 9y ago

The distill.pub explanation is pretty good https://distill.pub/2016/augmented-rnns/ https://distill.pub/2016/augmented-rnns/

xgb84j 9y ago

When you want to process a sequence of vectors while incorporating information from a second sequence, you use attention. Attention means you create a weighted sum of all vectors in the second sequence for each vector in the first sequence. You basically add a customized summary vector of the second sequence to each vector of the first.