6 ms·
Attention Mechanism in Deep Learning
- Yuval_Halevi 7y ago>When we think about the English word “Attention”, we know that it means directing your focus at something and taking greater notice. The Attention mechanism in Deep Learning is based off this concept of directing your focus, and it pays greater attention to certain factors when processing the data. I actually think they should rename it to 'Focus Mechanism'
- elcomet 7y agoWhy do you think Focus Mechanism is more appropriate than Attention?
- nerdponx 7y agoLess anthropomorphism in machine learning is good IMO.
- bitL 7y agoWe are literally talking about intelligence, for which the best model in nature are humans. It's difficult not to be anthropomorphic in general.
- nerdponx 7y agoThe article is about deep learning, not AI.
- bitL 7y agoThere is very limited vocabulary for concepts you see in Deep Learning; even anthropomorphic ones are usually badly used, but you aren't going to have many fans if you start talking about key-value weighting of intermediate layers instead of "attention".
- JoeSamoa 7y agoThank you for defending this point so rigorously. I agree.
- darkmighty 7y agoExactly. Shannon could have called his metric a 'Logarithmic measure on probability spaces', but he chose to call it 'Information' -- even though it's not exactly what we mean when we talk about information informally; the inspiration and analogy was very important to the work (personally I'd have called S 'Average Information', but that's with hindsight).\* As long as you clearly define and state things (definitions) and can separate the context-specific [technical] meaning from general usage, I find it's a very good strategy not only for popularization purposes but to inspire and draw some valuable intuition from our daily lives into technical matter. Naming things in the sciences is an art :) *: It should be obvious I don't love the usual term 'Kullball-Leibler Divergence' in place of 'Relative Information', although in this case the names are obscure and difficult to pronounce enough to give it an air of rigour and nobility.
- joe_the_user 7y agoWell, attention implies something that can shift rapidly, that might have some meaning and that can lead to action. Focus is broadly like attention but without those implications. I don't know enough to say which one is more appropriate here but that's how the terms seem to "color" the concepts.
- codesushi42 7y agoI disagree. "Attention" is all you need.
- physicsyogi 7y agoFocus is usually directed at a single thing though, it's a narrowing of attention. Attention mechanisms can pay attention to more than one thing at a time.
- kuu 7y agoIn case you find Attention (and specially transformers) interesting I have some saved links introducing the topic: http://www.peterbloem.nl/blog/transformers http://www.peterbloem.nl/blog/transformers https://nostalgebraist.tumblr.com/post/185326092369/the-transformer-explained https://nostalgebraist.tumblr.com/post/185326092369/the-tran... https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf https://papers.nips.cc/paper/7181-attention-is-all-you-need.... http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/ https://arxiv.org/pdf/1807.03819.pdf https://arxiv.org/pdf/1807.03819.pdf
- BiasRegularizer 7y agoWhile Transformer's Self Attention(SA) is great, there are many applications where SA doesn't apply. For a more comprehensive overview of attention mechanisms, I often find myself coming back to Lilian Weng's post: https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html https://lilianweng.github.io/lil-log/2018/06/24/attention-at...
- phkahler 7y agoWould it be more accurate to use the word "importance" than "attention"? I feel like the later is encroaching on "intention" and conciousness more than these techniques warrant.
- BiasRegularizer 7y ago"Importance" is a fairly overused word in DL. e.g. importance sampling and importance weighted gradients. "Attention" works by creating inductive bias for the upstream network, which is analogous to human attention, and the word itself is much more intuitive. Keep in mind machine learning is largely a descriptive science(modeling the behavior), whereas neuroscience is more prescriptive. So from the behavioral perspective, attention is better suited than importance.
- msamwald 7y agoOn the other hand the word "attention" in deep learning is often used non-intuitively, e.g. "one token attends to another token" in self attention is hardly analogous to human cognitive processes.
- abhgh 7y agoIn the context of attention, there is a very interesting recent paper that warns against conflating attention and token importance - "Is Attention Interpretable?" [1]. This is an accepted paper in ACL-2019: [1] https://www.aclweb.org/anthology/P19-1282 https://www.aclweb.org/anthology/P19-1282
- stochastic_monk 7y agoSee also Attention is Not Explanation [0]. [0] https://www.aclweb.org/anthology/N19-1357 https://www.aclweb.org/anthology/N19-1357
- abhgh 7y agoThanks!
- physicsyogi 7y agoThere's a rebuttal to this as well: Attention is not not Explanation. https://arxiv.org/abs/1908.04626 https://arxiv.org/abs/1908.04626
- hhtoyou 7y agoSO, TRUE THANKS TO LUKAS KAISER !
- eanzenberg 7y agoIs there any work on attention for cnns or other computer vision algo's?
- lucidrains 7y agohttps://arxiv.org/abs/1904.09925 https://arxiv.org/abs/1904.09925
- ilaksh 7y agoDid a FAANG company patent it already? If so, can we safely assume that since such a patent is ridiculous, it should be ignored in relation to any commercial service that might use these techniques?
- abakus 7y agoFor a very easy to understand explanation of the transformer and attention mechansim, see: https://blue-season.github.io/transformer-in-5-minutes/ https://blue-season.github.io/transformer-in-5-minutes/