4 ms·

Why do you think Focus Mechanism is more appropriate than Attention?

by elcomet 7y ago

nerdponx 7y ago
Less anthropomorphism in machine learning is good IMO.
- bitL 7y ago
  We are literally talking about intelligence, for which the best model in nature are humans. It's difficult not to be anthropomorphic in general.
  - nerdponx 7y ago
    The article is about deep learning, not AI.
    - bitL 7y ago
      There is very limited vocabulary for concepts you see in Deep Learning; even anthropomorphic ones are usually badly used, but you aren't going to have many fans if you start talking about key-value weighting of intermediate layers instead of "attention".
      - JoeSamoa 7y ago
        Thank you for defending this point so rigorously. I agree.
  - darkmighty 7y ago
    Exactly. Shannon could have called his metric a 'Logarithmic measure on probability spaces', but he chose to call it 'Information' -- even though it's not exactly what we mean when we talk about information informally; the inspiration and analogy was very important to the work (personally I'd have called S 'Average Information', but that's with hindsight).\* As long as you clearly define and state things (definitions) and can separate the context-specific [technical] meaning from general usage, I find it's a very good strategy not only for popularization purposes but to inspire and draw some valuable intuition from our daily lives into technical matter. Naming things in the sciences is an art :) *: It should be obvious I don't love the usual term 'Kullball-Leibler Divergence' in place of 'Relative Information', although in this case the names are obscure and difficult to pronounce enough to give it an air of rigour and nobility.
joe_the_user 7y ago
Well, attention implies something that can shift rapidly, that might have some meaning and that can lead to action. Focus is broadly like attention but without those implications. I don't know enough to say which one is more appropriate here but that's how the terms seem to "color" the concepts.

Why do you think Focus Mechanism is more appropriate than Attention?

nerdponx 7y ago

Less anthropomorphism in machine learning is good IMO.

bitL 7y ago
We are literally talking about intelligence, for which the best model in nature are humans. It's difficult not to be anthropomorphic in general.
- nerdponx 7y ago
  The article is about deep learning, not AI.
  - bitL 7y ago
    There is very limited vocabulary for concepts you see in Deep Learning; even anthropomorphic ones are usually badly used, but you aren't going to have many fans if you start talking about key-value weighting of intermediate layers instead of "attention".
    - JoeSamoa 7y ago
      Thank you for defending this point so rigorously. I agree.
- darkmighty 7y ago
  Exactly. Shannon could have called his metric a 'Logarithmic measure on probability spaces', but he chose to call it 'Information' -- even though it's not exactly what we mean when we talk about information informally; the inspiration and analogy was very important to the work (personally I'd have called S 'Average Information', but that's with hindsight).\* As long as you clearly define and state things (definitions) and can separate the context-specific [technical] meaning from general usage, I find it's a very good strategy not only for popularization purposes but to inspire and draw some valuable intuition from our daily lives into technical matter. Naming things in the sciences is an art :) *: It should be obvious I don't love the usual term 'Kullball-Leibler Divergence' in place of 'Relative Information', although in this case the names are obscure and difficult to pronounce enough to give it an air of rigour and nobility.

joe_the_user 7y ago

Well, attention implies something that can shift rapidly, that might have some meaning and that can lead to action. Focus is broadly like attention but without those implications. I don't know enough to say which one is more appropriate here but that's how the terms seem to "color" the concepts.