3 ms·
Would it be more accurate to use the word "importance" than "attention"? I feel like the later is encroaching on "intention" and conciousness more than these te
by phkahler 7y ago
Would it be more accurate to use the word "importance" than "attention"? I feel like the later is encroaching on "intention" and conciousness more than these techniques warrant.
- BiasRegularizer 7y ago"Importance" is a fairly overused word in DL. e.g. importance sampling and importance weighted gradients. "Attention" works by creating inductive bias for the upstream network, which is analogous to human attention, and the word itself is much more intuitive. Keep in mind machine learning is largely a descriptive science(modeling the behavior), whereas neuroscience is more prescriptive. So from the behavioral perspective, attention is better suited than importance.
- msamwald 7y agoOn the other hand the word "attention" in deep learning is often used non-intuitively, e.g. "one token attends to another token" in self attention is hardly analogous to human cognitive processes.