5 ms·
Quite the opposite — if you have a long sequence only a smattering of the words will influence the meaning of the current word. Everything else is “noise”. Att
by schopra909 2y ago
Quite the opposite — if you have a long sequence only a smattering of the words will influence the meaning of the current word. Everything else is “noise”.
Attention is really good at finding this smattering of words (ie assign most weight there). But it struggles to put exactly 0 on the other words.
- absoflutely 2y agowhy say lot word when few word do
- dartos 2y agoFew word no do tho
- 1024core 2y agoPhew!
- kridsdale3 2y agoU+1FAE5
- dartos 2y agoI mean wouldn’t it be unlikely that SoftmaxA[n] - SoftmaxB[n] is exactly 0? Even if 2 attention layers learn two different things, I would imagine the corresponding weights in each layer wouldn’t exactly cancel each other out.