4 ms·
> predict a weight of exactly zero for some of the values Wouldn’t this be pretty unlikely, though?
by dartos 2y ago
> predict a weight of exactly zero for some of the values
Wouldn’t this be pretty unlikely, though?
- schopra909 2y agoQuite the opposite — if you have a long sequence only a smattering of the words will influence the meaning of the current word. Everything else is “noise”. Attention is really good at finding this smattering of words (ie assign most weight there). But it struggles to put exactly 0 on the other words.
- absoflutely 2y agowhy say lot word when few word do
- dartos 2y agoFew word no do tho
- 1024core 2y agoPhew!
- kridsdale3 2y agoU+1FAE5
- dartos 2y agoI mean wouldn’t it be unlikely that SoftmaxA[n] - SoftmaxB[n] is exactly 0? Even if 2 attention layers learn two different things, I would imagine the corresponding weights in each layer wouldn’t exactly cancel each other out.