4 ms·
Can someone ELI5 how linear attention works? Isn't that impossible in principle?
by choeger 4y ago
Can someone ELI5 how linear attention works? Isn't that impossible in principle?
- sebzim4500 4y agoIt's not really attention, it's 'just' a exponential moving average over time where different channels have different decay rates. This is a simplification, in the actual architecture there are also convolution layers.