4 ms·
It's not really attention, it's 'just' a exponential moving average over time where different channels have different decay rates. This is a simplification, in
by sebzim4500 4y ago
It's not really attention, it's 'just' a exponential moving average over time where different channels have different decay rates. This is a simplification, in the actual architecture there are also convolution layers.