3 ms·
Is there any intuition why does it even work? It seems very unexpected.
by xpl 2y ago
Is there any intuition why does it even work? It seems very unexpected.
- cs702 2y agoThe intuition is that the relative frequency at which past tokens get attention from future tokens is a good proxy for their relative importance. The model the authors use, in fact, maps attention scores to features in the frequency domain.