2 ms·
It's actually an effect that happens in the (re-)alignment process due to harmonic properties of the positional encoding in the attention matrix. (I recommend
by cookiengineer 2mo ago
It's actually an effect that happens in the (re-)alignment process due to harmonic properties of the positional encoding in the attention matrix.
(I recommend reading and implementing the Attention is all you need paper. By hand. Otherwise you won't learn anything from it.)