2 ms·

I think most transformer models use sin and cos for positional encoding. They were recommended in the original transformer paper, Attention is All You Need. Bu

I think most transformer models use sin and cos for positional encoding. They were recommended in the original transformer paper, Attention is All You Need. But in that case it's not quite Fourier analysis because those functions aren't applied to the input; they are just constants that are added to the input because transformers are permutation invariant (without some positional encoding). Was there a model that applied sin and cos to the input?

gessha 4y ago
In a recent-ish paper, there was an analysis of positional encodings using Neural Tangent Kernels which showed positional encodings are very similar to Fourier transformations of the inputs: Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, https://arxiv.org/abs/2006.10739 https://arxiv.org/abs/2006.10739