In a recent-ish paper, there was an analysis of positional encodings using Neural Tangent Kernels which showed positional encodings are very similar to Fourier

In a recent-ish paper, there was an analysis of positional encodings using Neural Tangent Kernels which showed positional encodings are very similar to Fourier transformations of the inputs: Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, https://arxiv.org/abs/2006.10739 https://arxiv.org/abs/2006.10739