4 ms·
This seems suboptimal considering the simple output from the original Viswani PE, which is solidly based on a well grounded foundation of eigenvectors
by chaxor 3y ago
This seems suboptimal considering the simple output from the original Viswani PE, which is solidly based on a well grounded foundation of eigenvectors for the discrete Fourier transform relation to the circulant matrix of a linear chain graph which is natural language of the sentence
- KRAKRISMOTT 3y agoI don't recall there being a DFT in the original attention is all you need paper.
- just_a_quack 3y agoThere's not. The positional encodings are generated using sines and cosines such that any offset in position can be described as a linear function on the original position. Using the DFT here would not make sense as the positional encodings are fixed anyway and during inference this method generalizes nicely because of the geometric progression created by the arguments of the positional encoding functions.
- chaxor 3y agoThere isn't a DFT directly, it's a more obvious statement here. The circulant matrix (linear graph of words) always has the same eigenvectors and is diagonalized via DFT. The PE in original Viswani is based on this, they just didn't put in all the details. So effectively the model gets hints from the PE that it's a linear graph because these are the eigenvectors.
- deleted 3y ago[deleted]
- taliesinb 3y agoplease clarify your suffix "which is natural language of the sentence"? are you referring to the positional encoding?
- ofirpress 3y agoThe ALiBi paper shows that our method beats the sinusoidal PE you refer to across many benchmarks. https://arxiv.org/abs/2108.12409 https://arxiv.org/abs/2108.12409