4 ms·
Note on the sinusoidal encoding, the reason it’s used is generally speaking twofold: 1 - To encode position somehow (which author details) 2 - Because sine is
by cuuupid 4y ago
Note on the sinusoidal encoding, the reason it’s used is generally speaking twofold:
1 - To encode position somehow (which author details)
2 - Because sine is easy “noise” for the network to learn.
There’s also a bunch of cool tricks here even down to the PyTorch implementation to optimize this encoding by exploiting the nature of sine/cosine which is an added reason for its popularity in Transformer architectures. If you like math I recommend diving into it as it’s quick but fun!
(Side note it’s also falling out of fashion for other encoding methods. e.g. rotary positional encoding is vastly popular in the reformer branch of transformers)
- WanderPanda 4y agoIs treating the position embeddings as trainable weights already out of fashion again?
- alexmolas 4y agoaccording to the Attention is all you need paper there wasn't huge differences between using sin/cos and trainable weights. But the paper is a little bit old, so I don't know what is the current sota regarding positional embeddings
- deleted 4y ago[deleted]
- yunwal 4y agoI’m new to this stuff, but as I understand it, the “Attention is all you need” paper stated that training the positional encoding weights didn’t improve results for language models specifically, but other papers found that vision transformers performed better with trainable weights.
- khazhoux 4y agoFor a newbie in all this, what's the reason that position isn't just encoded as an integer?
- pnt12 4y agoNot an expert, but took my masters in data science some years ago: My interpretation is the sinusoidal converts the values from 0..N to 0..1. This may result in more predictable changes than using large integer numbers, such that the same word in different positions doesn't lose all its meaning. Using integers and having a layer to compute this operation could also work, so maybe this is an optimization, eg it reduces training time or yields better results.