3 ms·
Is treating the position embeddings as trainable weights already out of fashion again?
by WanderPanda 4y ago
Is treating the position embeddings as trainable weights already out of fashion again?
- alexmolas 4y agoaccording to the Attention is all you need paper there wasn't huge differences between using sin/cos and trainable weights. But the paper is a little bit old, so I don't know what is the current sota regarding positional embeddings
- deleted 4y ago[deleted]
- yunwal 4y agoI’m new to this stuff, but as I understand it, the “Attention is all you need” paper stated that training the positional encoding weights didn’t improve results for language models specifically, but other papers found that vision transformers performed better with trainable weights.