4 ms·
Why use RoPE over Alibi? Truly relative positional embeddings help increase context length even after training. Ref: [1]: https://ofir.io/The-Use-Case-for-Rela
by tshrjn007 3y ago
Why use RoPE over Alibi?
Truly relative positional embeddings help increase context length even after training.
Ref:
[1]: https://ofir.io/The-Use-Case-for-Relative-Position-Embeddings https://ofir.io/The-Use-Case-for-Relative-Position-Embedding...