5 ms·
How does modified RoPE compare to the sliding window attention used by Mistral?
by nullc 3y ago
How does modified RoPE compare to the sliding window attention used by Mistral?
- redox99 3y agoThis is still full attention so it should be better in benchmarks, whereas the sliding window attention is more efficient so it allows higher context sizes.