4 ms·
This is still full attention so it should be better in benchmarks, whereas the sliding window attention is more efficient so it allows higher context sizes.
by redox99 3y ago
This is still full attention so it should be better in benchmarks, whereas the sliding window attention is more efficient so it allows higher context sizes.