3 ms·
Unless I've misunderstood something, the sliding context window should decrease memory usage at inference compared to normal flash attention.
by sebzim4500 3y ago
Unless I've misunderstood something, the sliding context window should decrease memory usage at inference compared to normal flash attention.