12 ms·
Why do you claim I "think it's only that"? The other part of the paper handling the sliding KV cache compares favourably with prefix caching, sure, but then we
by avereveard 3y ago
Why do you claim I "think it's only that"?
The other part of the paper handling the sliding KV cache compares favourably with prefix caching, sure, but then we moved away prefix caching for serving since a while now, with paged attention (which should really have been called paged KV cache but oh well) offering a lot of interesting improvement in that area including supporting extremely well parallel decoding.
And I do not care enough to compare the streaming cache with the paged attention cache directly, first because it's work they should have done and not I, second because dropping token silently is something that confuses and frustrated users significantly enough that it puts me down from wanting to investigate further.