4 ms·
Why do you think it's only that? I've seen a lot of dismissive comments on AI articles that were shown to misunderstand or underappreciate the article.
by ogogmad 3y ago
Why do you think it's only that? I've seen a lot of dismissive comments on AI articles that were shown to misunderstand or underappreciate the article.
- avereveard 3y agoWhy do you claim I "think it's only that"? The other part of the paper handling the sliding KV cache compares favourably with prefix caching, sure, but then we moved away prefix caching for serving since a while now, with paged attention (which should really have been called paged KV cache but oh well) offering a lot of interesting improvement in that area including supporting extremely well parallel decoding. And I do not care enough to compare the streaming cache with the paged attention cache directly, first because it's work they should have done and not I, second because dropping token silently is something that confuses and frustrated users significantly enough that it puts me down from wanting to investigate further.