3 ms·
Yeah "by conveniently dropping token out of attention we can have infinite tokens" not exactly a breakthrough
by avereveard 3y ago
Yeah "by conveniently dropping token out of attention we can have infinite tokens" not exactly a breakthrough
- ogogmad 3y agoWhy do you think it's only that? I've seen a lot of dismissive comments on AI articles that were shown to misunderstand or underappreciate the article.
- avereveard 3y agoWhy do you claim I "think it's only that"? The other part of the paper handling the sliding KV cache compares favourably with prefix caching, sure, but then we moved away prefix caching for serving since a while now, with paged attention (which should really have been called paged KV cache but oh well) offering a lot of interesting improvement in that area including supporting extremely well parallel decoding. And I do not care enough to compare the streaming cache with the paged attention cache directly, first because it's work they should have done and not I, second because dropping token silently is something that confuses and frustrated users significantly enough that it puts me down from wanting to investigate further.
- ttul 3y agoThe nuance is that their technique allows you to slide the context window along without introducing a jagged discontinuity in generation. At least that’s my feel for it.
- fastball 3y agoYes, but is that really why people want the context window to be "infinite"? In my experience, the desire for bigger context is the ability to say, dump an 1000 page book into an LLM and ask questions about any part of it, not just the last chapter.
- ttul 3y agoI think the title is misleading. This is an MIT PR piece.