4 ms·
Not sure if @dang is the right way to say the title is incorrect here, but shouldn't it match the paper? 1. The correct title is Layer-Condensed KV Cache for E
by WhitneyLand 2y ago
Not sure if @dang is the right way to say the title is incorrect here, but shouldn't it match the paper?
1. The correct title is Layer-Condensed KV Cache for Efficient Inference of Large Language Models.
2. The paper does make a 26x claim later in the introduction, but it’s an outlier.
26x is for only one benchmark and that benchmark is CPU based, not GPU based like 99% of transformer loads actually run on.
If you look at GPU only workloads, the improvements range from 1.4x to 4.7x.
- kiraaa 2y agoon gpu that is still huge.
- 0cf8612b2e1e 2y agoEven if it is “only” the 40% lower end, that is a gargantuan savings. So many groups are compute constrained, every bit helps.
- josephg 2y agoSure; but 40% improvement is much less than a 26x improvement. If 40% is the realistic figure, cite that. Changing the title to include an outlier of 26x is click baity.
- VeejayRampay 2y agoit sure is huge, but it's still far from 26x
- huhuhu111 2y ago[dead]
- imtringued 2y agoIt's not an outlier. You will see more than 26x improvement if you try this on an even deeper LLM with more layers. The deepest model they have applied it on has 30 billion parameters. Edit: I apologize. The table was cut off on mobile and I didn't see that they sneaked in GPU+CPU offloading for the 25x result.
- wglb 2y ago> Not sure if @dang is the right way to say the title is incorrect here, but shouldn't it match the paper? Best way to address this is to email hn@ycombinator.com. It is not possible to read all posts to HN. (Also, the convention is to use 'dang rather than @dang, borrowing from a lisp convention.)