4 ms·
KVarN: Native vLLM backend for KV-cache quantization by Huawei
- shockembopper 5mo ago[dead]
- v3ss0n 5mo agoWhy this is not a PR for vLLM ?
- esafak 5mo agoIt's the output of a research paper; the authors are not trying to build up vLLM, and they probably have no incentive to do so. You can submit a PR, though! It's easier now while the divergence is low, so don't wait. Since there are six authors, I bet you could get help with the inevitable review chores if you just take the step of creating the PR. edit: It might not be clear that it is based on vLLM 0.22, which is the current version: https://github.com/huawei-csl/KVarN/commit/d6290e99098d7426dcce01cdd8cc57a2eecf21a0 https://github.com/huawei-csl/KVarN/commit/d6290e99098d7426d.... All you have to do is create a diff off it; it's fairly straightforward.
- jmalicki 5mo agoAnd with the help of AI, pointing at AI at this paper and saying "making a vLLM PR from this paper" tends to work surprisingly well, even if you need to nudge it a little bit along the way.
- thefox96 5mo agoit should be easy to do btw
- woadwarrior01 5mo agoLast I heard, vLLM was backed by a company that has raised $150m in seed funding. I'm sure they've got the resources to port it.
- electronsoup 5mo agoWhy this is not a PR for llama.cpp
- throwa356262 5mo agoBetter performance than TQ and better quality than FP16? Am I reading this right??
- pbich 5mo ago[dead]
- thefox96 5mo agoFaster than Fp16, not better quality i guess
- qeternity 5mo agoIt's not better quality: 59.3% vs 59.4% fp16 on AIME 25
- sheepscreek 5mo ago0.1% is within margin of error. Depending on the performance boost, it might be worthwhile taking a minuscule quality hit.
- qeternity 5mo agoI think it very much is worth it! But the point was that quality didn't magically increase.
- electroglyph 5mo agoany divergence (even if the benchmark is better) from full precision is error
- 7e 5mo agoJust pretend that it is the next step update when training. You didn’t train your model to step=inf, I hope?
- 0xjeffro 5mo agoyao yao ling xian
- mikeayles 5mo ago[dead]
- sspoisk 5mo ago[flagged]
- lukasc-ch 5mo ago... and it's on llama.cpp that to this guy! https://www.reddit.com/r/LocalLLaMA/comments/1txlhxu/i_implemented_kvarn_in_my_llamacpp_fork_and_ran/ https://www.reddit.com/r/LocalLLaMA/comments/1txlhxu/i_imple...
- lukasc-ch 5mo agoThis is awesome! Let's give them some stars: - https://github.com/huawei-csl/KVarN https://github.com/huawei-csl/KVarN (original repo, vLLM implementation) - https://github.com/Anbeeld/beellama.cpp https://github.com/Anbeeld/beellama.cpp (llama.cpp implementation + awesome evals)