4 ms·
this approach to managing KV cache can work with 4bit. imagine the speedup of pagedattention with quantization..
by george_123 3y ago
this approach to managing KV cache can work with 4bit. imagine the speedup of pagedattention with quantization..
- zhisbug 3y agoyep, it is agonistic to 4-bit. You can deploy a 4-bit model and still use vllm + pagedattention to double or even triple your serving throughput.
- ynniv 3y agoIf this were submitted as a new comment it would be at the top of the page.
- ipsum2 3y agoprobably mean agnostic, agonistic implies the opposite.
- zhisbug 3y agooops typo
- baobabKoodaa 3y agoYou mean like, theoretically, in the future? Or you mean today?