3 ms·
I'm spoiled by 4 bit and unfortunately it doesn't appear to be supposed here so this isn't of much use to me, but it's awesome to see people working on the infe
by bioemerl 3y ago
I'm spoiled by 4 bit and unfortunately it doesn't appear to be supposed here so this isn't of much use to me, but it's awesome to see people working on the inference speed side of things regardless.
- george_123 3y agothis approach to managing KV cache can work with 4bit. imagine the speedup of pagedattention with quantization..
- zhisbug 3y agoyep, it is agonistic to 4-bit. You can deploy a 4-bit model and still use vllm + pagedattention to double or even triple your serving throughput.
- ynniv 3y agoIf this were submitted as a new comment it would be at the top of the page.
- ipsum2 3y agoprobably mean agnostic, agonistic implies the opposite.
- zhisbug 3y agooops typo
- baobabKoodaa 3y agoYou mean like, theoretically, in the future? Or you mean today?