3 ms·
yep, it is agonistic to 4-bit. You can deploy a 4-bit model and still use vllm + pagedattention to double or even triple your serving throughput.
by zhisbug 3y ago
yep, it is agonistic to 4-bit. You can deploy a 4-bit model and still use vllm + pagedattention to double or even triple your serving throughput.
- ynniv 3y agoIf this were submitted as a new comment it would be at the top of the page.
- ipsum2 3y agoprobably mean agnostic, agonistic implies the opposite.
- zhisbug 3y agooops typo
- baobabKoodaa 3y agoYou mean like, theoretically, in the future? Or you mean today?