Its more nuanced than this. Sometimes pure 8 bit inference (without any quantization tricks) reduces llm performance enough to impact certain tasks like retriev

Its more nuanced than this. Sometimes pure 8 bit inference (without any quantization tricks) reduces llm performance enough to impact certain tasks like retrieval. Its also important to distinguish 8 bit compute from the actual quantization scheme. vLLM in fact supports 4 bit AWQ, but the computation is not done in 4 bits on the GPU, and it wouldn't even be close to pure 4 bit inference even if the hardware and vLLM supported that.