3 ms·
In my experience there's really no reason to run any model above Q6_K, the performance is identical and you shave off almost 2 GB of VRAM of a 7B model compared
by moffkalast 3y ago
In my experience there's really no reason to run any model above Q6_K, the performance is identical and you shave off almost 2 GB of VRAM of a 7B model compared to Q8. To those of us with single digit amounts, that's highly significant. But most people seem to go for 4 bits anyway and it's the AWQ standard too. If you think it'll make the model look bad, then don't worry, it's only the relative performance that matters.
I would think that having an OpenAI standard compatible API would be a higher priority over a python wrapper, since then it can act as a drop in replacement for most any backend.
- austinvhuang 3y agoA nice side effect of implementing cpu simd is you just need enough regular RAM, which tends to be far less scarce than VRAM. Nonetheless, I get your point that more aggressive quantization is valuable + will share with the modeling team.
- moffkalast 3y agoTrue, it's the only way I can for example run Mixtral on a 8GB GPU, but main memory will always have more latency so some tradeoff tends to be worth it. And parts like the prompt batch buffer and most of the context generally have to be in VRAM if you want to use cuBLAS, with OpenBLAS it's maybe less of a problem, but it is slower.