3 ms·
I'm running 4-bit quantized llamas on torch/cuda with https://github.com/qwopqwop200/GPTQ-for-LLaMa https://github.com/qwopqwop200/GPTQ-for-LLaMa, and I'm seein
by lxe 4y ago
I'm running 4-bit quantized llamas on torch/cuda with https://github.com/qwopqwop200/GPTQ-for-LLaMa https://github.com/qwopqwop200/GPTQ-for-LLaMa, and I'm seeing significant tokens/second perf degradation compared to 8-bit bitsandbytes mode. I'm very new to this, and understand very little detail, but I thought it would be faster?
- sillysaurusx 4y agoEh, I’d expect it to be slower if anything. Think about it like this. If you write an image renderer, bitmap would be the fastest, because it’s already decompressed. 4-bit quantization is a compression algorithm. It depends on the details of memory bandwidth vs compute though.
- washadjeffmad 4y agoIn case anyone catches this late (and anything older than a few hours in AI/ML is!), some of the original llama HF weights were not built correctly and gave poor output. Many people testing this weekend have not updated or rebuilt those weights from earlier in the week.