3 ms·
Do we have some studies on the impact of extreme quantization on these models? In my understanding, the efforts to make them run in commodity hardware involves
by haolez 4y ago
Do we have some studies on the impact of extreme quantization on these models? In my understanding, the efforts to make them run in commodity hardware involves the quantization of the weights to 8 bits or even 4 bits. Is the model still reliable after this?
- O__________O 4y agoYes, intro on the topic if anyone is interested, include topic of benchmarking degradation: - https://huggingface.co/blog/hf-bitsandbytes-integration https://huggingface.co/blog/hf-bitsandbytes-integration
- haolez 4y agoFantastic article. Just a heads up for anyone reading this reply.