7 ms·
Can anyone provide any additional details on the EXL2[0]/GPTQ[1] quantisation, which seems to be the main reason for a speedup in this model? I had a quick loo
by mijoharas 3y ago
Can anyone provide any additional details on the EXL2[0]/GPTQ[1] quantisation, which seems to be the main reason for a speedup in this model?
I had a quick look at the paper which is _reasonably_ clear, but if anyone else has any other sources that are easy to understand, or a quick explanation to give more insight into it, I'd appreciate it.
[0] https://github.com/turboderp/exllamav2#exl2-quantization https://github.com/turboderp/exllamav2#exl2-quantization
[1] https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323
- qeternity 3y ago> which seems to be the main reason for a speedup in this model I wouldn't agree with this as there are substantial perf improvements on GPTQ models of ca. 20%.
- mijoharas 3y agoOk, this article seems to have a nice more approachable explanation[0] [0] https://towardsdatascience.com/4-bit-quantization-with-gptq-36b0f4f02c34 https://towardsdatascience.com/4-bit-quantization-with-gptq-...