At that level of quantization / distillation, smaller models like phi-2 (q&a) and wavecoder-6.7b (code-gen) might be preferable over QLoRAd ones: https://huggin

At that level of quantization / distillation, smaller models like phi-2 (q&a) and wavecoder-6.7b (code-gen) might be preferable over QLoRAd ones: https://huggingface.co/microsoft/phi-2 https://huggingface.co/microsoft/phi-2 > 2bit is pretty damn terrible Wait till you go hybrid [0] or even 1bit [1] [0] https://github.com/efeslab/Atom https://github.com/efeslab/Atom [1] https://github.com/IST-DASLab/qmoe https://github.com/IST-DASLab/qmoe