4 ms·
With at least 2x80G A100 needed for QLoRA finetuning and the 4bit quantized model requiring 90G only for the weights at inference time, I doubt this will put a
by imjonse 3y ago
With at least 2x80G A100 needed for QLoRA finetuning and the 4bit quantized model requiring 90G only for the weights at inference time, I doubt this will put a dent in Llamas popularity outside large labs and enterprises. It may encourage more 1 bit quantization research though :)
- ilaksh 3y agoIt says 8 A100 for normal inference. How different is the 4bit performance?
- Tostino 3y agoNot that I've tested it with this model, but I have for Llama2 models...it makes a minimal difference. I haven't found anything that was noticeable with ~30-70b models, and from what I can tell from the scaling charts, it matters even less with the larger parameter count models.