4 ms·

Pretty sure an fp32 7b model uses just shy of 27gb of vram. The performance losses at int8 are minuscule, even down to int5 is still very usable. Why would you

Pretty sure an fp32 7b model uses just shy of 27gb of vram. The performance losses at int8 are minuscule, even down to int5 is still very usable. Why would you try and run it fp32 or fp16?

Sohcahtoa82 3y ago
My naive intuition is that you'd want to train on the highest precision you have the horsepower for, and then after training, you might be able to get away with quantizing to a lower precision for inference. I would also expect that fp32 might be overkill, but that anything less than fp16 would start to show a noticeable loss in quality, especially in training. I can't imagine int8 being good at all. And I would think training with int5 would produce garbage. I didn't even know int5 was an option. Do you have any published benchmarks comparing the quality of results between the different precisions, going all the way down to int5?
- eightysixfour 3y ago
  https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684 You are broadly better off running a larger, more quantized model, than a smaller model at a higher precision. A 65b parameter model with 2bit quantization still outperforms an fp16 30b parameter model in perplexity and qualitative testing. No one I know does fp32, there's no benefit. People train LoRAs at low precision regularly. The folks on /r/localllama are really pushing the boundaries of what can be done on consumer hardware, and the boundaries are further out than you'd think. Also, if you want to really blow-up your intuition, check out model merges...
  - Sohcahtoa82 3y ago
    That's really interesting to know. Kind of mind-blowing, tbh. Does all that apply to training as well, or just inference?
    - eightysixfour 3y ago
      Yes, and there are techniques that get it closer than naive approaches, like quantization aware training.

4 ms·

Pretty sure an fp32 7b model uses just shy of 27gb of vram. The performance losses at int8 are minuscule, even down to int5 is still very usable. Why would you

by eightysixfour 3y ago

Pretty sure an fp32 7b model uses just shy of 27gb of vram. The performance losses at int8 are minuscule, even down to int5 is still very usable. Why would you try and run it fp32 or fp16?

Sohcahtoa82 3y ago
My naive intuition is that you'd want to train on the highest precision you have the horsepower for, and then after training, you might be able to get away with quantizing to a lower precision for inference. I would also expect that fp32 might be overkill, but that anything less than fp16 would start to show a noticeable loss in quality, especially in training. I can't imagine int8 being good at all. And I would think training with int5 would produce garbage. I didn't even know int5 was an option. Do you have any published benchmarks comparing the quality of results between the different precisions, going all the way down to int5?
- eightysixfour 3y ago
  https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684 You are broadly better off running a larger, more quantized model, than a smaller model at a higher precision. A 65b parameter model with 2bit quantization still outperforms an fp16 30b parameter model in perplexity and qualitative testing. No one I know does fp32, there's no benefit. People train LoRAs at low precision regularly. The folks on /r/localllama are really pushing the boundaries of what can be done on consumer hardware, and the boundaries are further out than you'd think. Also, if you want to really blow-up your intuition, check out model merges...
  - Sohcahtoa82 3y ago
    That's really interesting to know. Kind of mind-blowing, tbh. Does all that apply to training as well, or just inference?
    - eightysixfour 3y ago
      Yes, and there are techniques that get it closer than naive approaches, like quantization aware training.