3 ms·

> It's like the AI was like "uh-huh...yeah...I know some of these words!" and spat something incoherent that seemed related. An accurate depiction of what's go

> It's like the AI was like "uh-huh...yeah...I know some of these words!" and spat something incoherent that seemed related. An accurate depiction of what's going on! I had terrible results with GPT2 but the newer models are much better. I recommend fine tuning mistral (I had better results than any llama based model)

Sohcahtoa82 3y ago
> I recommend fine tuning mistral What kind of hardware does that take? I've got an RTX 3080, i9-9900K, and 32 GB of RAM right now. I eventually plan on upgrading to an RTX 5090 when those come out, as well as i9-15900K (Or whatever they're gonna call it) and 128 GB of RAM once the next Intel CPU comes out. My understanding is that even with an RTX 5090 (Assuming 32 GB of VRAM, nothing has been announced yet), if I use fp32, I still won't have enough GPU memory to do inference on a 7B model, let alone training. Though if I quantize down to fp16, I could do inference, but still not training.
- eightysixfour 3y ago
  Pretty sure an fp32 7b model uses just shy of 27gb of vram. The performance losses at int8 are minuscule, even down to int5 is still very usable. Why would you try and run it fp32 or fp16?
  - Sohcahtoa82 3y ago
    My naive intuition is that you'd want to train on the highest precision you have the horsepower for, and then after training, you might be able to get away with quantizing to a lower precision for inference. I would also expect that fp32 might be overkill, but that anything less than fp16 would start to show a noticeable loss in quality, especially in training. I can't imagine int8 being good at all. And I would think training with int5 would produce garbage. I didn't even know int5 was an option. Do you have any published benchmarks comparing the quality of results between the different precisions, going all the way down to int5?
    - eightysixfour 3y ago
      https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684 You are broadly better off running a larger, more quantized model, than a smaller model at a higher precision. A 65b parameter model with 2bit quantization still outperforms an fp16 30b parameter model in perplexity and qualitative testing. No one I know does fp32, there's no benefit. People train LoRAs at low precision regularly. The folks on /r/localllama are really pushing the boundaries of what can be done on consumer hardware, and the boundaries are further out than you'd think. Also, if you want to really blow-up your intuition, check out model merges...
      - Sohcahtoa82 3y ago
        That's really interesting to know. Kind of mind-blowing, tbh. Does all that apply to training as well, or just inference?
        eightysixfour 3y ago
        Yes, and there are techniques that get it closer than naive approaches, like quantization aware training.

Sohcahtoa82 3y ago

> I recommend fine tuning mistral What kind of hardware does that take? I've got an RTX 3080, i9-9900K, and 32 GB of RAM right now. I eventually plan on upgrading to an RTX 5090 when those come out, as well as i9-15900K (Or whatever they're gonna call it) and 128 GB of RAM once the next Intel CPU comes out. My understanding is that even with an RTX 5090 (Assuming 32 GB of VRAM, nothing has been announced yet), if I use fp32, I still won't have enough GPU memory to do inference on a 7B model, let alone training. Though if I quantize down to fp16, I could do inference, but still not training.

eightysixfour 3y ago
Pretty sure an fp32 7b model uses just shy of 27gb of vram. The performance losses at int8 are minuscule, even down to int5 is still very usable. Why would you try and run it fp32 or fp16?
- Sohcahtoa82 3y ago
  My naive intuition is that you'd want to train on the highest precision you have the horsepower for, and then after training, you might be able to get away with quantizing to a lower precision for inference. I would also expect that fp32 might be overkill, but that anything less than fp16 would start to show a noticeable loss in quality, especially in training. I can't imagine int8 being good at all. And I would think training with int5 would produce garbage. I didn't even know int5 was an option. Do you have any published benchmarks comparing the quality of results between the different precisions, going all the way down to int5?
  - eightysixfour 3y ago
    https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684 You are broadly better off running a larger, more quantized model, than a smaller model at a higher precision. A 65b parameter model with 2bit quantization still outperforms an fp16 30b parameter model in perplexity and qualitative testing. No one I know does fp32, there's no benefit. People train LoRAs at low precision regularly. The folks on /r/localllama are really pushing the boundaries of what can be done on consumer hardware, and the boundaries are further out than you'd think. Also, if you want to really blow-up your intuition, check out model merges...
    - Sohcahtoa82 3y ago
      That's really interesting to know. Kind of mind-blowing, tbh. Does all that apply to training as well, or just inference?
      - eightysixfour 3y ago
        Yes, and there are techniques that get it closer than naive approaches, like quantization aware training.