https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684 You are broadly better off running a larger, more quantized m

https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684 You are broadly better off running a larger, more quantized model, than a smaller model at a higher precision. A 65b parameter model with 2bit quantization still outperforms an fp16 30b parameter model in perplexity and qualitative testing. No one I know does fp32, there's no benefit. People train LoRAs at low precision regularly. The folks on /r/localllama are really pushing the boundaries of what can be done on consumer hardware, and the boundaries are further out than you'd think. Also, if you want to really blow-up your intuition, check out model merges...