I can quantize models up to 70B just fine with around 40-50 GB of system RAM, using the GGMLv3 format. GGUF seems not optimised yet, since quantizing with a ne

I can quantize models up to 70B just fine with around 40-50 GB of system RAM, using the GGMLv3 format. GGUF seems not optimised yet, since quantizing with a newer version of llama.cpp supporting the format fails on the same hardware. I expect that to be fixed shortly. For inference, I understand that the hardware requirements will be identical as before.