Yeah! That's pretty much my quantization workflow for small models. I've got a dual 3090 box under my desk to quantize 7B and smaller models and rent machines o

Yeah! That's pretty much my quantization workflow for small models. I've got a dual 3090 box under my desk to quantize 7B and smaller models and rent machines on vast.ai for larger models which need > 24GB of RAM for larger models (Mixtral needs an 80GB A100, for instance).