I haven’t quantized these myself since TheBloke has been the main provider for all the quantized models. But when I did a 8 bit quantization to see how it compa

I haven’t quantized these myself since TheBloke has been the main provider for all the quantized models. But when I did a 8 bit quantization to see how it compares to the transformers library load_in_8bit 4 months ago(?), it didn’t use my GPU but loaded each shard into the RAM during the conversion. I had an old 4C/8T CPU and the conversion took like 30 mins for a 13B.