4 ms·
I think it's about the quantization process and loading the model. I needed just above 64GB of RAM + Swap to quantize the 35B model to int4. Not to mention much
by raihansaputra 4y ago
I think it's about the quantization process and loading the model. I needed just above 64GB of RAM + Swap to quantize the 35B model to int4. Not to mention much slower inference time.
- montebicyclelo 4y agoHmm, wonder if the whole model is being loaded into memory for the quantisation, and whether that's necessary. (Also, shame the models can't be distributed in 4bit, (due to the license).)
- raihansaputra 4y agoYeah it's basically wholly loaded to memory for quantisation. Some optimization should be possible, and yes the quantized models can't be distributed sadly.