You’d have to quantize the parameters to about 2.7 bits per parameter (24 GB / 70G * 8bits/B) - the model was likely trained at fp16 or fp32 so that would be pr

You’d have to quantize the parameters to about 2.7 bits per parameter (24 GB / 70G * 8bits/B) - the model was likely trained at fp16 or fp32 so that would be pretty challenging. Not impossible but probably not readily available at the moment w most current quantization libraries. Quality would likely be degraded. But 2 4090s might be doable at ~4bits