4 ms·

So on a 4090 you cannot run the 70b model right?

• pizza 3y ago
  You’d have to quantize the parameters to about 2.7 bits per parameter (24 GB / 70G * 8bits/B) - the model was likely trained at fp16 or fp32 so that would be pretty challenging. Not impossible but probably not readily available at the moment w most current quantization libraries. Quality would likely be degraded. But 2 4090s might be doable at ~4bits
• nickolas_t 3y ago
  Sadly no, perhaps on a high end GPU in the year 2027(?)