4 ms·
What would the four bit quantized version need for ram to run inference?
by swader999 3y ago
What would the four bit quantized version need for ram to run inference?
- logicchains 3y agoShould be under 128GB.
- Taek 3y agoWebsite says 320 GB for GPTQ / int4. I'm not sure why, my guess would be multi-head attention