5 ms·
I’d love to run this on a single 24gb 3090 - how much dram / SSD space do I need for a decent LLM, when it’s quantised to 4bits?
by dharma1 4y ago
I’d love to run this on a single 24gb 3090 - how much dram / SSD space do I need for a decent LLM, when it’s quantised to 4bits?
- Miraste 4y agoI've been trying this, and with compression on (4 bits) you can fit the entire 30B model on the 3090.
- dharma1 4y agoOK so don't need offloading at all for the quantised model - nice. In practice, how good is the 30B model vs 175B?
- Miraste 4y agoI don't have access to 175B for comparison. In a vacuum, 30B isn't very good. In the neighborhood of GPT-NeoX-20B, I think, but not good. It repeats itself easily and has a tenuous relationship with the topic. It's still much better than anything I could run locally before now.