4 ms·
OK so don't need offloading at all for the quantised model - nice. In practice, how good is the 30B model vs 175B?
by dharma1 4y ago
OK so don't need offloading at all for the quantised model - nice.
In practice, how good is the 30B model vs 175B?
- Miraste 4y agoI don't have access to 175B for comparison. In a vacuum, 30B isn't very good. In the neighborhood of GPT-NeoX-20B, I think, but not good. It repeats itself easily and has a tenuous relationship with the topic. It's still much better than anything I could run locally before now.