3 ms·
If this quantization method works with smaller models, it would enable running up to 33B models with only 12GB VRAM. Especially important for democratizing acc
by tarruda 3y ago
If this quantization method works with smaller models, it would enable running up to 33B models with only 12GB VRAM.
Especially important for democratizing access to Mistral MoE new model.
- MrNeon 3y agoIIRC quantizing small models causes a higher relative drop in the metrics.