3 ms·
Another factor is that larger models degrade less when quantized. You have to wonder if running a huge model, say, 300B parameters at 2-bit quantization might
by jiggawatts 3y ago
Another factor is that larger models degrade less when quantized.
You have to wonder if running a huge model, say, 300B parameters at 2-bit quantization might be "optimal" in that it would fit into a single A100 or H100 GPU and likely outperform an 80B parameter 8-bit model...
- Ambix 3y agoNot sure here. The LLaMA models - yes, all weights fit in the small range between -2.0 .. 2.0 And some other models have more crazy numbers with even more crazier outliers within them, like you might have a weight of 12.00 between long array of typical small numbers around 0.00 I've read story about attempt to quantize RWKV model into the 4/5 bits which failed short due to the presence of outlier weights. The author told somewhere that bigger models had worse perplexity because of this.