6 ms·
"4-bit and 8-bit show little degradation" - This is the most interesting part!
by souvic 3y ago
"4-bit and 8-bit show little degradation" - This is the most interesting part!
- logicchains 3y agoIt makes sense. Falcon 40B was trained on around 1 trillion tokens. If it was trained to saturation, then Falcon 180B would need to be trained on 180/40=4.5 trillion tokens to saturate it, but it was only trained on 3.5 trillion. And if 1 trillion wasn't enough to saturate the 40B model, then 3.5 trillion definitely won't be enough to saturate the 180B model. If not trained to saturation, then a model can be quantised without losing too much, as it still has "empty space" that can be removed by compression, so to speak.