7 ms·
- 180B parameters - Trained on 3.5 trillion tokens - 7 million GPU hours - Quality on par with PaLM 2, outperforming Llama 2 and GPT -3.5 across benchmarks
by osanseviero 3y ago
- 180B parameters
- Trained on 3.5 trillion tokens
- 7 million GPU hours
- Quality on par with PaLM 2, outperforming Llama 2 and GPT
-3.5 across benchmarks
- 4-bit and 8-bit show little degradation
- deleted 3y ago[deleted]
- souvic 3y ago"4-bit and 8-bit show little degradation" - This is the most interesting part!
- logicchains 3y agoIt makes sense. Falcon 40B was trained on around 1 trillion tokens. If it was trained to saturation, then Falcon 180B would need to be trained on 180/40=4.5 trillion tokens to saturate it, but it was only trained on 3.5 trillion. And if 1 trillion wasn't enough to saturate the 40B model, then 3.5 trillion definitely won't be enough to saturate the 180B model. If not trained to saturation, then a model can be quantised without losing too much, as it still has "empty space" that can be removed by compression, so to speak.