I did a very similar analysis with Llama 65B being trained on 5.6T tokens assuming token length of 4 characters and comparing with a quantized model size of ~38

I did a very similar analysis with Llama 65B being trained on 5.6T tokens assuming token length of 4 characters and comparing with a quantized model size of ~38GB. The 3% number was a conservative rounding of the same calculation, but retaining fp16 rather than quantizing to 4 bits. Here's my original back of the napkin analysis: https://news.ycombinator.com/item?id=36681440 https://news.ycombinator.com/item?id=36681440