3 ms·
Any guess on the pre-training tokens/flops they consumed?
by Tejush 2mo ago
Any guess on the pre-training tokens/flops they consumed?
- lostmsu 2mo agoTechnical report has a graph vs Kimi K2 with 1e21 FLOPs (but they don't claim that's the entirety of pretraining)
- JacobAsmuth 2mo ago1e21 flops is hilariously wrong. for reference the llama 3 8B model (https://arxiv.org/pdf/2407.21783 https://arxiv.org/pdf/2407.21783) used 10 times that many flops. This model is 350x bigger in total params and 12x bigger in active params and was trained on 3x the data.