4 ms·
Are frontier models actually astronomically expensive to train? GLM 5.2 was trained on ~30T tokens, so ~10^25 FLOPs. Say you get B300's for $5/h (pretty high),
by blintz 2mo ago
Are frontier models actually astronomically expensive to train? GLM 5.2 was trained on ~30T tokens, so ~10^25 FLOPs. Say you get B300's for $5/h (pretty high), and you get 50% MFU; that's ~$15M. Of course there's also a bunch of risk that the training itself goes badly, post-training, etc. But still, compared to the inference spend after, it's not that crazy.
- avaer 2mo agoNobody spends 15M on inference only to check at the end that their money was wasted.
- user43928 2mo agoAre there resources I could read to learn more about how the labs work? I am curious about what changed since GPT 4.5 and other unsuccessful attempts to train large models, compared to now.