3 ms·
> One requires reporting of any model trained using more than 10^26 integer or floating point operations total On an unrelated note, OpenAI announces GPT5 will
by Ukv 3y ago
> One requires reporting of any model trained using more than 10^26 integer or floating point operations total
On an unrelated note, OpenAI announces GPT5 will be trained with fixed-point arithmetic.
- ryanschneider 3y agoThere’s actually a lower bound for integer ops: https://twitter.com/DavidVorick/status/1719097248699879831 https://twitter.com/DavidVorick/status/1719097248699879831 > (i) any model that was trained using a quantity of computing pow greater than 10^26 integer or floating-point operations, or using primarily biological sequence data and using a quantity of computing power greater than 10^23 integer or floating-point operations;
- Ukv 3y agoFor integer and floating-point operations, but not for fixed-point operations which are neither.
- jakobson14 3y agoHow would you even measure that, with current tooling? There some option for counting float ops in pytorch or something?
- ketzo 3y agoTime to train * flops per chip * number of chips in training fleet You can estimate it quite accurately, actually
- jakobson14 3y agoThen you're including all kinds of time for other delays, like copying data over PCIe into the GPU for the next batch, or waiting for the CPU to orchestrate things. You could probably get a better estimate based on the network structure, the number of training epochs, and the size of each batch.