3 ms·
Is there a handy table for this? My napkin math has either underestimated throughput by 2 orders of magnitude or the above estimate is high.
by lumost 2y ago
Is there a handy table for this? My napkin math has either underestimated throughput by 2 orders of magnitude or the above estimate is high.
- codetrotter 2y ago(1,000,000,000,000/63,000)/(60*60) (1T tokens / 63k tokens per second) / (60 seconds per minute * 60 minutes per hour) Is approx 4400 hours So I guess that’s how the calculation went. Or did you mean a source for the number of tokens per second?
- lumost 2y agoTokens per second ;) I can do the arithmetic on my own.
- YetAnotherNick 2y agoYou require 6 * parameter * token flops[1] to train LLM. Which means (flop/s of H100 * MFU) / (6 * parameter) token per second. Assuming MFU of 40%, it is (1000 * 10^12 * 0.4) / (6 * 10^9) token/sec = 67,000 token/sec. This repo[2] by Meta achieves 48% MFU, or 80k token/second. [1]: https://arxiv.org/pdf/2001.08361 https://arxiv.org/pdf/2001.08361 [2]: https://github.com/facebookresearch/lingua https://github.com/facebookresearch/lingua