6 ms·
This is not very far off from Llama3 training. 400 TFLOPS per GPU [1] and 6.4M GPU hours [2] puts Llama3 70B at 9.2*10^24 (so 10^25) floating point ops. [1] ht
by ninjagoo 2y ago
This is not very far off from Llama3 training. 400 TFLOPS per GPU [1] and 6.4M GPU hours [2] puts Llama3 70B at 9.2*10^24 (so 10^25) floating point ops.
[1] https://ai.meta.com/blog/meta-llama-3/ https://ai.meta.com/blog/meta-llama-3/
[2] https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct