7 ms·
This is worrisome for open source - this isn't some far off future limit. Llama3 training at 400 TFLOPS per GPU [1] and 6.4M GPU hours [2] puts Llama3-70B at 9.
by ninjagoo 2y ago
This is worrisome for open source - this isn't some far off future limit. Llama3 training at 400 TFLOPS per GPU [1] and 6.4M GPU hours [2] puts Llama3-70B at 9.2*10^24 (so 10^25) floating point ops.
Llama3-400B might be well past this threshold.
[1] https://ai.meta.com/blog/meta-llama-3/ https://ai.meta.com/blog/meta-llama-3/
[2] https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct