3 ms·
It was closer to 40% efficiency AFAIK
by WithinReason 2y ago
It was closer to 40% efficiency AFAIK
- Aissen 2y agoI'm curious, do you have any pointers? This is what this article mentions: > Those improvements resulted in an overall effective training time of more than 95% https://ai.meta.com/blog/meta-llama-3/ https://ai.meta.com/blog/meta-llama-3/
- logicchains 2y agoThe known SOTA for GPU flops utilisation for training on that many GPUs is somewhere between 50-60%, e.g. https://github.com/NVIDIA/Megatron-LM https://github.com/NVIDIA/Megatron-LM . If they really managed to get 95%, it's a huge advance in the state of the art. I guess by training time they meant with respect to downtime of the hardware, not utilisation of the hardware potential.
- Aissen 2y ago> I guess by training time they meant with respect to downtime of the hardware, not utilisation of the hardware potential. That was my understanding as well, those are two different levels of efficiency.
- WithinReason 2y agoThat means that 5% of the time GPUs were not doing any training, e.g. the server was busy with checkpointing so the training paused. 40% is according to Karpathy: https://twitter.com/karpathy/status/1781028605709234613 https://twitter.com/karpathy/status/1781028605709234613