3 ms·
> very long time On one of Meta's 24k H100 clusters running at 95% efficiency, it's 2.3 days.
by Aissen 3y ago
> very long time
On one of Meta's 24k H100 clusters running at 95% efficiency, it's 2.3 days.
- WithinReason 3y agoIt was closer to 40% efficiency AFAIK
- Aissen 3y agoI'm curious, do you have any pointers? This is what this article mentions: > Those improvements resulted in an overall effective training time of more than 95% https://ai.meta.com/blog/meta-llama-3/ https://ai.meta.com/blog/meta-llama-3/
- logicchains 3y agoThe known SOTA for GPU flops utilisation for training on that many GPUs is somewhere between 50-60%, e.g. https://github.com/NVIDIA/Megatron-LM https://github.com/NVIDIA/Megatron-LM . If they really managed to get 95%, it's a huge advance in the state of the art. I guess by training time they meant with respect to downtime of the hardware, not utilisation of the hardware potential.
- Aissen 3y ago> I guess by training time they meant with respect to downtime of the hardware, not utilisation of the hardware potential. That was my understanding as well, those are two different levels of efficiency.
- WithinReason 3y agoThat means that 5% of the time GPUs were not doing any training, e.g. the server was busy with checkpointing so the training paused. 40% is according to Karpathy: https://twitter.com/karpathy/status/1781028605709234613 https://twitter.com/karpathy/status/1781028605709234613