The known SOTA for GPU flops utilisation for training on that many GPUs is somewhere between 50-60%, e.g. https://github.com/NVIDIA/Megatron-LM https://github.c

The known SOTA for GPU flops utilisation for training on that many GPUs is somewhere between 50-60%, e.g. https://github.com/NVIDIA/Megatron-LM https://github.com/NVIDIA/Megatron-LM . If they really managed to get 95%, it's a huge advance in the state of the art. I guess by training time they meant with respect to downtime of the hardware, not utilisation of the hardware potential.