That means that 5% of the time GPUs were not doing any training, e.g. the server was busy with checkpointing so the training paused. 40% is according to Karpath

That means that 5% of the time GPUs were not doing any training, e.g. the server was busy with checkpointing so the training paused. 40% is according to Karpathy: https://twitter.com/karpathy/status/1781028605709234613 https://twitter.com/karpathy/status/1781028605709234613