MFU is indeed very useful. Today we found that while scaling Karpathy’s nanoGPT to multiple H100 nodes the MFU calculation itself was dropping MFU performance![

MFU is indeed very useful. Today we found that while scaling Karpathy’s nanoGPT to multiple H100 nodes the MFU calculation itself was dropping MFU performance![1] Commenting it out improved iter performance by almost 30% 1. https://github.com/modal-labs/multinode-training-guide/blob/main/nanoGPT/model.py https://github.com/modal-labs/multinode-training-guide/blob/...