I did some of the work in the post (though mostly post-setup). Speaking in generalities: the initial failure rates of these units are much higher than those of

I did some of the work in the post (though mostly post-setup). Speaking in generalities: the initial failure rates of these units are much higher than those of traditional non-GPU machines. In general, the failure rates decline significantly during the operating life of hardware. So you deal with a bunch of issues up front that you try to resolve to reach a much more stable state. There was a recent Meta engineering blog post that echoed some of our own experiences wrangling GPUs and high performance networks: https://engineering.fb.com/2024/06/12/data-infrastructure/training-large-language-models-at-scale-meta/ https://engineering.fb.com/2024/06/12/data-infrastructure/tr...