We noticed that the training run crashed because one of the GPUs fell off the bus. Power cycling the host server didn't help and diagnostics showed thermal dama

We noticed that the training run crashed because one of the GPUs fell off the bus. Power cycling the host server didn't help and diagnostics showed thermal damage. We were able to swap in a different node, but apparently the entire host server needed to be replaced. We've generally noticed a relatively high failure rate for H100 hardware and I'm not quite sure what is behind that.