3 ms·
What's the story behind the melted h100? I've been having down clocking issues when using fp8 because of thermals as well.
by ipsum2 3y ago
What's the story behind the melted h100? I've been having down clocking issues when using fp8 because of thermals as well.
- rushingcreek 3y agoWe noticed that the training run crashed because one of the GPUs fell off the bus. Power cycling the host server didn't help and diagnostics showed thermal damage. We were able to swap in a different node, but apparently the entire host server needed to be replaced. We've generally noticed a relatively high failure rate for H100 hardware and I'm not quite sure what is behind that.
- ipsum2 3y agoThe entire server? That's crazy. Are you doing FP8 training or did you encounter this with BF16?
- davidzweig 3y agoCheck PLX chips are getting enough airflow, assuming you have them?
- alecco 3y agoFWIW, 4090 has fp8 throttling issues: https://forums.developer.nvidia.com/t/ada-geforce-rtx-4090-fp8-cublaslt-performance/250737/4 https://forums.developer.nvidia.com/t/ada-geforce-rtx-4090-f...
- taneq 3y agoYeah, pics and story time!