Fair enough. Your original comment did not make this distinction. I dug and found https://www.fiala.me/pubs/papers/sc12-redmpi.pdf https://www.fiala.me/pubs/pa

Fair enough. Your original comment did not make this distinction. I dug and found https://www.fiala.me/pubs/papers/sc12-redmpi.pdf https://www.fiala.me/pubs/papers/sc12-redmpi.pdf the title of which is "Detection and Correction of Silent Data Corruption for Large-Scale High-Performance Computing." They state that for a cray there was a double bit flip about 1x/day for 75k modules. To (probably incorrectly) extrapolate, if you have a server with 16 modules that would be equivalent to a single double failure about once every 13 years.