9 ms·
Fail-slow at scale: Evidence of performance faults in large systems (2018)
- AaronFriel 4y agoAs systems become more complex, so too do their failure modes. This paper documents over a half a dozen slow performance modes of SSDs with several different internal failure modes caused by the silicon, firmware, etc. Contrast that with the modes they discovered for CPUs: no internal factors, only external factors such as power delivery or cooling capacity. That's quite a testament to the construction of the most sophisticated manufactured goods on the planet.
- rlupi 4y agoWell, this is just because they were looking for a specific kind of fault (performance regressions); CPUs tend to fail in a different way. (me: 10y+ Google SRE, 5y in ML and Borg SRE teams working very closely with hardware)
- ultrahax 4y agoI've seen this first hand, across datacenter providers. The most common one is some sort of power supply failure that causes the CPUs to under-volt and throttle; the machine won't go down entirely, the clock will just go through the floor. It's common enough we set up alerting to watch for it specifically.