6 ms·

Fail-slow at scale: Evidence of hardware performance degradation in prod systems

64 points by godelmachine 9y ago

LeonM 9y ago
> There was a case where a fan in a compute node stopped working, and to compensate this failing fan, fans in other compute nodes started to operate at their maximal speed, which then generated heavy noise and vibration that degraded the disk performance. I've never heard of such thing, can vibrations from a fan really influence disk performance?
- blattimwind 9y ago
  Yes. Loud noises and even yelling degrade disk performance as well. https://www.youtube.com/watch?v=tDacjrSCeq4 https://www.youtube.com/watch?v=tDacjrSCeq4
  - joncrane 9y ago
    This is one of the classics! I love it. Screaming at a HDD.
  - godelmachine 9y ago
    I started screaming in my cubicle watching this.
- vardump 9y ago
  Yes. Mechanical disks are very sensitive to vibrations.
- meuk 9y ago
  In the days of burning CD's I was always told that you should be quiet and not touch the PC when you were burning. In my experience there was indeed a correlation between noise and the success of burning. I'm not sure if this is a placebo effect or not, and if this also holds for reading CD's. On the other hand, if you have a laptop with an SSD (which is non-mechanical) there shouldn't be too much sensitivity to vibrations, I think.
  - janci 9y ago
    I remember how I didn't dare to touch the computer during burning other than move the mouse a few pixels here and there so it has it peace and the screensaver do not kick in. CDs were expensive and burnig was a hour-long process.
    - jebeng 9y ago
      That upgrade path from 2x to 52x was so nice though.
- gpapilion 9y ago
  I previously had a vendor replace hundreds of systems due to a poor fan design where they mounted it under a spinning disk. It is very surprising how sensitive they are.
gpapilion 9y ago
I had a large scale issue caused by a bug in the fan control system reading a temp that was deadlocked. This caused cpu performance to tank, since the cpu was thermal throttled. To make matters worse the environmental sensors reported no issues, because they were all frozen at a single reading as well.
godelmachine 9y ago
Blue screen errors are most often caused by RAM issues. I remember very well that I was facing blue screen error at my work once, & upon searching I found that all I need to do is remove the RAM, clean and dust it, and put it back into the same or another RAM socket.
xkgt 9y ago
This is a HN gem! Wonder how it didn't make it to first page. I once had a mobile app perform very poorly only for large downloads on a particular SSID. No other applications had such issue. It turned out to be due to a firmware bug which scrapped TCP SACK field, but the problem manifested only for large data transfers over unreliable, high latency connections. Got us all chasing our tails for a while though...
riza_on 9y ago
The authors are still collecting more stories for their longer journal version of the paper. If your institution has 10+ stories to contribute and are interested to be part of the journal paper, please go ahead contact the first two authors of the original paper.

64 points by godelmachine 9y ago

> There was a case where a fan in a compute node stopped working, and to compensate this failing fan, fans in other compute nodes started to operate at their maximal speed, which then generated heavy noise and vibration that degraded the disk performance. I've never heard of such thing, can vibrations from a fan really influence disk performance?

blattimwind 9y ago
Yes. Loud noises and even yelling degrade disk performance as well. https://www.youtube.com/watch?v=tDacjrSCeq4 https://www.youtube.com/watch?v=tDacjrSCeq4
- joncrane 9y ago
  This is one of the classics! I love it. Screaming at a HDD.
- godelmachine 9y ago
  I started screaming in my cubicle watching this.
vardump 9y ago
Yes. Mechanical disks are very sensitive to vibrations.
meuk 9y ago
In the days of burning CD's I was always told that you should be quiet and not touch the PC when you were burning. In my experience there was indeed a correlation between noise and the success of burning. I'm not sure if this is a placebo effect or not, and if this also holds for reading CD's. On the other hand, if you have a laptop with an SSD (which is non-mechanical) there shouldn't be too much sensitivity to vibrations, I think.
- janci 9y ago
  I remember how I didn't dare to touch the computer during burning other than move the mouse a few pixels here and there so it has it peace and the screensaver do not kick in. CDs were expensive and burnig was a hour-long process.
  - jebeng 9y ago
    That upgrade path from 2x to 52x was so nice though.
gpapilion 9y ago
I previously had a vendor replace hundreds of systems due to a poor fan design where they mounted it under a spinning disk. It is very surprising how sensitive they are.

gpapilion 9y ago

I had a large scale issue caused by a bug in the fan control system reading a temp that was deadlocked. This caused cpu performance to tank, since the cpu was thermal throttled. To make matters worse the environmental sensors reported no issues, because they were all frozen at a single reading as well.

godelmachine 9y ago

Blue screen errors are most often caused by RAM issues. I remember very well that I was facing blue screen error at my work once, & upon searching I found that all I need to do is remove the RAM, clean and dust it, and put it back into the same or another RAM socket.

xkgt 9y ago

This is a HN gem! Wonder how it didn't make it to first page. I once had a mobile app perform very poorly only for large downloads on a particular SSID. No other applications had such issue. It turned out to be due to a firmware bug which scrapped TCP SACK field, but the problem manifested only for large data transfers over unreliable, high latency connections. Got us all chasing our tails for a while though...

riza_on 9y ago

The authors are still collecting more stories for their longer journal version of the paper. If your institution has 10+ stories to contribute and are interested to be part of the journal paper, please go ahead contact the first two authors of the original paper.