5 ms·

The paper links to this entertaining YouTube video from 2009, in which someone demonstrates the effect by literally shouting at a rack of servers: https://www.

The paper links to this entertaining YouTube video from 2009, in which someone demonstrates the effect by literally shouting at a rack of servers: https://www.youtube.com/watch?v=tDacjrSCeq4 https://www.youtube.com/watch?v=tDacjrSCeq4 (EDIT: replaced the link I found via YouTube's search with the original, thanks robotmlg!)

jboxee 9y ago
that someone is Brendan Gregg! He's awesome! http://www.brendangregg.com/ http://www.brendangregg.com/
rosser 9y ago
Several jobs ago, we were in the data center and doing this came up. We had an idle disk shelf with which to test, so we started up a bulk IO operation (`dd` or something; I don't remember what, specifically), and watched the dip in the IO rate correlate perfectly with someone crouching next to the shelf and screaming at it.
robotmlg 9y ago
That's a reposted video, the original is from Bryan Cantrill (formerly of Sun, now CTO of Joyent): https://www.youtube.com/watch?v=tDacjrSCeq4 https://www.youtube.com/watch?v=tDacjrSCeq4
- btbuildem 9y ago
  Wow.. so, how much interference is caused by the ambient noise of the server room itself? Is there some "critical threshold" for noise beyond which you'd have to install sound dampeners or smth?
  - baruch 9y ago
    I've had a case that caused disk performance to drop because the memory correctable errors rose above a server BMC threshold. Turned out that the BMC beeped the internal speaker which vibrated the chassis at the right harmonic frequency that matches the HDD sensitivity spot and reduced performance. After that we disconnected the speaker on all servers. The overall nose may impact the HDD performance but mostly if the vibration is at the right frequencies. Some are a lot more important than others.
Klathmon 9y ago
Holy shit that's one hell of a demonstration! I wonder if somewhere out there there's a sysadmin pulling their hair out trying to figure out why their I/O shits the bed every now and then.
katastic 9y ago
If shouting works... couldn't the typical LOUD data center noises of other machines and AC, also cause disk latency? I know vibration affects hard drives, but (AFAIK) data center drives are mounted on anti-vibration mounts for exactly that reason. But sound? Goodness!
a012 9y ago
I remembered to read somewhere that the fire suppression in DC could kill HDDs just in the same way (if they use gaseous agents). Never had fire drills in our DC to "confirm" that though.
- lathiat 9y ago
  Ive had that happen. Only killed 1 or 2 disks but it knocked a lot more disks offline requiring a reboot. Not bad out of 2 racks I guess. In that case the valve holding back the gas failed so it was an unexpected release. Would not have wanted to be in there at the time - no warning!

jboxee 9y ago

that someone is Brendan Gregg! He's awesome! http://www.brendangregg.com/ http://www.brendangregg.com/

rosser 9y ago

Several jobs ago, we were in the data center and doing this came up. We had an idle disk shelf with which to test, so we started up a bulk IO operation (`dd` or something; I don't remember what, specifically), and watched the dip in the IO rate correlate perfectly with someone crouching next to the shelf and screaming at it.

robotmlg 9y ago

That's a reposted video, the original is from Bryan Cantrill (formerly of Sun, now CTO of Joyent): https://www.youtube.com/watch?v=tDacjrSCeq4 https://www.youtube.com/watch?v=tDacjrSCeq4

btbuildem 9y ago
Wow.. so, how much interference is caused by the ambient noise of the server room itself? Is there some "critical threshold" for noise beyond which you'd have to install sound dampeners or smth?
- baruch 9y ago
  I've had a case that caused disk performance to drop because the memory correctable errors rose above a server BMC threshold. Turned out that the BMC beeped the internal speaker which vibrated the chassis at the right harmonic frequency that matches the HDD sensitivity spot and reduced performance. After that we disconnected the speaker on all servers. The overall nose may impact the HDD performance but mostly if the vibration is at the right frequencies. Some are a lot more important than others.

Klathmon 9y ago

Holy shit that's one hell of a demonstration! I wonder if somewhere out there there's a sysadmin pulling their hair out trying to figure out why their I/O shits the bed every now and then.

katastic 9y ago

If shouting works... couldn't the typical LOUD data center noises of other machines and AC, also cause disk latency? I know vibration affects hard drives, but (AFAIK) data center drives are mounted on anti-vibration mounts for exactly that reason. But sound? Goodness!

a012 9y ago

I remembered to read somewhere that the fire suppression in DC could kill HDDs just in the same way (if they use gaseous agents). Never had fire drills in our DC to "confirm" that though.

lathiat 9y ago
Ive had that happen. Only killed 1 or 2 disks but it knocked a lot more disks offline requiring a reboot. Not bad out of 2 racks I guess. In that case the valve holding back the gas failed so it was an unexpected release. Would not have wanted to be in there at the time - no warning!