8 ms·

Shouting in the Datacenter (2008) [video]

437 points by mmastrac 4y ago

SulphurSmell 4y ago
I recall this fondly when it came out. I had a few racks of Sun servers at that time...and anyone I showed the video too were convinced it was bullshit. April Fools anyone? However, Brendan and Bryan know their onions...and all the naysayers had to pause and think a bit. It too crazy to believe...and too crazy to not believe. I miss this stuff. Edit: This was done when everyone ran spinning media. I wonder how modern SSDs hold up?
- bayindirh 4y ago
  This works. I tried this on a Sun ZFS7320 under synthetic load to utilize all disks. We all laughed to the video, too.
  - gorkish 4y ago
    I did it on our fishworks box as well, before we put data on it. That could have been an amazing product if not for Oracle.
1-6 4y ago
Old man yells in cloud.
- deleted 4y ago
  [deleted]
- ivandenysov 4y ago
  Who's laughing now?
  - j1br 4y ago
    Amazon
jtwaleson 4y ago
Oh man. Once, when I was a young padawan doing maintenance by myself on one of our three racks in a colocation DC, I dropped a 1U server from about 2m/6ft high. It was scary as hell. While trying to grab it I hit the off button on one of the other servers, causing an outage for about 50 customers. What's more, it fell on the floor and just missed the SAN with the spinning disk array at the bottom of the rack. Everything was fine, including the fallen server, but shit, that was stupid. The vibration alone could have caused a lot of data corruption.
- egwor 4y ago
  I’ve had old drives that a bit of a whack on the side have given them the kick to start working again (such that you can get data off)
  - jacquesm 4y ago
    Old grease, you probably broke the sticking just enough for the motor to overcome it and then after a bit of turning the grease warms up enough that it will liquify.
humanistbot 4y ago
What a classic! One of the best feature demo videos of all time.
marcosdumay 4y ago
Does yelling at your disks shorten their lifespan?
- 0xbadcafebee 4y ago
  Only one way to find out... weeks later "Yeah, we had to let him go. Too burned out. Caught him screaming at the SAN."
  - willcipriano 4y ago
    End up like this guy: https://m.youtube.com/watch?v=ghv31AUufNk https://m.youtube.com/watch?v=ghv31AUufNk
dmazin 4y ago
In case anyone needs to know, the shouter in question is none other than Brendan Gregg (of flame graph, eBPF, and Systems Performance fame). And the guy who took the video is Bryan Cantrill of Oxide Computer, DTrace, and Joyent.
- devonkim 4y ago
  I think Brendan is still peeved that this video is his most viewed video on Youtube even after all this time.
  - sydney6 4y ago
    Yes, whilst he has given so many other valuable talks around systems profiling, etc.. I firmly believe that this [1] still should be hanging in every system administrators bedroom around. https://www.brendangregg.com/Perf/linux_observability_tools.png https://www.brendangregg.com/Perf/linux_observability_tools....
    - devonkim 4y ago
      I do have it hanging in my office rather than my bedroom. I wouldn't recommend putting it in the bedroom unless one's spouse is also a sysadmin.
    - twic 4y ago
      Maybe he should instrument his talks to see how much shouting there is in them, and check how that correlates with view count.
      - devonkim 4y ago
        This is a flame graph I'd be interested in seeing from him on April Fool's Day
    - jacquesm 4y ago
      I think mcelog/rasdaemon deserve a mention there in the RAM section?
tomalaci 4y ago
I have a colleague who was on the on-call rotation for some of our systems hosted in a datacenter our company owned. During this rotation he got unlucky in that the datacenter detected fire and decided to disperse gas that displaces oxygen (I think it was Halon gas?). Those systems are no joke - you have a short warning before you are supposed to put on a mask or get out of the server room otherwise you will just suffocate. Anyways, no fire damage was detected but oh boy... he had to run around various tech stores left and right to buy about 50-100 enterprise grade spinning disks before returning to the site. The fire suppression system somehow broke most of the hard drives in the server room. This was more than a decade ago, however. Most of the disks are replaced with SSDs and so far they seem to be much more resilient than the spinning ones.
- treffer 4y ago
  I was once told by a datacenter operator that this is due to the pressure change when they pump the gas in and that we should plan on replacing all HDDs should the fire alarm go off. ...and I think it was Argon...
  - tomalaci 4y ago
    Thanks for jogging my memory! It was indeed Argon, not Halon, and the pressure issues that broke HDDs.
    - treffer 4y ago
      And the other comment made me google noise vs. pressure.... Looks like Siemens did an investigation and tested various parameters (pressure, noise, vibration) independently. They point towards noise, not pressure. Really interesting as I am _sure_ I was told it is pressure. But that might be 100% wrong! https://virz.de/wp-content/uploads/2020/05/2015-White-Paper-Potenzielle-Probleme-mit-Computerfestplatten-beim-Einsatz-von-Feuerlo%CC%88schanlagen.pdf https://virz.de/wp-content/uploads/2020/05/2015-White-Paper-...
      - TylerE 4y ago
bitwize 4y ago
Fus ro data loss!
- voidfunc 4y ago
  Underappreciated comment
dinom 4y ago
Lol, this was brought up before... at least one time: https://hn.algolia.com/?dateRange=all&query=tDacjrSCeq4&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&query=tDacjrSCeq4&sort...
- mynameisvlad 4y ago
  > Please don't complain that a submission is inappropriate. If a story is spam or off-topic, flag it. Don't feed egregious comments by replying; flag them instead. If you flag, please don't also comment that you did. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
anfractuosity 4y ago
There's an interesting little paper 'Acoustic Denial of Service Attacks on HDDs' where they use a function generator to find resonant frequencies which cause issues with spinning disks
- segfaultbuserr 4y ago
  There's also a paper Hard Drive of Hearing: Disks that Eavesdrop with a Synthesized Microphone. An APT attacker can potentially reflash the firmware on an HDD and using it as a microphone for eavesdropping. https://andrewkwong.org/docs/Kwong-HDDphone-IEEE-SP-2019.pdf https://andrewkwong.org/docs/Kwong-HDDphone-IEEE-SP-2019.pdf
- jmarneweck 4y ago
  Janet Jackson's song Rhythm Nation caused certain 5400 rpm HDD's in laptops to crash: https://devblogs.microsoft.com/oldnewthing/20220816-00/?p=106994 https://devblogs.microsoft.com/oldnewthing/20220816-00/?p=10...
anonymousiam 4y ago
Same effect mentioned here: https://arstechnica.com/gadgets/2022/08/janet-jacksons-rhythm-nation-is-officially-a-security-threat-for-some-old-laptops/ https://arstechnica.com/gadgets/2022/08/janet-jacksons-rhyth... and here: https://devblogs.microsoft.com/oldnewthing/20220816-00/?p=106994 https://devblogs.microsoft.com/oldnewthing/20220816-00/?p=10...
somecommit 4y ago
Yeah I noticed this phenomenon already, every times I'm shouting at my collaborators they start crying and procrastinating, classic.
dmitryminkovsky 4y ago
I wouldn't want to spend too much time in there without ear protection! Is it common for data centers to be this loud?
- c0balt 4y ago
  Yeah, the fan noise is that loud. You should use hearing protection in there, even when staying for a short amount of time. Though I have heard about some special DCs that use water cooling and as such can defer the noise away from the racks.
- kqr 4y ago
  Sure -- especially in the hot aisles it can be hard to have conversations. I doubt it's much worse than a nightclub though.
- quercusa 4y ago
  Eh? Certainly back when all the disks spun. That and the server fans and the A/C make quite the racket.
- yellowapple 4y ago
  I've been in my share of datacenters and most of 'em are at least a little bit quieter now. Still loud, but not to the extent that you have to shout constantly.
- guenthert 4y ago
  It varies. Once I've been at a customer's data center with raised floors some ten years ago. It felt like an hurricane when you removed one of the tiles. Loud and cold (ear protection and wind breaker mandatory). Hard to communicate there, harder still to think. Not a nice place to be. Keep it in mind when choosing your career.
  - namibj 4y ago
    I got a "3M Peltor X5A" a couple years back as a "thinking cap": put on to think in thought-hostile environments. Works like a charm. Probably best 30 bucks I ever spent. For comms use a wind-proof mic on a headset boom and earphones under the X5A.
bcantrill 4y ago
Pretty funny to see this video at the top of HN! My kids still think I'm an idiot for not having somehow parlayed the enduring popularity of this video into being a YouTuber -- despite my protestations that I was merely the videographer here. If anyone is curious about the video's origins, I had a fun conversation about it with Ben Sigelman a few years ago.[0] [0] https://www.youtube.com/watch?v=_IYzD_NR0W4 https://www.youtube.com/watch?v=_IYzD_NR0W4
- panick21_ 4y ago
  Will oxide racks have a mode where you can make them slow down when you shout at them?
- pinklady 4y ago
  that was really funny
  - Noumenon72 4y ago
    Make sure to read the first few YouTube comments for the full humor
    - merek 4y ago
      Best comment IMO > Even soullsess machine cant work properly when it`s being shouted at
    - pinklady 4y ago
      I'll be sure to check it out
- acj 4y ago
  28:45 in the video. https://youtu.be/_IYzD_NR0W4?t=1725 https://youtu.be/_IYzD_NR0W4?t=1725
- timvdalen 4y ago
  Heard about this on the Changelog podcast recently, great interview!
arberx 4y ago
Its a Solaris machine...not surprised
- rubiquity 4y ago
  Can you point me to your references that indicate HDDs running Solaris are the only ones prone to loud noises disrupting their mechanical nature?
  - arberx 4y ago
    It's a joke
annoyingnoob 4y ago
We do Vibration Testing where I work (think satellite parts, simulating launch, etc). We moved to SSDs a long time ago because some of our test gear would shutdown at certain frequencies/levels during vib tests.
roxaaaane 4y ago
Always makes me laugh ! We need more funny content on the top page!
NegativeLatency 4y ago
Reminds me of the mummy shouting thing: https://www.youtube.com/watch?v=G8qhrURrQbI https://www.youtube.com/watch?v=G8qhrURrQbI
brendangregg 4y ago
FWIW I did a "making of" video soon after, in 2009: https://www.youtube.com/watch?v=lMPozJFC8g0 https://www.youtube.com/watch?v=lMPozJFC8g0 . Only bit I missed discussing there was that I did try to download freeware frequency generators wondering if there was some "rust note" of HDD harmonics, but those I downloaded didn't work and out of frustration I decided to start yelling instead.
- kregasaurusrex 4y ago
  Thanks for sharing!
kodon 4y ago
I got all solid state drives so I can scream non-stop
issung 4y ago
This reminds me of an old favourite video, also in a Data Centre, also low-tier virality: https://www.youtube.com/watch?v=LeFtwtvy4Wc https://www.youtube.com/watch?v=LeFtwtvy4Wc Any other Data Centre classics I should watch?
gala8y 4y ago
Years ago someone had an idea to put electron microscope in a basement of a uni building next to a busy, city-center street in Cracow, Poland, only to be able to use it during nights due to unstable images. Fun times.
DjlbrilH 4y ago
rkuykendall-com 4y ago
Does this mean you could improve datacenter performance and lifespan by installing acoustic tiles and such?

437 points by mmastrac 4y ago

I recall this fondly when it came out. I had a few racks of Sun servers at that time...and anyone I showed the video too were convinced it was bullshit. April Fools anyone? However, Brendan and Bryan know their onions...and all the naysayers had to pause and think a bit. It too crazy to believe...and too crazy to not believe. I miss this stuff. Edit: This was done when everyone ran spinning media. I wonder how modern SSDs hold up?

bayindirh 4y ago
This works. I tried this on a Sun ZFS7320 under synthetic load to utilize all disks. We all laughed to the video, too.
- gorkish 4y ago
  I did it on our fishworks box as well, before we put data on it. That could have been an amazing product if not for Oracle.

1-6 4y ago

Old man yells in cloud.

deleted 4y ago
[deleted]
ivandenysov 4y ago
Who's laughing now?
- j1br 4y ago
  Amazon

jtwaleson 4y ago

Oh man. Once, when I was a young padawan doing maintenance by myself on one of our three racks in a colocation DC, I dropped a 1U server from about 2m/6ft high. It was scary as hell. While trying to grab it I hit the off button on one of the other servers, causing an outage for about 50 customers. What's more, it fell on the floor and just missed the SAN with the spinning disk array at the bottom of the rack. Everything was fine, including the fallen server, but shit, that was stupid. The vibration alone could have caused a lot of data corruption.

egwor 4y ago
I’ve had old drives that a bit of a whack on the side have given them the kick to start working again (such that you can get data off)
- jacquesm 4y ago
  Old grease, you probably broke the sticking just enough for the motor to overcome it and then after a bit of turning the grease warms up enough that it will liquify.

humanistbot 4y ago

What a classic! One of the best feature demo videos of all time.

marcosdumay 4y ago

Does yelling at your disks shorten their lifespan?

0xbadcafebee 4y ago
Only one way to find out... weeks later "Yeah, we had to let him go. Too burned out. Caught him screaming at the SAN."
- willcipriano 4y ago
  End up like this guy: https://m.youtube.com/watch?v=ghv31AUufNk https://m.youtube.com/watch?v=ghv31AUufNk

dmazin 4y ago

In case anyone needs to know, the shouter in question is none other than Brendan Gregg (of flame graph, eBPF, and Systems Performance fame). And the guy who took the video is Bryan Cantrill of Oxide Computer, DTrace, and Joyent.

devonkim 4y ago
I think Brendan is still peeved that this video is his most viewed video on Youtube even after all this time.
- sydney6 4y ago
  Yes, whilst he has given so many other valuable talks around systems profiling, etc.. I firmly believe that this [1] still should be hanging in every system administrators bedroom around. https://www.brendangregg.com/Perf/linux_observability_tools.png https://www.brendangregg.com/Perf/linux_observability_tools....
  - devonkim 4y ago
    I do have it hanging in my office rather than my bedroom. I wouldn't recommend putting it in the bedroom unless one's spouse is also a sysadmin.
  - twic 4y ago
    Maybe he should instrument his talks to see how much shouting there is in them, and check how that correlates with view count.
    - devonkim 4y ago
      This is a flame graph I'd be interested in seeing from him on April Fool's Day
  - jacquesm 4y ago
    I think mcelog/rasdaemon deserve a mention there in the RAM section?

tomalaci 4y ago

I have a colleague who was on the on-call rotation for some of our systems hosted in a datacenter our company owned. During this rotation he got unlucky in that the datacenter detected fire and decided to disperse gas that displaces oxygen (I think it was Halon gas?). Those systems are no joke - you have a short warning before you are supposed to put on a mask or get out of the server room otherwise you will just suffocate. Anyways, no fire damage was detected but oh boy... he had to run around various tech stores left and right to buy about 50-100 enterprise grade spinning disks before returning to the site. The fire suppression system somehow broke most of the hard drives in the server room. This was more than a decade ago, however. Most of the disks are replaced with SSDs and so far they seem to be much more resilient than the spinning ones.

treffer 4y ago
I was once told by a datacenter operator that this is due to the pressure change when they pump the gas in and that we should plan on replacing all HDDs should the fire alarm go off. ...and I think it was Argon...
- tomalaci 4y ago
  Thanks for jogging my memory! It was indeed Argon, not Halon, and the pressure issues that broke HDDs.
  - treffer 4y ago
    And the other comment made me google noise vs. pressure.... Looks like Siemens did an investigation and tested various parameters (pressure, noise, vibration) independently. They point towards noise, not pressure. Really interesting as I am _sure_ I was told it is pressure. But that might be 100% wrong! https://virz.de/wp-content/uploads/2020/05/2015-White-Paper-Potenzielle-Probleme-mit-Computerfestplatten-beim-Einsatz-von-Feuerlo%CC%88schanlagen.pdf https://virz.de/wp-content/uploads/2020/05/2015-White-Paper-...
    - TylerE 4y ago

bitwize 4y ago

Fus ro data loss!

voidfunc 4y ago
Underappreciated comment

dinom 4y ago

Lol, this was brought up before... at least one time: https://hn.algolia.com/?dateRange=all&query=tDacjrSCeq4&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&query=tDacjrSCeq4&sort...

mynameisvlad 4y ago
> Please don't complain that a submission is inappropriate. If a story is spam or off-topic, flag it. Don't feed egregious comments by replying; flag them instead. If you flag, please don't also comment that you did. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

anfractuosity 4y ago

There's an interesting little paper 'Acoustic Denial of Service Attacks on HDDs' where they use a function generator to find resonant frequencies which cause issues with spinning disks

segfaultbuserr 4y ago
There's also a paper Hard Drive of Hearing: Disks that Eavesdrop with a Synthesized Microphone. An APT attacker can potentially reflash the firmware on an HDD and using it as a microphone for eavesdropping. https://andrewkwong.org/docs/Kwong-HDDphone-IEEE-SP-2019.pdf https://andrewkwong.org/docs/Kwong-HDDphone-IEEE-SP-2019.pdf
jmarneweck 4y ago
Janet Jackson's song Rhythm Nation caused certain 5400 rpm HDD's in laptops to crash: https://devblogs.microsoft.com/oldnewthing/20220816-00/?p=106994 https://devblogs.microsoft.com/oldnewthing/20220816-00/?p=10...

anonymousiam 4y ago

Same effect mentioned here: https://arstechnica.com/gadgets/2022/08/janet-jacksons-rhythm-nation-is-officially-a-security-threat-for-some-old-laptops/ https://arstechnica.com/gadgets/2022/08/janet-jacksons-rhyth... and here: https://devblogs.microsoft.com/oldnewthing/20220816-00/?p=106994 https://devblogs.microsoft.com/oldnewthing/20220816-00/?p=10...

somecommit 4y ago

Yeah I noticed this phenomenon already, every times I'm shouting at my collaborators they start crying and procrastinating, classic.

dmitryminkovsky 4y ago

I wouldn't want to spend too much time in there without ear protection! Is it common for data centers to be this loud?

c0balt 4y ago
Yeah, the fan noise is that loud. You should use hearing protection in there, even when staying for a short amount of time. Though I have heard about some special DCs that use water cooling and as such can defer the noise away from the racks.
kqr 4y ago
Sure -- especially in the hot aisles it can be hard to have conversations. I doubt it's much worse than a nightclub though.
quercusa 4y ago
Eh? Certainly back when all the disks spun. That and the server fans and the A/C make quite the racket.
yellowapple 4y ago
I've been in my share of datacenters and most of 'em are at least a little bit quieter now. Still loud, but not to the extent that you have to shout constantly.
guenthert 4y ago
It varies. Once I've been at a customer's data center with raised floors some ten years ago. It felt like an hurricane when you removed one of the tiles. Loud and cold (ear protection and wind breaker mandatory). Hard to communicate there, harder still to think. Not a nice place to be. Keep it in mind when choosing your career.
- namibj 4y ago
  I got a "3M Peltor X5A" a couple years back as a "thinking cap": put on to think in thought-hostile environments. Works like a charm. Probably best 30 bucks I ever spent. For comms use a wind-proof mic on a headset boom and earphones under the X5A.

bcantrill 4y ago

Pretty funny to see this video at the top of HN! My kids still think I'm an idiot for not having somehow parlayed the enduring popularity of this video into being a YouTuber -- despite my protestations that I was merely the videographer here. If anyone is curious about the video's origins, I had a fun conversation about it with Ben Sigelman a few years ago.[0] [0] https://www.youtube.com/watch?v=_IYzD_NR0W4 https://www.youtube.com/watch?v=_IYzD_NR0W4

panick21_ 4y ago
Will oxide racks have a mode where you can make them slow down when you shout at them?
pinklady 4y ago
that was really funny
- Noumenon72 4y ago
  Make sure to read the first few YouTube comments for the full humor
  - merek 4y ago
    Best comment IMO > Even soullsess machine cant work properly when it`s being shouted at
  - pinklady 4y ago
    I'll be sure to check it out
acj 4y ago
28:45 in the video. https://youtu.be/_IYzD_NR0W4?t=1725 https://youtu.be/_IYzD_NR0W4?t=1725
timvdalen 4y ago
Heard about this on the Changelog podcast recently, great interview!

arberx 4y ago

Its a Solaris machine...not surprised

rubiquity 4y ago
Can you point me to your references that indicate HDDs running Solaris are the only ones prone to loud noises disrupting their mechanical nature?
- arberx 4y ago
  It's a joke

annoyingnoob 4y ago

We do Vibration Testing where I work (think satellite parts, simulating launch, etc). We moved to SSDs a long time ago because some of our test gear would shutdown at certain frequencies/levels during vib tests.

roxaaaane 4y ago

Always makes me laugh ! We need more funny content on the top page!

NegativeLatency 4y ago

Reminds me of the mummy shouting thing: https://www.youtube.com/watch?v=G8qhrURrQbI https://www.youtube.com/watch?v=G8qhrURrQbI

brendangregg 4y ago

FWIW I did a "making of" video soon after, in 2009: https://www.youtube.com/watch?v=lMPozJFC8g0 https://www.youtube.com/watch?v=lMPozJFC8g0 . Only bit I missed discussing there was that I did try to download freeware frequency generators wondering if there was some "rust note" of HDD harmonics, but those I downloaded didn't work and out of frustration I decided to start yelling instead.

kregasaurusrex 4y ago
Thanks for sharing!

kodon 4y ago

I got all solid state drives so I can scream non-stop

issung 4y ago

This reminds me of an old favourite video, also in a Data Centre, also low-tier virality: https://www.youtube.com/watch?v=LeFtwtvy4Wc https://www.youtube.com/watch?v=LeFtwtvy4Wc Any other Data Centre classics I should watch?

gala8y 4y ago

Years ago someone had an idea to put electron microscope in a basement of a uni building next to a busy, city-center street in Cracow, Poland, only to be able to use it during nights due to unstable images. Fun times.

DjlbrilH 4y ago

rkuykendall-com 4y ago

Does this mean you could improve datacenter performance and lifespan by installing acoustic tiles and such?