4 ms·

I have a colleague who was on the on-call rotation for some of our systems hosted in a datacenter our company owned. During this rotation he got unlucky in that

I have a colleague who was on the on-call rotation for some of our systems hosted in a datacenter our company owned. During this rotation he got unlucky in that the datacenter detected fire and decided to disperse gas that displaces oxygen (I think it was Halon gas?). Those systems are no joke - you have a short warning before you are supposed to put on a mask or get out of the server room otherwise you will just suffocate. Anyways, no fire damage was detected but oh boy... he had to run around various tech stores left and right to buy about 50-100 enterprise grade spinning disks before returning to the site. The fire suppression system somehow broke most of the hard drives in the server room. This was more than a decade ago, however. Most of the disks are replaced with SSDs and so far they seem to be much more resilient than the spinning ones.

treffer 4y ago
I was once told by a datacenter operator that this is due to the pressure change when they pump the gas in and that we should plan on replacing all HDDs should the fire alarm go off. ...and I think it was Argon...
- tomalaci 4y ago
  Thanks for jogging my memory! It was indeed Argon, not Halon, and the pressure issues that broke HDDs.
  - treffer 4y ago
    And the other comment made me google noise vs. pressure.... Looks like Siemens did an investigation and tested various parameters (pressure, noise, vibration) independently. They point towards noise, not pressure. Really interesting as I am _sure_ I was told it is pressure. But that might be 100% wrong! https://virz.de/wp-content/uploads/2020/05/2015-White-Paper-Potenzielle-Probleme-mit-Computerfestplatten-beim-Einsatz-von-Feuerlo%CC%88schanlagen.pdf https://virz.de/wp-content/uploads/2020/05/2015-White-Paper-...
    - TylerE 4y ago
      Noise really is just pressure, with a high frequency time component.
      - samatman 4y ago
        This is particularly noticeable with over-ear noise cancelling headphones. The feeling of anti-sound is quite distinctive.
        thenthenthen 4y ago
        It does feel like ‘pressure’ on your ears/inside your head. Not very comfy actually
- _carbyau_ 4y ago
  My partner worked in a Research School of Chemistry. On the top floor of the building a tip-over failure replacing an argon bottle broke the valve off the end and emptied the compressed lot of Argon in short order. Argon being colourless, ordourless, heavier than air = whole building "anti-suffocation" evacuation. But no fire problem!
jsty 4y ago
> The fire suppression system somehow broke most of the hard drives in the server room Possibly another case of loud noise when the gas is released - see e.g. https://www.theregister.com/2018/04/26/decibels_destroy_disk_drives_in_digiplex_data_centre/ https://www.theregister.com/2018/04/26/decibels_destroy_disk... https://www.bbc.co.uk/news/technology-37337868 https://www.bbc.co.uk/news/technology-37337868
walrus01 4y ago
spinning hard drives of the pre-helium-filled generation are not in fact totally sealed. if you disassemble one you will find the tiny air filter module which covers a hole in the casing for equalizing pressure with the outside air.
- ignaloidas 4y ago
  I wouldn't be surprised if the fact that argon is heavier caused the failure. Argon gets into the drives, assisted by the larger pressure outside of the drive at the time of the release, and it wouldn't come out within a reasonable time even when the pressure came back to normal, since there's not much capability for venting gasses out of the hard drive. And since argon is significantly heavier, it causes larger air drag on the mechanical parts of the drive, eventually making the head driving coils too weak to operate at the needed speeds.
jamiek88 4y ago
> This was more than a decade ago He’d be even more on an inclined plane, wrapped helically around an axis nowadays as there probably isn’t even one never mind multiple local places to buy that amount of enterprise drives. Might be a micro center if you’re lucky.
jacquesm 4y ago
The shockwave from a sudden release like that can take out the better part of a raid array in a matter of moments. I never understood why they have to release it all at once instead of gradually because the number of times that I've seen accidental releases dwarfs the number of real firefighting incidents.

treffer 4y ago

I was once told by a datacenter operator that this is due to the pressure change when they pump the gas in and that we should plan on replacing all HDDs should the fire alarm go off. ...and I think it was Argon...

tomalaci 4y ago
Thanks for jogging my memory! It was indeed Argon, not Halon, and the pressure issues that broke HDDs.
- treffer 4y ago
  And the other comment made me google noise vs. pressure.... Looks like Siemens did an investigation and tested various parameters (pressure, noise, vibration) independently. They point towards noise, not pressure. Really interesting as I am _sure_ I was told it is pressure. But that might be 100% wrong! https://virz.de/wp-content/uploads/2020/05/2015-White-Paper-Potenzielle-Probleme-mit-Computerfestplatten-beim-Einsatz-von-Feuerlo%CC%88schanlagen.pdf https://virz.de/wp-content/uploads/2020/05/2015-White-Paper-...
  - TylerE 4y ago
    Noise really is just pressure, with a high frequency time component.
    - samatman 4y ago
      This is particularly noticeable with over-ear noise cancelling headphones. The feeling of anti-sound is quite distinctive.
      - thenthenthen 4y ago
        It does feel like ‘pressure’ on your ears/inside your head. Not very comfy actually
_carbyau_ 4y ago
My partner worked in a Research School of Chemistry. On the top floor of the building a tip-over failure replacing an argon bottle broke the valve off the end and emptied the compressed lot of Argon in short order. Argon being colourless, ordourless, heavier than air = whole building "anti-suffocation" evacuation. But no fire problem!

jsty 4y ago

> The fire suppression system somehow broke most of the hard drives in the server room Possibly another case of loud noise when the gas is released - see e.g. https://www.theregister.com/2018/04/26/decibels_destroy_disk_drives_in_digiplex_data_centre/ https://www.theregister.com/2018/04/26/decibels_destroy_disk... https://www.bbc.co.uk/news/technology-37337868 https://www.bbc.co.uk/news/technology-37337868

walrus01 4y ago

spinning hard drives of the pre-helium-filled generation are not in fact totally sealed. if you disassemble one you will find the tiny air filter module which covers a hole in the casing for equalizing pressure with the outside air.

ignaloidas 4y ago
I wouldn't be surprised if the fact that argon is heavier caused the failure. Argon gets into the drives, assisted by the larger pressure outside of the drive at the time of the release, and it wouldn't come out within a reasonable time even when the pressure came back to normal, since there's not much capability for venting gasses out of the hard drive. And since argon is significantly heavier, it causes larger air drag on the mechanical parts of the drive, eventually making the head driving coils too weak to operate at the needed speeds.

jamiek88 4y ago

> This was more than a decade ago He’d be even more on an inclined plane, wrapped helically around an axis nowadays as there probably isn’t even one never mind multiple local places to buy that amount of enterprise drives. Might be a micro center if you’re lucky.

jacquesm 4y ago

The shockwave from a sudden release like that can take out the better part of a raid array in a matter of moments. I never understood why they have to release it all at once instead of gradually because the number of times that I've seen accidental releases dwarfs the number of real firefighting incidents.