7 ms·
> Fire destroyed SBG2 This is crazy. SBG2 was HUGE and if this isn't a translation error on the part of Octave (which I could understand given the stress and
by AdamJacobMuller 6y ago
> Fire destroyed SBG2
This is crazy.
SBG2 was HUGE and if this isn't a translation error on the part of Octave (which I could understand given the stress and ESL) I have a hard time fathoming what kind of fire could destroy a whole facility with nearly 1000 racks of equipment spread out across separated halls.
I'm really hoping "destroyed" here means "we lost all power and network core and there's smoke/fire/physical damage to SOME of that"
I can't even fathom a worst-case scenario of a transformer explosion (which does occur and I've seen the aftermath of) having this big of an impact. Datacenters are built to contain and mitigate these kinds of issues. Fire breaks, dry-pipe sprinkler systems and fire-extinguishing gas systems are all designed to prevent a fire from becoming large-scale.
Really glad nobody was hurt. OVH is gonna have a bad time cleaning all this up.
- qbasic_forever 6y agoJust guessing, but maybe a fire suppression system going off could wipe out all the machines? The couple datacenters I've been inside were small, old and used halon gas which wasn't supposed to destroy the machines. No idea how it works in big places these days.
- atkbrah 6y agoA few years back there was an incident in Sweden where noise coming from the gas based fire suppression system going off destroyed hard drives [1]. 1. https://www.theregister.com/2018/04/26/decibels_destroy_disk_drives_in_digiplex_data_centre/ https://www.theregister.com/2018/04/26/decibels_destroy_disk...
- abrookewood 6y agoI've also seen a weird video (lost to time unfortunately), where someone showed that they could yell at their servers in a data centre and introduce errors (or something similar). Was very strange to see, but they had a console up clearly showing that it was having an impact.
- regecks 6y agohttps://www.youtube.com/watch?v=tDacjrSCeq4&fmt=18 https://www.youtube.com/watch?v=tDacjrSCeq4&fmt=18
- lrem 6y agoBecause of course it was Bryan :D
- jeffrallen 6y agoBrian just giving his opinion on anything would give hard drives errors... That guy loves to run his mouth (and I love to listen).
- mwcampbell 6y agoActually, although the video is on Bryan Cantrill's YouTube account, that was Brendan Gregg.
- bcantrill 6y agoI was merely the videographer! For anyone who is curious, I discussed the backstory of that video with Ben Sigelman.[0] [0] https://www.youtube.com/watch?v=_IYzD_NR0W4#t=28m46s https://www.youtube.com/watch?v=_IYzD_NR0W4#t=28m46s
- atkbrah 6y agoI actually planned to include a link to this video in my original response but then left it out. Thanks for posting!
- gcbirzan 6y agoING in Romania had the same issue: https://www.datacenterdynamics.com/en/news/noise-from-fire-drill-breaks-ing-bank-data-center/ https://www.datacenterdynamics.com/en/news/noise-from-fire-d...
- yread 6y agoWe had the same issue at a customer site. To add since the decibels were outside the rated environment the warranty was void on the harddisks and they had to be replaced even if they were not destroyed.
- dylan604 6y agoIf a fire supression kicks in or the fire department shows up with their hoses, would they still say the fire destroyed it or just say destroyed due to fire and water damage? Also, fire suppression system do fail. There was an infamous incident in LA for one of the studios. They built a warehouse to be a tape vault with tapes going back to the 80s. A fire started, but the suppression system failed because there was not enough pressure in the system. Total loss. Got to keep your safety equipment properly tested!
- proggy 6y agoIt wasn’t just “tapes going back to the 80s.” Those were just the media Universal initially admitted to losing. No, that building was the mother lode. It had film archives going back over 50 years, and worst of all — unreleased audio tape masters for thousands upon thousands of recording artists. The amount of “remastered” album potential that fire destroyed is probably in the billions of dollars, let alone the historical loss of all those recordings by historical persons that will never be heard due to a failure of a fire prevention system. Fascinating case study in why you should never put all your eggs in one basket. https://en.m.wikipedia.org/wiki/2008_Universal_Studios_fire https://en.m.wikipedia.org/wiki/2008_Universal_Studios_fire
- arp242 6y agoIn 1996 Surinam lost many of their government archives after a fire burned the building down. I can find surprisingly few English-language resources for this (only Dutch ones); guess it's a combination of a small country + before the internet really took off.
- AdamJacobMuller 6y agoThis is why I'm such a fan of digitizing. If you have 1 film master you effectively have none. Do some 8K or 16K scans of that master and effectively manage the resulting data and you're effectively 100% immune from future loss in perpetuity. Losing parts of history like that is tragic.
- lrem 6y agoThere's a problem with testing sprinklers: engaging them can be damaging to contents and even structures. So, we're talking about completely emptying the facility, then taking it offline to dry for a time. I've never heard about this being done to anything that was already operational (but I wasn't researching this either).
- rgj 6y agoNo, "destroyed" looks like this: https://mobile.twitter.com/abonin_DNA/status/1369538028243456000 https://mobile.twitter.com/abonin_DNA/status/136953802824345... EDIT a picture from earlier this night https://pbs.twimg.com/media/EwGqV17XMAMF_wa?format=jpg&name=large https://pbs.twimg.com/media/EwGqV17XMAMF_wa?format=jpg&name=...
- baq 6y agoI’d say destroyed is an adequate word, yes.
- AdamJacobMuller 6y agoI'm really shocked, that's an incredible amount of damage. Lots of people lost data.
- baggy_trough 6y agoThe pictures seem pretty clear that a lot of it is gone, judging by the blackened holes in the walls with firehoses being sprayed into.
- ikiris 6y agoThis was effectively posted on the outages list as well by someone trustworthy. The pictures also look pretty bad from the outside.
- sofixa 6y agoOVH design their own datacenters, so it's possible that they missed something or some system or another didn't work as intended, thus the heavy damage.
- rgj 6y agoThey did not have a fire suppression system, only smoke detection. So yeah, they missed something.
- iagovar 6y agoWhere did you get this?
- rgj 6y agohttps://www.ovh.com/world/us/about-us/datacenters.xml https://www.ovh.com/world/us/about-us/datacenters.xml
- draugadrotten 6y agoThey say different here: "The rooms are also equipped with state of the art fire detection and extinction systems." https://www.ovh.com/world/solutions/centres-de-donnees.xml https://www.ovh.com/world/solutions/centres-de-donnees.xml and here "Every data center room is fitted with a fire detection and extinction system, as well as fire doors. OVHcloud complies with the APSAD R4 rule for the installation of mobile and portable fire extinguishers and has the N4 certificate of conformity for all our data centers." https://us.ovhcloud.com/about/company/security https://us.ovhcloud.com/about/company/security
- sofixa 6y agoThat's for its North America DCs, not European ones.
- kuschku 6y agoWhich is understandable, as Halon based fire suppression systems have been illegal for quite some time :/
- hedora 6y agoData centers frequently burn down, or are destroyed by natural disaster. These days, fire suppression systems need to be non-lethal, so inert gasses are out. Water is too, for obvious reasons. Last I checked, they flooded the inside of the DC with airborne powder that coats everything (especially the inside of anything with a running fan). Once that deploys, the machines in the data center are a write off even if the fire was minor.
- nixgeek 6y agoMost datacenters I’ve worked with in the last 5 years use water.
- vinay_ys 6y agoI wonder if they had batteries inside the containers. That can make a bad situation really worse.