12 ms·
OVH CEO Octave Klaba speaking about the incident [video]
- iptrans 6y agoCan somebody add [video] to the title?
- 5h 6y ago[video - poor audio] maybe edit: That comment was snide, my heart goes out to the OVH team, the message within the video was good, forthright & honest. I hope it will be well received by their customers - just a shame it's a bit difficult to listen to!
- FDSGSG 6y agoIsn't that already implied by the title?
- dang 6y agoSure. Done.
- gautamcgoel 6y agoWhat incident?
- aidos 6y agoThey had a fire.
- 1123581321 6y ago...sale.
- kevinmgranger 6y agohttps://news.ycombinator.com/item?id=26407323 https://news.ycombinator.com/item?id=26407323
- sergiotapia 6y agoIs it true that OVH has literally all data in that single datacenter?
- FDSGSG 6y agoNo. OVH has 27(-1) datacenters.
- tyingq 6y agoI'm guessing the other two SBG datacenters took some kind of hit. The buildings are very close together: https://cdn.baxtel.com/data-center/ovh-strasbourg-campus/photos/flickity_display_OVH-Strasbourg-Campus-OVH-Strasbourg-Campus-showing-all-3-Buildings-2019.png https://cdn.baxtel.com/data-center/ovh-strasbourg-campus/pho... And the fire looked really hot...like melting steel containers hot: https://pbs.twimg.com/media/EwGqV17XMAMF_wa?format=jpg&name=large https://pbs.twimg.com/media/EwGqV17XMAMF_wa?format=jpg&name=...
- ArchOversight 6y agoSBG 1 had some rooms impacted, SBG 3 and 4 were not impacted at all. However, they need to bring new power online, as well as bring a new network setup/fiber online to get network connectivity.
- Nomikos 6y agoNo, they have 26 left https://www.ovh.com/world/us/about-us/datacenters.xml https://www.ovh.com/world/us/about-us/datacenters.xml
- sergiotapia 6y agoWhat a stupid rumor thanks for clarifying
- batmansmk 6y agoNope. 27 in the world, 10 in France, 1 caught fire, 2 others are stopped for inspection. https://www.ovh.com/world/us/about-us/datacenters.xml https://www.ovh.com/world/us/about-us/datacenters.xml
- tormeh 6y agoThe raw video link is https://www.ovh.com/fr/images/sbg/Octave-Klaba-speaking-en-vid.mp4 https://www.ovh.com/fr/images/sbg/Octave-Klaba-speaking-en-v...
- dang 6y agoThat seems like a more precise URL so I've changed to that from https://www.ovh.com/fr/images/sbg/index-en.html https://www.ovh.com/fr/images/sbg/index-en.html for now. Thanks!
- lovedswain 6y agotl;dr UPS maintenance was performed by a vendor the day before the fire. Fire department used a thermal camera to isolate source of fire, it seemed to originate with 2 UPSes, one of which was the recently maintained UPS
- mgbmtl 6y agoOther random bits: SBG-2 was an older generation datacenter, had ventilation issues? They have 4 other datacenters who have a similar design. Others, including SBG-3, have newer designs. They're building 2500 servers per week. For the offline buildings that are not destroyed, they have to rebuild the electrical distribution and network. It was not clear if they are also moving servers physically.
- verytrivial 6y agoI read his description and literal hand waving as saying that for efficiency reasons, in 2011 they were sort of build like how you'd build a camp fire: convective airflow drawn through and out the top.
- fanf2 6y agoRight, this is the OVH “tower” data center design. Here’s a google maps link to the OVH Strasbourg site https://goo.gl/maps/rQjir8byKNwMDZrm9 https://goo.gl/maps/rQjir8byKNwMDZrm9 with (from south to north) SBG1 (a collection of shipping containers), SBG2 (a figure 8 shape), and SBG3 (a box). The idea of the tower design is to take ambient air in through the outside walls, through I think just one row of servers, then into a central void where convection pulls the air out.
- ikiris 6y agoAKA a draft furnace.
- jeffbee 6y agoThis is taking to extremes my maxim that UPSes cause more outages than they prevent.
- justicezyx 6y agoThis would be one of inherient difference between smaller vs. giga players in cloud hosting. AWS/Google/Azure, if this happens, there should only be limited outage to a small fraction of customers. As a matter of fact, Google had such an incident before, and literally no customers (internal and external) noticed.
- ev1 6y agoThis is an apples to oranges comparison. OVH largely sells bare metal; their public cloud wasn't really impacted. If you are using AWS, Google, or Azure, ran a single (or multiple machines) inside a single AZ with no backups and opted out of snapshots, you would face the exact same situation. I can definitely say I see people complaining about how everything they have is down on AWS when us-east-1 goes down periodically, while large players that deploy sanely like Netflix fail over to another region seamlessly. This [only owning a single machine at all] is what most of their customers whinging the most were doing. People that have actual sane production workloads on AWS or GCP are not going to be running 100% of their workload on a single EC2 instance with no backups. People that are running on OVH are running often things like gameservers etc that monopolise 100% of a physical machine and don't support horizontal scaling. You quite literally cannot force a srcds/hlds server to "load balance" dynamically and fail over on heartbeat. Often they are kids or students too, and the $30/m for a machine with 32-64GB ram is all they can afford (though this doesn't absolve them of paying $1-2/m more for offsite backups elsewhere) You can provision more physical machines with the OVH API and have them be up in a different city in a minute or two. You get linespeed bandwidth between OVH DCs. It's up to you to use it.
- justicezyx 6y agoOh good to know. I don't use OvH, and my limited understanding were from their products page which lists VM style offerings. I had assumed VMs were the major use cases on OvH.
- ev1 6y agoThe pricing on their dedicated servers are cheaper than most 1-2GB VMs on cloud: https://www.ovhcloud.com/en/bare-metal/ https://www.ovhcloud.com/en/bare-metal/ - this is their flagship and most expensive brand, the cheaper ones are even less The funniest tweets demanding their data and saying they'll lose everything are the people running: https://twitter.com/Sensity_RP/status/1369496048998223873 https://twitter.com/Sensity_RP/status/1369496048998223873 - GTA5 multiplayer gameserver that begs for donations. Running on one of the cheaper sub-brands of OVH probably (soyoustart, for GAME ddos-protection). $30-40/machine. https://twitter.com/pdfshift/status/1369550522479480833 https://twitter.com/pdfshift/status/1369550522479480833 - I can't tell if this is a troll https://twitter.com/KatsanosAlex/status/1369501497348812801 https://twitter.com/KatsanosAlex/status/1369501497348812801 - I can't tell if this is a troll
- nickdothutton 6y agoLucky it wasn’t a UPS explosion. https://h2tools.org/lessons/battery-room-explosion https://h2tools.org/lessons/battery-room-explosion
- riffic 6y agoThere are a couple good bell system practice documents covering lead-acid battery hazards: * http://etler.com/docs/bsp-archive/157/157-601-701_I19.pdf http://etler.com/docs/bsp-archive/157/157-601-701_I19.pdf * http://etler.com/docs/bsp-archive/157/157-601-101_I7.pdf http://etler.com/docs/bsp-archive/157/157-601-101_I7.pdf
- bscphil 6y agoI realize it's probably just paranoia on my part but I am terrified of UPSs and can't bring myself to have one in my house. Gigantic batteries lying around in a flammable environment seems way too scary. My always-on server is read-only about 99% of the time, so I just put up with the outages when they happen (about 2 a year). If for some reason my OS eventually gets hosed because of this, I'll rebuild it. Maybe one day if have a basement and some kind of concrete compartment to put the battery in I'll feel a bit better about it... but even then, not much you can do about gas leakage if that's a possibility with more recent UPSs.
- Dylan16807 6y agoA single lead-acid battery is putting out such a tiny amount of hydrogen. Put it in any room. It's a big fat nothing in a fire too: https://www.youtube.com/watch?v=ndfe0c00gwo https://www.youtube.com/watch?v=ndfe0c00gwo Something could short but shorts are possible and dangerous even if no batteries are involved. Or get a LiFePo battery and put it in a metal tub.
- Macha 6y agoThat looks like enough fire that spreading to my wooden floors or desks, carpets or skirting boards seems more likely than not?
- deftnerd 6y agoAre there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in? It's almost tradition to have a rack with UPS's in the bottom and then the rest of the space filled with servers or drive arrays. We wouldn't ever think of putting a tiny backup generator in the bottom of every rack, so why do we put a battery storage system there? Also, with the advances in battery chemistry technology that improve reliability and density, it's only a matter of time until Lithium chemistry batteries are available and that also increases the risk of fire. Is there any reason not to move backup power to another room, or even to a separate structure like how they put backup generators on a pad outside of the building?
- tyingq 6y agoThere are plenty of datacenters with a separate battery room, sure.
- sschueller 6y agoSo far I have never been in one where this was not the case. Also they always have fire suppression.
- mikepurvis 6y agoFor the extreme opposite of that, Google famously trolled everyone 10 years ago by announcing that every one of their servers had its own in-chassis 12V battery: https://www.cnet.com/news/google-uncloaks-once-secret-server-10209580/ https://www.cnet.com/news/google-uncloaks-once-secret-server...
- jeffbee 6y agoYes but Google also later moved to a 48VDC power architecture with the batteries at the bottom of the rack. See http://apec.dev.itswebs.com/Portals/0/APEC%202017%20Files/Plenary/APEC%20Plenary%20Google.pdf http://apec.dev.itswebs.com/Portals/0/APEC%202017%20Files/Pl... page 6.
- 6y ago
- tpmx 6y agoI think the key issue here is that there wasn't a functioning fire suppresssion system in place. Second question: is such a system required for this kind of operation? Maybe?
- bayindirh 6y agoIf you're running an IT operation this big, you should have both fire detection units and oxygen replacing suppressants. That should be mandatory. Otherwise, it'd be very hard to contain. Especially when all your servers have RAID controllers with Li-Ion batteries or supercapacitors or other extremely trigger-happy components. Oh, and cooling systems. You're just kindling the fire with it at the beginning.
- brmgb 6y ago> oxygen replacing suppressants No, they are a major risk to the employees working there. I would much rather have a data center destroyed by for every twenty years without victims than mandating the user of oxygen replacing fire suppressants.
- bayindirh 6y agoI understand the concerns you have, however I think there's a good middle ground. Would you consider the following procedure acceptable? - Isolate all rooms with fire-proof doors. - Keep fire supression system at manual. - When fire breaks try to contain (we have 24h watch). - If fails trigger fire supression system. It has 90 second delay and activated per room. - Leave premsises, make the calls. Fire control and supressant control is not inside the system room. Also fire-proof doors seal the room reasonably well, so chemical doesn't move freely. Also there are better chemicals which break down faster and less harmful to everything. BTW, We use Novec.
- pcl 6y agoOther industries use lock-out keys when people are in harm's way. It seems like it'd be easy enough to design an oxygen-replacement system that has lock-outs that people engage whenever they need to enter the protected rooms. https://en.wikipedia.org/wiki/Lockout–tagout https://en.wikipedia.org/wiki/Lockout–tagout
- makkesk8 6y agoIf it turns out it's the upses that caught fire, one can't help but wonder if it would be a better idea to house the upses/backup power solutions in an adjacent smaller building outfitted with sprinklers perhaps?
- yholio 6y agoThe problem with water in the same place with high power equipment is that it instantly turns the room into a death trap for any personnel, now everything is potentially live. Also, in the first part of a lithium battery fire, dropping water on them is quite explosive. It will eventually quench the fire but on the short run it will make it worse, filling the room with explosive hydrogen and poisonous lithium hydroxide. So when your water sprinklers engage over your UPS, you better be sure there's nobody around: https://www.youtube.com/watch?v=cTJh_bzI0QQ https://www.youtube.com/watch?v=cTJh_bzI0QQ
- walshemj 6y agoYou would not be using water here
- jacquesm 6y agoYou would be using Halon injection in a normal DC, after sufficient warning to employees to gtfo if they hadn't done that already.
- fanf2 6y agoNot halon any more, but instead an ozone-friendly fire suppression gas such as argonite
- jacquesm 6y agoThank you! Totally missed that Halon was 'out'.
- Dylan16807 6y agoThat's basically true but I want to note that halon is a gas that actually suppresses fire while argonite is just there to get rid of oxygen.
- notJim 6y agoNot to be rude, but it's really wild to me that even after all this time during the pandemic, the CEO doesn't have a headset he can use so that the audio is intelligible. It's gotta be one of the highest ROI investments you can possibly make at this point.
- abluecloud 6y agoa functioning fire suppression system would have probably been better but second to that i guess a mic would be a good investment
- Thaxll 6y agoSo putting water on servers don't destroy them?
- deleted 6y ago[deleted]
- LinuxBender 6y agoThis is typically a FM200 system in the datacenter [1] which is a pressurized gas. There are also often water "dry-pipes" in place that can pressurize if the FM200 fails. Fire suppression codes vary by country. [1] - https://en.wikipedia.org/wiki/Automatic_fire_suppression https://en.wikipedia.org/wiki/Automatic_fire_suppression
- __turbobrew__ 6y agoThere are gas based fire suppression system such as FM-200 and Novec 1230. If you aren't using gas based fire suppression systems it is basically an amateur operation. The small data centre at my work (10-ish racks) has a FM-200 fire suppression system, it really isn't that expensive to set up.
- subssn21 6y agoMost Fir suppression systems used in Electronics are Halon, CO2, or some other O2 replacing substance
- dtx1 6y agoGood to get out a response out quick, but this is too quick, the audio is garbage.
- tyingq 6y agoTwo interesting pictures from the earlier story... How close the 3 data centers are in SBG: https://cdn.baxtel.com/data-center/ovh-strasbourg-campus/photos/flickity_display_OVH-Strasbourg-Campus-OVH-Strasbourg-Campus-showing-all-3-Buildings-2019.png https://cdn.baxtel.com/data-center/ovh-strasbourg-campus/pho... How hot that fire was. I'm pretty sure the orange spots are holes melted in the walls that are made from metal shipping containers: https://pbs.twimg.com/media/EwGqV17XMAMF_wa?format=jpg&name=large https://pbs.twimg.com/media/EwGqV17XMAMF_wa?format=jpg&name=...
- jacquesm 6y agoI love the lack of fluff, lack of political bs and the honesty on display here.