15 ms·

OVHcloud fire: SBG2 data center had no extinguisher, no power cut-out

420 points by detaro 5y ago

katekoch 5y ago
[flagged]
ge96 5y ago
I have been using their services for several years now. I remember one time there was a fire and they reimbursed me for the down time.
- karamanolev 5y ago
  Is the reimbursement just pro-rated for the service costs (i.e. 1 day downtime = 1/30th discount of the monthly bill) or something more? If it's just service costs, I find it almost inconsequential compared to the costs to the business of having downtime.
  - ge96 5y ago
    I can see that to be fair my stuff is just personal websites nobody looks at. Edit: aside from my own cheap stuff, I also use AWS but I have not gotten to the scale where if AWS went down the site runs off something else.
  - FDSGSG 5y ago
    This is cheap self-service hosting, the customer is expected to deal with things like backups themselves. If your business suffers from a single DC going down, it's your fault.
catwell 5y ago
Disclaimer: I am French and I had servers (well, my employer did, and I was the admin) in that DC. From the report (not the post): something that really annoyed the firemen is that not only was there no universal electrical cut-off, there were 4 different electrical backups, which they had to figure out how to cut off one by one... It's the same thing as the self-cooling design: OVH optimized for what typically matters in a DC. You want an energy-efficient design and you never want power to go down. Well... except when you do. I suppose by now them and other hosting providers have taken that issue into account and are modifying their DC designs accordingly.
- deleted 5y ago
  [deleted]
- tuananh 5y ago
  seems like they have done an exceptionly well done, high availability power design :D
- Zealotux 5y ago
  I remember back when OVH was smugly mocking anyone who had concerns about their WC system, to be fair the concerns were kind of ridiculous but ultimately their system _did_ fail catastrophically. Glad to have left that company years before the fire, never doing business with them ever again.
  - rosndo 5y ago
    > Glad to have left that company years before the fire, never doing business with them ever again. What alternatives are there? Not many good ones at a comparable price point. (Yeah I know about online.net, Hetzner and so on. OVH offers a much more polished product and vastly better network) At the prices OVH offers, you can have your servers replicated in multiple datacenters for less money than many DCs would charge you for a single server with the same specs. At this price point it should be perfectly fine if a datacenter burns down occasionally. At least their network is otherwise very reliable.
    - xmodem 5y ago
      Hetzner
      - rosndo 5y ago
        But… Hetzner offers an even lower quality service at an even lower price point. (But only to low-volume buyers, OVH offers vastly better volume discounts) It would be strange to expect them to deal with the situation any differently if there was a catastrophic event in one of their DCs resulting in destroyed servers. When working with dedicated servers at this price point, you’re very much expected to deal with your own backups.
bilekas 5y ago
I remember this fire, it was definitely one of the biggest disruptions for us at the time. > it took three hours to cut off the power supply because there was no universal cut-off. This seems really egregious given the nature of data centers, I would argue less so than the wooden ceilings, those were treated to survive an hour long fire, one would thing if the building was grounded with no electricity the fire could have been handled much faster.
detaro 5y ago
(Title truncated from original "OVHcloud fire report: SBG2 data center had wooden ceilings, no extinguisher, and no power cut-out" to fit HN length limit)
londons_explore 5y ago
In a datacenter fire, the risk to human life is very small (has anyone ever died in a datacenter fire, apart from being suffocated by a halon system?). It's just property risk, data loss and service downtime. Therefore, it's a business decision. I have worked in a business that, during a fire, prioritized maintaining service uptime over putting the fire out. The end result: They had to buy more new servers, but customer workloads were migrated away within 15 minutes and saw no outage. For them, it was the right decision.
- jeremyjh 5y ago
  Except the fire damaged other buildings and spread lead all the way to Germany.
  - 0xbeefbeefbeef 5y ago
    The datacenter is about 100ft from the German border.
  - jeroenhd 5y ago
    "All the way to Germany" doesn't say much if the building that's on fire is practically on the German border.
- michaelt 5y ago
  Well, most fires involve risk to human life if firefighters have to enter the building to extinguish it.
  - mannykannot 5y ago
    Indeed, and they also often enter buildings to look for people who are in harm's way. OHVcloud's omissions did not amount to a good business decision, it was an irresponsible one.
  - Nextgrid 5y ago
    Did this one involve said risk? Maybe risk to life was eliminated by a combination of low occupancy, trained staff and adequate escape routes so that there's no reason for firefighters to enter it, and the building was essentially designed to be disposable should the worst happen? I don't see this as a particularly bad thing - if risk to life and neighbouring properties was correctly managed (and it seems like it was here), why not allow this?
staticelf 5y ago
At the time of the fire I used nodechef that hosted their services on OVH and seemingly all their backups as well (in the same datacenter). Turns out when extraordinary events happen promises of backups and such aren't always kept. We lost some data because of that, luckily we had our own backups. A good reminder to make sure you have backups and that they are working correctly. No matter what promises anyone gives you should always have your own backup strategy that's disconnected from the vendor you use. It was Nodechefs fault, not OVH obviously but perhaps it's interesting for others.
- Cthulhu_ 5y ago
  Make sure to read the T's and C's and availability / retention rates closely; it's a process that involves decoding the legalese and trying to associate it with RL situations. Amazon's S3 for example offers a 99.999999999% data durability guarantee, with other bits implying they can withstand a datacenter going up in flames. But there's two caveats there; data availability is lower (so if that datacenter goes up in flame your data may not be lost, but it may also not be directly accessible until they restore their backups), and if they do lose data, what is the consequences to them? It'll be a financial compensation at best.
  - terom 5y ago
    Breaking the S3 durability guarantees badly enough to get press attention would be a critical reputational hit, with the consequences going far beyond just the contractual SLA compensation for the customers. It would still heal with time, though. There was a period after some nasty incidents where the general advice was to avoid EBS for anything critical... but that's over now.
    - lbriner 5y ago
      Would it be critical though? Unless there are lots of alternatives at a similar availability and price-point, most people aren't just going to jump to another provider. Clouds sound largely interchangeable but you try swapping from Azure to AWS or vice-versa: for anything more than the simplest system, this would probably take 12 months+
curiousgal 5y ago
I have been an ovh customer for over 8 years but they kept increasing their prices and lowering their offerings, I finally moved on and have zero sympathy for them.
- lodddgn 5y ago
  What do you mean? OVH has hugely improved their offerings in that time. They introduced world class DDoS protection for all clients. They massively improved their global connectivity, and continue to do so. They’ve drastically improved their support, hiring much much more English speaking staff than they used to have. Not to mention generally improved tooling offered to customers.
  - lbriner 5y ago
    Don't be upset if people aren't going to take your word for it with an account created 46 minutes ago!
    - rosndo 5y ago
      You can confirm all of those things by googling a little.
  - deleted 5y ago
    [deleted]
  - pzduniak 5y ago
    I gave OVH a chance by suggesting it to a client. Prior to that point, I had been using an OVH dedicated server for personal stuff without any issues. We opted for OVH Public Cloud and their Managed Kubernetes (which was considered stable at that point). Kubernetes randomly froze up after deployment, requiring manual intervention from the staff. We had no way to restart the Control Plane on our own. It always took 3+ messages / 1h+ on the phone to reach someone capable of handling it. After months of complaining, they didn't address anything. That was late 2020. What forced us to migrate away was when we got a VM with a broken NVMe SSD that broke PostgreSQL data and OVH refused to acknowledge the problem. Phone support was still practically unreachable. After spending countless hours debugging their crappy services, we just gave up and moved to Oracle Cloud. Best decision ever - their discounts are great and support is excellent. They're clearly trying to challenge the big 3. As we were leaving OVH, we send one last email that finally reached someone who gave a crap about customers. They replied that they will "have a manager talk with us"... I don't think that ever happened. Even if they improve their services, I don't trust them anymore. It was beyond horrible.
fmajid 5y ago
Irresponsible design. It's not just the fire and the damage to businesses, but the report lists concerns about lead from the UPS batteries being spread all the way to Germany as a result of the plume from the fire, as well as in the water from the firemen. Fortunately, in this case, they measured the water and found no significant amounts, nor did the German environmental authorities, but it could easily have been as bad as the Notre-Dame fire where a huge chunk of innermost Paris was contaminated by lead from the destroyed roof.
- userbinator 5y ago
  Of all the things to be the worried about being released by a fire...?! No. Lead is way down on the list. I'd be far more concerned about the other carcinogens from stuff burning.
  - fmajid 5y ago
    It was explicitly listed in the report, so the fire department and environmental defense authorities of both countries do not share your nonchalance.
- mtmail 5y ago
  > all the way to Germany For reference, the border to Germany is about 250m from the building https://www.openstreetmap.org/relation/11917092 https://www.openstreetmap.org/relation/11917092
taubek 5y ago
Aren't there some regulations that would prevent such building to get operating permit?
- krisoft 5y ago
  I don't understand this sentiment. It is an industrial site. Nobody got hurt. The fire didn't spread to other people's property. Why are you all so upset about this?
  - cbg0 5y ago
    It could have spread, and people could have gotten hurt. If you have a useful head on your shoulders, then you should have a proactive mindset and try to prevent issues instead of sending thoughts and prayers after the fact.
    - krisoft 5y ago
      > It could have spread, and people could have gotten hurt. Could it though? It is a standalone installation on an industrial site. There are about ten meter-ish standoff between them and their neighbours. (which is also an industrial site) > If you have a useful head on your shoulders, then you should have a proactive mindset and try to prevent issues And if you have an even more useful head on your shoulders you calibrate your level of upset to the level of actual danger.
      - lbriner 5y ago
        If you think 10m is enough of a fire break, you have no experience of fire. The infra-red heat that comes off of a fire is incredibly intense and can easily spread those distances as that distance, especially if pieces of flammable debris were to bridge the gap. It woul dbe fair enough if that design was recognised and signed-off by the local inspectors, then no-one would complain but it sounds like they were just not doing the risk assessment properly.
      - magicalhippo 5y ago
ovi256 5y ago
The report points out (Image 8) design choices that contributed to a raging fire, once the fire started: * no emergency electricity cut-off device, an "economic strategy choice of the site operator" - the electrical room where the fire started was hot, 400C at the door (measured by thermal camera), with meter long electrical arcing from the door, thundering deafening sounds. Making access of utility technicians "difficult" - it took them 2 hours to cut incoming electrical service from the utility. On site UPS devices also had no cut-off, so they kept supplying. * emergency water network provided only 70m3/h at the site. A firefighting boat arrived, Europa 1, was called, supplying 14.5m3/min max flow rate. The freeflowing air cooling design, a good design choice as it saves on operating costs for cooling, contributed to nourishing the fire.
- stingraycharles 5y ago
  So basically, my takeaway is: * Freeflowing air in DCs is good for cooling( * It’s bad when you have a fire; * An improved DC design would allow an operator to $somehow stop the freeflowing air (although one could argue that it’s not free flowing anymore if one can control it); * I’d like to know how much money really was saved by not allowing the UPSes to be cut off. I’m very curious how the insurance companies respond, and whether they’ll demand e.g. UPS supplies to be able to cut off. Or maybe in general, the fire department should be more aware of these types of trade-offs being made, and give their approval accordingly.
  - lbriner 5y ago
    It is VERY common in buildings to have ventilation dampers that are closed when a fire alarm starts exactly for this reason. I am most surprised that the design sounds like it was made a certain way deliberately even though the basics were got wrong. Does Strasbourg not have any fire regulations that require fireman accessible power cut-outs, automatic extinguishers etc? It doesn't matter just that their equipment burned but the severity of the fire was massively dangerous to the fire brigade and anything else nearby if it couldn't be extinguished properly.
    - 5y ago
GreyStache 5y ago
The gallery in the article links to this drone footage I hadn't seen before: https://www.youtube.com/watch?v=BrGsCD2nVrk https://www.youtube.com/watch?v=BrGsCD2nVrk
ksec 5y ago
And now I am wondering if Hetzner has similar problem or if their DC are better designed. In the old days the two are frequently mentioned together.
jbverschoor 5y ago
Orrr we can finally replace x86 with energy efficient arm chips!
boringg 5y ago
Unrelated tidbit of knowledge i learned yesterday. The first fire brigade was created by Julius Caesars partner/general Crassus. He put together a team of about 500 people to put out fires that happened on a near daily basis in Rome. The catch: he was a land speculator who would flip burnt homes on the cheap so his brigade would run to a fire but until the owner sold to Crassus on the cheap they would watch the home burn. Talk about a hard nose business or completely unethical leverage, wow.
- coopierez 5y ago
  Hopefully 2000 years in the future people will talk about the US healthcare system with similar incredulity.
  - jjeaff 5y ago
    Doesn't seem equivalent, since at least treatment for immediate medical emergencies is given regardless of ability to pay.
- belter 5y ago
  Then Nero decided to make a Game Day...
mwcampbell 5y ago
I wonder if the same corner-cutting happens in the OVHcloud US data centers.
annoyingnoob 5y ago
I deal with a ton of spam/phishing/malware that comes from OVH datacenters - OVH does nothing with complaints. Sometimes you really do reap what you sow.
nervoustwit 5y ago
"This meant that some OVH customers found that their servers continued running after the fire started." So it's not ALL bad news.
nervoustwit 5y ago
"This meant that some OVH customers found that their servers continued running after the fire started." It's not ALL bad news.
3boll 5y ago
For anyone interested, I maintained a server within the affected DC. OVH provided 3x the price of the service for the downtime. But for the recovery we needed to buy a new server from them as our backups were only accesible from dedicated machines... At the end we basically received 2x the price of the server when discounting the temp machine. Communication during the downtime was not bad from OVH side, taking into account the huge amount of affected servers. At the end, as a small customer I cant do or ask for much more. As it's not worth neither the money or the time. IMHO, 3x is not covering any business loses for anyone. We got our own backups within the OVH network and it took 3 additional days to be able to access them as the network was a mess after the fire. That for some business is going to be a huge sum.
- 3boll 5y ago
  Since then I have changed my backup strategy... Now with diff providers. Ready for the next fire;)
  - toast0 5y ago
    Just a heads up that different providers doesn't always mean different buildings. Different cities is more important for a physical disaster than different companies. For operational disasters (BGP/DNS/accounting error/abuse detection false positive/other customer DDoS, etc) different company is useful too, of course.
- hinkley 5y ago
  This is the illusion of SLAs. These money-back-guarantees only refund the cost of the service. They aren't insurance for loss of business. If I'm using your service for $5k a month and making less than $5k a month because of it, my finance people might rightfully ask where my head is at. 2x is better than 1x, but in general I think we are looking for higher rates of return for that. This hardware has to pay for my development and really my entire payroll after all. I also can't trust that you losing $5k an hour motivates you to fix the problem ASAP as me losing $5k an hour, let alone if I'm losing more than that.
- rosndo 5y ago
moralestapia 5y ago
>no general electrical cut-off switch This is so weird I can hardly believe it, maybe some details were lost in the writing. Were they connecting everything directly to the grid? Even the most basic electrical setup goes through a fusebox with switches that turn everything on/off. Perhaps that box was burning as well, or the fire blocked access to it, idk. If there truly was no way to cut power from the site then, wow, that was just an abysmally stupid decision.
- darkwater 5y ago
  I think they refer to the absence of one single mains switch to turn off everything in the datacenter. It took 3 hours for the firefighter to find all the individual switches and turn them off, according to the report.
  - bombcar 5y ago
    There still should be a utility box somewhere near, if necessary the utility should have been able to kill power to the whole sub-division. Every commercial building in the US I've seen has a big huge switch outside (temptation to throw it has been moderate sometimes). The point about UPS continuing to provider power even after the utility is cut is critical, though.
    - jwandborg 5y ago
      Datacenters sometimes have redundant main power providers, I'm guessing that could change the assumption to "there should be a number of supply boxes nearby, which all need to be interfered with".
      - bombcar 5y ago
        Usually then the boxes are heavily marked with how many there are, where the others are, and other info.
- malfist 5y ago
  It said there were meter long electrical arcs in the "power room". If I had to guess, I'd guess there was a shutoff in the power room, but not one accessible outside of it.
encoderer 5y ago
Who else is not surprised that discount hosting providers cut corners?
gunapologist99 5y ago
It's a great thing for startups that OVH provides servers at such an amazing low cost. Yes, there is always a risk that someone will make a mistake in the building design. There's a chance that eliminating some redundancies increases the possibility of a failure. There's always a chance that something bad will go wrong. However, this isn't just a matter of Hanlon's razor (incompetence vs malice), but more of a matter of an intelligent guesstimate of risk versus a lack of knowledge in some areas (wasn't this OVH's very first datacenter?), and a strong focus on reducing costs. Perhaps the latter went too far, and definitely some obvious mistakes were made by not having a universal power cut off of some sort, but dealing with the amount of power on tap in a datacenter is always dangerous, even when there is no fire at all. I'm not saying we need to give OVH a complete pass on this. I'm just saying that there are a lot of extenuating circumstances and, except for the power cut off, it's not clear that OVH made any choices due to extreme negligence or cost-cutting. In other words, they didn't do anything immoral. At worst, it appears that (even from the most anti-OVH party here), this was just a mistake in the design of a new (at the time) style of datacenter, and it did work properly for many years before there was a problem. Making a mistake is not immoral.
- ddaalluu2 5y ago
  "Amazing low cost"? They're one of the most expensive data center operators I know. Hetzner is about half the price. But neither can have more than 3 HDD per machine which is just absurd.
  - dspillett 5y ago
    > neither can have more than 3 HDD per machine which is just absurd Not sure where you get that information from. Some of Hetzner's “auction” machines have 10 drives, many of OVH's offerings support more than three (even some of the budget range, Kimsufi, are 4x2T).
    - ddaalluu2 5y ago
      I'm not talking about auctions. I'm talking about regular servers. And I get those facts from ACTUALLY FUCKING TRYING TO CONFIGURE MORR THAN 3 HDDS on those machines. You go ahead and try it before talking out your ass and downvoting.
gcoguiec 5y ago
As a former OVH employee, I was constantly reminded to avoid "surqualité" and try to understand OVH's "bricolocracie" better. I never could. I hope this shock will incite the company to improve the quality of its infrastructure and products. I only wish OVH the best.
- breakingcups 5y ago
  As a non-French speaker, could you explain the second term to me?
  - charliedevolve 5y ago
    Roughly, "ruled by whatever you have on hand"?
  - nacos 5y ago
    "Bricolo" is kind of a slang term for "Bricolage" which can be translated to DIY / Tinkering. -cracy as in the ancient greek word that is used in democracy/meritocracy/... In this specific context, I believe parent is hinting (in a negative way) about the internal culture of OVH that could be interpreted as not up to professional standards ?
    - kakwa_ 5y ago
      Indeed "Bricolage", in a private context (typically DIY light home renovation) doesn't bear any negative connotations. In fact, several hardware store chains use the term (Mr. Bricolage, Brico Depot, Bricorama). But using the term when describing a product or in a professional context is far more negative, often describing something which doesn't look well made or designed.
      - dmw_ng 5y ago
        Similar concept from India https://en.wikipedia.org/wiki/Jugaad https://en.wikipedia.org/wiki/Jugaad
  - 5y ago
watsocd 5y ago
I have done work on a power system for a data center in a previous life. It is not easy to shut power off in a data center. They are designed that way intentionally. Yes, it is fairly easy to shut down utility power. But then you have automatic diesel generators that will start. If you shut them down, then you have battery powered UPS units. In the building that I knew well in Canada, they had a well guarded and covered button behind the security desk that was labeled 'EPO': Emergency Power Off. This button would send a signal to all systems (utility switches, diesel generator, and UPS units) to immediately shutdown or don't even start (diesels).
- jve 5y ago
  Small correction: UPS start after utility power goes down and only THEN Diesel stats up. Because you can't start diesel generator in a fraction of a heartbeat.
cheapzahhh 5y ago
OVH has always been known in France as a super cheap provider run by amateurs. Use it to host a Minecraft server maybe. Use it for anything else and you're an idiot.
StreamBright 5y ago
And hosting on OVH is significantly cheaper than on AWS. Right?

420 points by detaro 5y ago

[flagged]

ge96 5y ago

I have been using their services for several years now. I remember one time there was a fire and they reimbursed me for the down time.

karamanolev 5y ago
Is the reimbursement just pro-rated for the service costs (i.e. 1 day downtime = 1/30th discount of the monthly bill) or something more? If it's just service costs, I find it almost inconsequential compared to the costs to the business of having downtime.
- ge96 5y ago
  I can see that to be fair my stuff is just personal websites nobody looks at. Edit: aside from my own cheap stuff, I also use AWS but I have not gotten to the scale where if AWS went down the site runs off something else.
- FDSGSG 5y ago
  This is cheap self-service hosting, the customer is expected to deal with things like backups themselves. If your business suffers from a single DC going down, it's your fault.

catwell 5y ago

Disclaimer: I am French and I had servers (well, my employer did, and I was the admin) in that DC. From the report (not the post): something that really annoyed the firemen is that not only was there no universal electrical cut-off, there were 4 different electrical backups, which they had to figure out how to cut off one by one... It's the same thing as the self-cooling design: OVH optimized for what typically matters in a DC. You want an energy-efficient design and you never want power to go down. Well... except when you do. I suppose by now them and other hosting providers have taken that issue into account and are modifying their DC designs accordingly.

deleted 5y ago
[deleted]
tuananh 5y ago
seems like they have done an exceptionly well done, high availability power design :D
Zealotux 5y ago
I remember back when OVH was smugly mocking anyone who had concerns about their WC system, to be fair the concerns were kind of ridiculous but ultimately their system _did_ fail catastrophically. Glad to have left that company years before the fire, never doing business with them ever again.
- rosndo 5y ago
  > Glad to have left that company years before the fire, never doing business with them ever again. What alternatives are there? Not many good ones at a comparable price point. (Yeah I know about online.net, Hetzner and so on. OVH offers a much more polished product and vastly better network) At the prices OVH offers, you can have your servers replicated in multiple datacenters for less money than many DCs would charge you for a single server with the same specs. At this price point it should be perfectly fine if a datacenter burns down occasionally. At least their network is otherwise very reliable.
  - xmodem 5y ago
    Hetzner
    - rosndo 5y ago
      But… Hetzner offers an even lower quality service at an even lower price point. (But only to low-volume buyers, OVH offers vastly better volume discounts) It would be strange to expect them to deal with the situation any differently if there was a catastrophic event in one of their DCs resulting in destroyed servers. When working with dedicated servers at this price point, you’re very much expected to deal with your own backups.

bilekas 5y ago

I remember this fire, it was definitely one of the biggest disruptions for us at the time. > it took three hours to cut off the power supply because there was no universal cut-off. This seems really egregious given the nature of data centers, I would argue less so than the wooden ceilings, those were treated to survive an hour long fire, one would thing if the building was grounded with no electricity the fire could have been handled much faster.

detaro 5y ago

(Title truncated from original "OVHcloud fire report: SBG2 data center had wooden ceilings, no extinguisher, and no power cut-out" to fit HN length limit)

londons_explore 5y ago

In a datacenter fire, the risk to human life is very small (has anyone ever died in a datacenter fire, apart from being suffocated by a halon system?). It's just property risk, data loss and service downtime. Therefore, it's a business decision. I have worked in a business that, during a fire, prioritized maintaining service uptime over putting the fire out. The end result: They had to buy more new servers, but customer workloads were migrated away within 15 minutes and saw no outage. For them, it was the right decision.

jeremyjh 5y ago
Except the fire damaged other buildings and spread lead all the way to Germany.
- 0xbeefbeefbeef 5y ago
  The datacenter is about 100ft from the German border.
- jeroenhd 5y ago
  "All the way to Germany" doesn't say much if the building that's on fire is practically on the German border.
michaelt 5y ago
Well, most fires involve risk to human life if firefighters have to enter the building to extinguish it.
- mannykannot 5y ago
  Indeed, and they also often enter buildings to look for people who are in harm's way. OHVcloud's omissions did not amount to a good business decision, it was an irresponsible one.
- Nextgrid 5y ago
  Did this one involve said risk? Maybe risk to life was eliminated by a combination of low occupancy, trained staff and adequate escape routes so that there's no reason for firefighters to enter it, and the building was essentially designed to be disposable should the worst happen? I don't see this as a particularly bad thing - if risk to life and neighbouring properties was correctly managed (and it seems like it was here), why not allow this?

staticelf 5y ago

At the time of the fire I used nodechef that hosted their services on OVH and seemingly all their backups as well (in the same datacenter). Turns out when extraordinary events happen promises of backups and such aren't always kept. We lost some data because of that, luckily we had our own backups. A good reminder to make sure you have backups and that they are working correctly. No matter what promises anyone gives you should always have your own backup strategy that's disconnected from the vendor you use. It was Nodechefs fault, not OVH obviously but perhaps it's interesting for others.

Cthulhu_ 5y ago
Make sure to read the T's and C's and availability / retention rates closely; it's a process that involves decoding the legalese and trying to associate it with RL situations. Amazon's S3 for example offers a 99.999999999% data durability guarantee, with other bits implying they can withstand a datacenter going up in flames. But there's two caveats there; data availability is lower (so if that datacenter goes up in flame your data may not be lost, but it may also not be directly accessible until they restore their backups), and if they do lose data, what is the consequences to them? It'll be a financial compensation at best.
- terom 5y ago
  Breaking the S3 durability guarantees badly enough to get press attention would be a critical reputational hit, with the consequences going far beyond just the contractual SLA compensation for the customers. It would still heal with time, though. There was a period after some nasty incidents where the general advice was to avoid EBS for anything critical... but that's over now.
  - lbriner 5y ago
    Would it be critical though? Unless there are lots of alternatives at a similar availability and price-point, most people aren't just going to jump to another provider. Clouds sound largely interchangeable but you try swapping from Azure to AWS or vice-versa: for anything more than the simplest system, this would probably take 12 months+

curiousgal 5y ago

I have been an ovh customer for over 8 years but they kept increasing their prices and lowering their offerings, I finally moved on and have zero sympathy for them.

lodddgn 5y ago
What do you mean? OVH has hugely improved their offerings in that time. They introduced world class DDoS protection for all clients. They massively improved their global connectivity, and continue to do so. They’ve drastically improved their support, hiring much much more English speaking staff than they used to have. Not to mention generally improved tooling offered to customers.
- lbriner 5y ago
  Don't be upset if people aren't going to take your word for it with an account created 46 minutes ago!
  - rosndo 5y ago
    You can confirm all of those things by googling a little.
- deleted 5y ago
  [deleted]
- pzduniak 5y ago
  I gave OVH a chance by suggesting it to a client. Prior to that point, I had been using an OVH dedicated server for personal stuff without any issues. We opted for OVH Public Cloud and their Managed Kubernetes (which was considered stable at that point). Kubernetes randomly froze up after deployment, requiring manual intervention from the staff. We had no way to restart the Control Plane on our own. It always took 3+ messages / 1h+ on the phone to reach someone capable of handling it. After months of complaining, they didn't address anything. That was late 2020. What forced us to migrate away was when we got a VM with a broken NVMe SSD that broke PostgreSQL data and OVH refused to acknowledge the problem. Phone support was still practically unreachable. After spending countless hours debugging their crappy services, we just gave up and moved to Oracle Cloud. Best decision ever - their discounts are great and support is excellent. They're clearly trying to challenge the big 3. As we were leaving OVH, we send one last email that finally reached someone who gave a crap about customers. They replied that they will "have a manager talk with us"... I don't think that ever happened. Even if they improve their services, I don't trust them anymore. It was beyond horrible.

fmajid 5y ago

Irresponsible design. It's not just the fire and the damage to businesses, but the report lists concerns about lead from the UPS batteries being spread all the way to Germany as a result of the plume from the fire, as well as in the water from the firemen. Fortunately, in this case, they measured the water and found no significant amounts, nor did the German environmental authorities, but it could easily have been as bad as the Notre-Dame fire where a huge chunk of innermost Paris was contaminated by lead from the destroyed roof.

userbinator 5y ago
Of all the things to be the worried about being released by a fire...?! No. Lead is way down on the list. I'd be far more concerned about the other carcinogens from stuff burning.
- fmajid 5y ago
  It was explicitly listed in the report, so the fire department and environmental defense authorities of both countries do not share your nonchalance.
mtmail 5y ago
> all the way to Germany For reference, the border to Germany is about 250m from the building https://www.openstreetmap.org/relation/11917092 https://www.openstreetmap.org/relation/11917092

taubek 5y ago

Aren't there some regulations that would prevent such building to get operating permit?

krisoft 5y ago
I don't understand this sentiment. It is an industrial site. Nobody got hurt. The fire didn't spread to other people's property. Why are you all so upset about this?
- cbg0 5y ago
  It could have spread, and people could have gotten hurt. If you have a useful head on your shoulders, then you should have a proactive mindset and try to prevent issues instead of sending thoughts and prayers after the fact.
  - krisoft 5y ago
    > It could have spread, and people could have gotten hurt. Could it though? It is a standalone installation on an industrial site. There are about ten meter-ish standoff between them and their neighbours. (which is also an industrial site) > If you have a useful head on your shoulders, then you should have a proactive mindset and try to prevent issues And if you have an even more useful head on your shoulders you calibrate your level of upset to the level of actual danger.
    - lbriner 5y ago
      If you think 10m is enough of a fire break, you have no experience of fire. The infra-red heat that comes off of a fire is incredibly intense and can easily spread those distances as that distance, especially if pieces of flammable debris were to bridge the gap. It woul dbe fair enough if that design was recognised and signed-off by the local inspectors, then no-one would complain but it sounds like they were just not doing the risk assessment properly.
    - magicalhippo 5y ago

ovi256 5y ago

The report points out (Image 8) design choices that contributed to a raging fire, once the fire started: * no emergency electricity cut-off device, an "economic strategy choice of the site operator" - the electrical room where the fire started was hot, 400C at the door (measured by thermal camera), with meter long electrical arcing from the door, thundering deafening sounds. Making access of utility technicians "difficult" - it took them 2 hours to cut incoming electrical service from the utility. On site UPS devices also had no cut-off, so they kept supplying. * emergency water network provided only 70m3/h at the site. A firefighting boat arrived, Europa 1, was called, supplying 14.5m3/min max flow rate. The freeflowing air cooling design, a good design choice as it saves on operating costs for cooling, contributed to nourishing the fire.

stingraycharles 5y ago
So basically, my takeaway is: * Freeflowing air in DCs is good for cooling( * It’s bad when you have a fire; * An improved DC design would allow an operator to $somehow stop the freeflowing air (although one could argue that it’s not free flowing anymore if one can control it); * I’d like to know how much money really was saved by not allowing the UPSes to be cut off. I’m very curious how the insurance companies respond, and whether they’ll demand e.g. UPS supplies to be able to cut off. Or maybe in general, the fire department should be more aware of these types of trade-offs being made, and give their approval accordingly.
- lbriner 5y ago
  It is VERY common in buildings to have ventilation dampers that are closed when a fire alarm starts exactly for this reason. I am most surprised that the design sounds like it was made a certain way deliberately even though the basics were got wrong. Does Strasbourg not have any fire regulations that require fireman accessible power cut-outs, automatic extinguishers etc? It doesn't matter just that their equipment burned but the severity of the fire was massively dangerous to the fire brigade and anything else nearby if it couldn't be extinguished properly.
  - 5y ago

GreyStache 5y ago

The gallery in the article links to this drone footage I hadn't seen before: https://www.youtube.com/watch?v=BrGsCD2nVrk https://www.youtube.com/watch?v=BrGsCD2nVrk

ksec 5y ago

And now I am wondering if Hetzner has similar problem or if their DC are better designed. In the old days the two are frequently mentioned together.

jbverschoor 5y ago

Orrr we can finally replace x86 with energy efficient arm chips!

boringg 5y ago

Unrelated tidbit of knowledge i learned yesterday. The first fire brigade was created by Julius Caesars partner/general Crassus. He put together a team of about 500 people to put out fires that happened on a near daily basis in Rome. The catch: he was a land speculator who would flip burnt homes on the cheap so his brigade would run to a fire but until the owner sold to Crassus on the cheap they would watch the home burn. Talk about a hard nose business or completely unethical leverage, wow.

coopierez 5y ago
Hopefully 2000 years in the future people will talk about the US healthcare system with similar incredulity.
- jjeaff 5y ago
  Doesn't seem equivalent, since at least treatment for immediate medical emergencies is given regardless of ability to pay.
belter 5y ago
Then Nero decided to make a Game Day...

mwcampbell 5y ago

I wonder if the same corner-cutting happens in the OVHcloud US data centers.

annoyingnoob 5y ago

I deal with a ton of spam/phishing/malware that comes from OVH datacenters - OVH does nothing with complaints. Sometimes you really do reap what you sow.

nervoustwit 5y ago

"This meant that some OVH customers found that their servers continued running after the fire started." So it's not ALL bad news.

nervoustwit 5y ago

"This meant that some OVH customers found that their servers continued running after the fire started." It's not ALL bad news.

3boll 5y ago

For anyone interested, I maintained a server within the affected DC. OVH provided 3x the price of the service for the downtime. But for the recovery we needed to buy a new server from them as our backups were only accesible from dedicated machines... At the end we basically received 2x the price of the server when discounting the temp machine. Communication during the downtime was not bad from OVH side, taking into account the huge amount of affected servers. At the end, as a small customer I cant do or ask for much more. As it's not worth neither the money or the time. IMHO, 3x is not covering any business loses for anyone. We got our own backups within the OVH network and it took 3 additional days to be able to access them as the network was a mess after the fire. That for some business is going to be a huge sum.

3boll 5y ago
Since then I have changed my backup strategy... Now with diff providers. Ready for the next fire;)
- toast0 5y ago
  Just a heads up that different providers doesn't always mean different buildings. Different cities is more important for a physical disaster than different companies. For operational disasters (BGP/DNS/accounting error/abuse detection false positive/other customer DDoS, etc) different company is useful too, of course.
hinkley 5y ago
This is the illusion of SLAs. These money-back-guarantees only refund the cost of the service. They aren't insurance for loss of business. If I'm using your service for $5k a month and making less than $5k a month because of it, my finance people might rightfully ask where my head is at. 2x is better than 1x, but in general I think we are looking for higher rates of return for that. This hardware has to pay for my development and really my entire payroll after all. I also can't trust that you losing $5k an hour motivates you to fix the problem ASAP as me losing $5k an hour, let alone if I'm losing more than that.
rosndo 5y ago

moralestapia 5y ago

>no general electrical cut-off switch This is so weird I can hardly believe it, maybe some details were lost in the writing. Were they connecting everything directly to the grid? Even the most basic electrical setup goes through a fusebox with switches that turn everything on/off. Perhaps that box was burning as well, or the fire blocked access to it, idk. If there truly was no way to cut power from the site then, wow, that was just an abysmally stupid decision.

darkwater 5y ago
I think they refer to the absence of one single mains switch to turn off everything in the datacenter. It took 3 hours for the firefighter to find all the individual switches and turn them off, according to the report.
- bombcar 5y ago
  There still should be a utility box somewhere near, if necessary the utility should have been able to kill power to the whole sub-division. Every commercial building in the US I've seen has a big huge switch outside (temptation to throw it has been moderate sometimes). The point about UPS continuing to provider power even after the utility is cut is critical, though.
  - jwandborg 5y ago
    Datacenters sometimes have redundant main power providers, I'm guessing that could change the assumption to "there should be a number of supply boxes nearby, which all need to be interfered with".
    - bombcar 5y ago
      Usually then the boxes are heavily marked with how many there are, where the others are, and other info.
malfist 5y ago
It said there were meter long electrical arcs in the "power room". If I had to guess, I'd guess there was a shutoff in the power room, but not one accessible outside of it.

encoderer 5y ago

Who else is not surprised that discount hosting providers cut corners?

gunapologist99 5y ago

It's a great thing for startups that OVH provides servers at such an amazing low cost. Yes, there is always a risk that someone will make a mistake in the building design. There's a chance that eliminating some redundancies increases the possibility of a failure. There's always a chance that something bad will go wrong. However, this isn't just a matter of Hanlon's razor (incompetence vs malice), but more of a matter of an intelligent guesstimate of risk versus a lack of knowledge in some areas (wasn't this OVH's very first datacenter?), and a strong focus on reducing costs. Perhaps the latter went too far, and definitely some obvious mistakes were made by not having a universal power cut off of some sort, but dealing with the amount of power on tap in a datacenter is always dangerous, even when there is no fire at all. I'm not saying we need to give OVH a complete pass on this. I'm just saying that there are a lot of extenuating circumstances and, except for the power cut off, it's not clear that OVH made any choices due to extreme negligence or cost-cutting. In other words, they didn't do anything immoral. At worst, it appears that (even from the most anti-OVH party here), this was just a mistake in the design of a new (at the time) style of datacenter, and it did work properly for many years before there was a problem. Making a mistake is not immoral.

ddaalluu2 5y ago
"Amazing low cost"? They're one of the most expensive data center operators I know. Hetzner is about half the price. But neither can have more than 3 HDD per machine which is just absurd.
- dspillett 5y ago
  > neither can have more than 3 HDD per machine which is just absurd Not sure where you get that information from. Some of Hetzner's “auction” machines have 10 drives, many of OVH's offerings support more than three (even some of the budget range, Kimsufi, are 4x2T).
  - ddaalluu2 5y ago
    I'm not talking about auctions. I'm talking about regular servers. And I get those facts from ACTUALLY FUCKING TRYING TO CONFIGURE MORR THAN 3 HDDS on those machines. You go ahead and try it before talking out your ass and downvoting.

gcoguiec 5y ago

As a former OVH employee, I was constantly reminded to avoid "surqualité" and try to understand OVH's "bricolocracie" better. I never could. I hope this shock will incite the company to improve the quality of its infrastructure and products. I only wish OVH the best.

breakingcups 5y ago
As a non-French speaker, could you explain the second term to me?
- charliedevolve 5y ago
  Roughly, "ruled by whatever you have on hand"?
- nacos 5y ago
  "Bricolo" is kind of a slang term for "Bricolage" which can be translated to DIY / Tinkering. -cracy as in the ancient greek word that is used in democracy/meritocracy/... In this specific context, I believe parent is hinting (in a negative way) about the internal culture of OVH that could be interpreted as not up to professional standards ?
  - kakwa_ 5y ago
    Indeed "Bricolage", in a private context (typically DIY light home renovation) doesn't bear any negative connotations. In fact, several hardware store chains use the term (Mr. Bricolage, Brico Depot, Bricorama). But using the term when describing a product or in a professional context is far more negative, often describing something which doesn't look well made or designed.
    - dmw_ng 5y ago
      Similar concept from India https://en.wikipedia.org/wiki/Jugaad https://en.wikipedia.org/wiki/Jugaad
- 5y ago

watsocd 5y ago

I have done work on a power system for a data center in a previous life. It is not easy to shut power off in a data center. They are designed that way intentionally. Yes, it is fairly easy to shut down utility power. But then you have automatic diesel generators that will start. If you shut them down, then you have battery powered UPS units. In the building that I knew well in Canada, they had a well guarded and covered button behind the security desk that was labeled 'EPO': Emergency Power Off. This button would send a signal to all systems (utility switches, diesel generator, and UPS units) to immediately shutdown or don't even start (diesels).

jve 5y ago
Small correction: UPS start after utility power goes down and only THEN Diesel stats up. Because you can't start diesel generator in a fraction of a heartbeat.

cheapzahhh 5y ago

OVH has always been known in France as a super cheap provider run by amateurs. Use it to host a Minecraft server maybe. Use it for anything else and you're an idiot.

StreamBright 5y ago

And hosting on OVH is significantly cheaper than on AWS. Right?