8 ms·

Disclaimer: I am French and I had servers (well, my employer did, and I was the admin) in that DC. From the report (not the post): something that really annoye

Disclaimer: I am French and I had servers (well, my employer did, and I was the admin) in that DC. From the report (not the post): something that really annoyed the firemen is that not only was there no universal electrical cut-off, there were 4 different electrical backups, which they had to figure out how to cut off one by one... It's the same thing as the self-cooling design: OVH optimized for what typically matters in a DC. You want an energy-efficient design and you never want power to go down. Well... except when you do. I suppose by now them and other hosting providers have taken that issue into account and are modifying their DC designs accordingly.

deleted 5y ago
[deleted]
tuananh 5y ago
seems like they have done an exceptionly well done, high availability power design :D
Zealotux 5y ago
I remember back when OVH was smugly mocking anyone who had concerns about their WC system, to be fair the concerns were kind of ridiculous but ultimately their system _did_ fail catastrophically. Glad to have left that company years before the fire, never doing business with them ever again.
- rosndo 5y ago
  > Glad to have left that company years before the fire, never doing business with them ever again. What alternatives are there? Not many good ones at a comparable price point. (Yeah I know about online.net, Hetzner and so on. OVH offers a much more polished product and vastly better network) At the prices OVH offers, you can have your servers replicated in multiple datacenters for less money than many DCs would charge you for a single server with the same specs. At this price point it should be perfectly fine if a datacenter burns down occasionally. At least their network is otherwise very reliable.
  - xmodem 5y ago
    Hetzner
    - rosndo 5y ago
      But… Hetzner offers an even lower quality service at an even lower price point. (But only to low-volume buyers, OVH offers vastly better volume discounts) It would be strange to expect them to deal with the situation any differently if there was a catastrophic event in one of their DCs resulting in destroyed servers. When working with dedicated servers at this price point, you’re very much expected to deal with your own backups.
  - hdjjhhvvhga 5y ago
    I would argue Hetzner is much better than OVH price-wise and I've been using a multiple-datacenter scheme with them for several years now for several of my clients. And they're very happy because the cost is so reasonable.
    - rosndo 5y ago
      Public rates offered by Hetzner are definitely slightly cheaper than OVH, but OVH offers vastly better volume discounts to even fairly small businesses (talking like 10-20k/mo spend). IME OVH also offers much better connectivity around the world (which makes sense, given that they operate at a much larger scale than Hetzner) Oh, and Hetzner billing support is absolutely terrible. I had to fight with them for months to stop charging for servers which had already been cancelled, after tens of emails they eventually owned up to having a bug on their end. It was like talking to a wall.
      - tomschwiha 5y ago
        I had the opposite experience - OVH support not responding for several days for fairly simple cases. They even forced me once to send a fax to them.. I also had several outages with OVH (Exchange and Dedicated Server) - none yet with Hetzner. Right now planning to finally leave OVH for Hetzner and Microsoft (Exchange)
belter 5y ago
Used them a few years ago with no complaints, but sounds like they need to then be a bit more specific on their compliance page: https://www.ovhcloud.com/fr/enterprise/certification-conformity/ https://www.ovhcloud.com/fr/enterprise/certification-conform... Edit: To clarify, they are claiming generic compliance with for example ISO/IEC 27001, 27017 et 27018. Does not look like it from the incident report. Maybe only some of their offers, and that is the detail I am referring to.
- quicksilver03 5y ago
  Their most recent ISO/IEC 27001 certificate is available at https://www.lne.fr/recherche-certificats/data_certificats/37383-1.pdf https://www.lne.fr/recherche-certificats/data_certificats/37... The SOA date and the effective date of the certificate are posterior to the fire incident in question.
  - belter 5y ago
    Fire was at OVH SBG2. OVH SBG1, SBG2 and SBG3 were all at: Rue du Bassin de l'Industrie, 67000 Strasbourg, France. The addendum to that document says is only related to activities on the sites mentioned. The address Rue du Bassin de l'Industrie, 67000 Strasbourg is listed, so sounds like they have now addressed the issues.
pid-1 5y ago
Since we are finding that post facto, I really doubt it. It means their customers did not care about compliance.
- boringg 5y ago
  I think there is an assumption of compliance on behalf of the customer.
  - sofixa 5y ago
    Yes, but OVH's reputation is low cost and cutting corners. They have a lower price than the competition, and it doesn't take a lot to figure out why.
richardfey 5y ago
> I suppose by now them and other hosting providers have taken that issue into account and are modifying their DC designs accordingly. Love your faith in humanity but I think we'd both be surprised about how little other hosting providers have changed after this incident.
- KennyBlanken 5y ago
  This setup is so far off standard practice it's shocking. Freeflowing does not excuse not having proper auto-closing dampers, triggered by the fire alarms or suppression system. Automatic power disconnects based on fire suppression system triggering are pretty standard. Most of the systems are fail-safe, meaning if the fire suppression panel loses power or is disconnected from the breakers feeding the room, the breakers trip off. That there were no automatic shutoffs or easily accessible breakers to disable the generators is shocking. Most NOCs I've visited, there are remote genset controls on one of the walls, often remote transfer switch controls as well. At the very least, on the way out the door someone in the NOC should have pushed the right buttons. I've never even heard of a commercial datacenter that didn't have fire suppression. Some companies go for preaction systems to reduce the chances of a "everything gets completely fucked by water" systems (gaseous systems are the gold standard, hugely expensive to install compared to water.) They had wood ceilings!? In my career in the US, I've been around numerous situations where it was pretty obvious the fire inspector was being paid off or at the very least had an extremely cordial relationship with the building management. My guess is that something similar was happening here. I don't understand how they managed to have insurance, or get the ISO certifications they claimed they had.
  - HstryrsrBttn 5y ago
    Autoclosing is a must. What fire department didn't do the walkthrough? Here that is obligatory and when that happens you can be sure that even a carton sti left from the newest installation is reprimanded. The last room I took take of was accepted with protocol by an architect. The carpenter and electrition had to explain every single change they made and justify that fire protection was reasonable (e.g. if necessary an extra layer of plasterboard and plaster on every small bit that could ignite was mandatory to gain an est. 30min on top here and there) ISO: what apart from documentation is necessary for ISO? What Standard are you referring to?
bayindirh 5y ago
A proper datacenter can be both efficient and safe. Add solid blidners to close the chimneys, and an oxygen suppressant system (NOVEC, etc.), and add motorized switch-fuses. They can all be orchestrated by a PLC, and a fire alarm/control system. Close blinds, release NOVEC, disable power rails to computers. That's all. It might not stop everything, but it can help a lot.
- k8sToGo 5y ago
  Of course you can, but I guess at that point you don't build your DC out of shipping containers.
  - bayindirh 5y ago
    Theoretically, you can. We looked into it. Also, Chinese build solid DC buildings out of containers. It's indeed possible.
    - asciimike 5y ago
      Which providers were you looking at? IIRC Huawei was the only company providing ISO container DCs, but the were super low density (e.g. 50kW total) compared to what the original Google/Sun/HP designs were.
  - gregoriol 5y ago
    Shipping containers are better against fire than wood. But here it's what was inside that seems to have been a problem, not the outer structure.
    - brohee 5y ago
      How are they better? Metal structures last minutes not hours when enough heat is involved...
  - AnssiH 5y ago
    Just for the record so people don't get the wrong idea, SBG2 was not built out of shipping containers. OVH has other DCs that are, though.
redm 5y ago
I've spent a lot of time in US Data-centers, and I can't remember seeing one without an EOD. I've even seen them pressed a few times accidentally when people exiting the floor thought they were "exit" buttons. Are they talking about the lack of an EOD or something more fundamental to the power system? It almost seems like you need something external to the facility in case of a fire where it's unsafe to enter the building.
rurban 5y ago
As architect I don't understand how you can blame OVH here. Local fire code is within the city council, which uses the local fire department, and then with the maintainer. Without proper planning and fire code measures you won't get a permit. How did the constructor got a permit at all? This is a wooden building for F 60?? This must be F90 for starters. Then the electrical planning: How did they get a permit at all? OVH was only a renter. First I would blame local fire department for not enforcing their fire codes.
- Nextgrid 5y ago
  It could be that only risk to life, limb and neighbouring properties was evaluated and (probably correctly) concluded that the combination of low-occupation, trained staff and sufficient escape routes meant that the risk was minimal. If someone wants to burn down their own property, why not let them? That's up to the business and their insurance to judge.
- mihaaly 5y ago
  You don't rent a log house to operate a glass manufacture inside! Each construction have limits of usage, quality or quantity wise, you know that. Those put their operation into a place must ascertein that the location can withstand the operations and its risks. Even renting a flat put limits on your activity in the contract, what you can or cannot do there. If they requested the proper fire rating and safety measures then it is not their mistake being deceived. Otherwise it is.

deleted 5y ago

[deleted]

tuananh 5y ago

seems like they have done an exceptionly well done, high availability power design :D

Zealotux 5y ago

I remember back when OVH was smugly mocking anyone who had concerns about their WC system, to be fair the concerns were kind of ridiculous but ultimately their system _did_ fail catastrophically. Glad to have left that company years before the fire, never doing business with them ever again.

rosndo 5y ago
> Glad to have left that company years before the fire, never doing business with them ever again. What alternatives are there? Not many good ones at a comparable price point. (Yeah I know about online.net, Hetzner and so on. OVH offers a much more polished product and vastly better network) At the prices OVH offers, you can have your servers replicated in multiple datacenters for less money than many DCs would charge you for a single server with the same specs. At this price point it should be perfectly fine if a datacenter burns down occasionally. At least their network is otherwise very reliable.
- xmodem 5y ago
  Hetzner
  - rosndo 5y ago
    But… Hetzner offers an even lower quality service at an even lower price point. (But only to low-volume buyers, OVH offers vastly better volume discounts) It would be strange to expect them to deal with the situation any differently if there was a catastrophic event in one of their DCs resulting in destroyed servers. When working with dedicated servers at this price point, you’re very much expected to deal with your own backups.
- hdjjhhvvhga 5y ago
  I would argue Hetzner is much better than OVH price-wise and I've been using a multiple-datacenter scheme with them for several years now for several of my clients. And they're very happy because the cost is so reasonable.
  - rosndo 5y ago
    Public rates offered by Hetzner are definitely slightly cheaper than OVH, but OVH offers vastly better volume discounts to even fairly small businesses (talking like 10-20k/mo spend). IME OVH also offers much better connectivity around the world (which makes sense, given that they operate at a much larger scale than Hetzner) Oh, and Hetzner billing support is absolutely terrible. I had to fight with them for months to stop charging for servers which had already been cancelled, after tens of emails they eventually owned up to having a bug on their end. It was like talking to a wall.
    - tomschwiha 5y ago
      I had the opposite experience - OVH support not responding for several days for fairly simple cases. They even forced me once to send a fax to them.. I also had several outages with OVH (Exchange and Dedicated Server) - none yet with Hetzner. Right now planning to finally leave OVH for Hetzner and Microsoft (Exchange)

belter 5y ago

Used them a few years ago with no complaints, but sounds like they need to then be a bit more specific on their compliance page: https://www.ovhcloud.com/fr/enterprise/certification-conformity/ https://www.ovhcloud.com/fr/enterprise/certification-conform... Edit: To clarify, they are claiming generic compliance with for example ISO/IEC 27001, 27017 et 27018. Does not look like it from the incident report. Maybe only some of their offers, and that is the detail I am referring to.

quicksilver03 5y ago
Their most recent ISO/IEC 27001 certificate is available at https://www.lne.fr/recherche-certificats/data_certificats/37383-1.pdf https://www.lne.fr/recherche-certificats/data_certificats/37... The SOA date and the effective date of the certificate are posterior to the fire incident in question.
- belter 5y ago
  Fire was at OVH SBG2. OVH SBG1, SBG2 and SBG3 were all at: Rue du Bassin de l'Industrie, 67000 Strasbourg, France. The addendum to that document says is only related to activities on the sites mentioned. The address Rue du Bassin de l'Industrie, 67000 Strasbourg is listed, so sounds like they have now addressed the issues.

pid-1 5y ago

Since we are finding that post facto, I really doubt it. It means their customers did not care about compliance.

boringg 5y ago
I think there is an assumption of compliance on behalf of the customer.
- sofixa 5y ago
  Yes, but OVH's reputation is low cost and cutting corners. They have a lower price than the competition, and it doesn't take a lot to figure out why.

richardfey 5y ago

> I suppose by now them and other hosting providers have taken that issue into account and are modifying their DC designs accordingly. Love your faith in humanity but I think we'd both be surprised about how little other hosting providers have changed after this incident.

KennyBlanken 5y ago
This setup is so far off standard practice it's shocking. Freeflowing does not excuse not having proper auto-closing dampers, triggered by the fire alarms or suppression system. Automatic power disconnects based on fire suppression system triggering are pretty standard. Most of the systems are fail-safe, meaning if the fire suppression panel loses power or is disconnected from the breakers feeding the room, the breakers trip off. That there were no automatic shutoffs or easily accessible breakers to disable the generators is shocking. Most NOCs I've visited, there are remote genset controls on one of the walls, often remote transfer switch controls as well. At the very least, on the way out the door someone in the NOC should have pushed the right buttons. I've never even heard of a commercial datacenter that didn't have fire suppression. Some companies go for preaction systems to reduce the chances of a "everything gets completely fucked by water" systems (gaseous systems are the gold standard, hugely expensive to install compared to water.) They had wood ceilings!? In my career in the US, I've been around numerous situations where it was pretty obvious the fire inspector was being paid off or at the very least had an extremely cordial relationship with the building management. My guess is that something similar was happening here. I don't understand how they managed to have insurance, or get the ISO certifications they claimed they had.
- HstryrsrBttn 5y ago
  Autoclosing is a must. What fire department didn't do the walkthrough? Here that is obligatory and when that happens you can be sure that even a carton sti left from the newest installation is reprimanded. The last room I took take of was accepted with protocol by an architect. The carpenter and electrition had to explain every single change they made and justify that fire protection was reasonable (e.g. if necessary an extra layer of plasterboard and plaster on every small bit that could ignite was mandatory to gain an est. 30min on top here and there) ISO: what apart from documentation is necessary for ISO? What Standard are you referring to?

bayindirh 5y ago

A proper datacenter can be both efficient and safe. Add solid blidners to close the chimneys, and an oxygen suppressant system (NOVEC, etc.), and add motorized switch-fuses. They can all be orchestrated by a PLC, and a fire alarm/control system. Close blinds, release NOVEC, disable power rails to computers. That's all. It might not stop everything, but it can help a lot.

k8sToGo 5y ago
Of course you can, but I guess at that point you don't build your DC out of shipping containers.
- bayindirh 5y ago
  Theoretically, you can. We looked into it. Also, Chinese build solid DC buildings out of containers. It's indeed possible.
  - asciimike 5y ago
    Which providers were you looking at? IIRC Huawei was the only company providing ISO container DCs, but the were super low density (e.g. 50kW total) compared to what the original Google/Sun/HP designs were.
- gregoriol 5y ago
  Shipping containers are better against fire than wood. But here it's what was inside that seems to have been a problem, not the outer structure.
  - brohee 5y ago
    How are they better? Metal structures last minutes not hours when enough heat is involved...
- AnssiH 5y ago
  Just for the record so people don't get the wrong idea, SBG2 was not built out of shipping containers. OVH has other DCs that are, though.

redm 5y ago

I've spent a lot of time in US Data-centers, and I can't remember seeing one without an EOD. I've even seen them pressed a few times accidentally when people exiting the floor thought they were "exit" buttons. Are they talking about the lack of an EOD or something more fundamental to the power system? It almost seems like you need something external to the facility in case of a fire where it's unsafe to enter the building.

rurban 5y ago

As architect I don't understand how you can blame OVH here. Local fire code is within the city council, which uses the local fire department, and then with the maintainer. Without proper planning and fire code measures you won't get a permit. How did the constructor got a permit at all? This is a wooden building for F 60?? This must be F90 for starters. Then the electrical planning: How did they get a permit at all? OVH was only a renter. First I would blame local fire department for not enforcing their fire codes.

Nextgrid 5y ago
It could be that only risk to life, limb and neighbouring properties was evaluated and (probably correctly) concluded that the combination of low-occupation, trained staff and sufficient escape routes meant that the risk was minimal. If someone wants to burn down their own property, why not let them? That's up to the business and their insurance to judge.
mihaaly 5y ago
You don't rent a log house to operate a glass manufacture inside! Each construction have limits of usage, quality or quantity wise, you know that. Those put their operation into a place must ascertein that the location can withstand the operations and its risks. Even renting a flat put limits on your activity in the contract, what you can or cannot do there. If they requested the proper fire rating and safety measures then it is not their mistake being deceived. Otherwise it is.