3 ms·

One of OVH's datacenter in Strasbourg, France is literally on fire

27 points by Signez 6y ago

Knight22 6y ago
http://travaux.ovh.net/?do=details&id=49484 http://travaux.ovh.net/?do=details&id=49484 "We are currently facing a major incident in our DataCenter of Strasbourg with a fire declared in the building SBG2. Firefighters were immediately on the scene but could not control the fire in SBG2. The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4. If your production is in Strasbourg, we recommend to activate your Disaster Recovery Plan. All our teams are fully mobilized along with the firefighters. We will keep you updated as more information becomes available."
crazyjncsu 6y ago
We had a bunch of servers get taken out by this. We were able to failover with backups from the last 12 hours. The incident sounded suspicious from the start, but the wording “could not control the fire in SBG2” sounds very ominous ... Chernobyl-y even. And seeing the compactness of the entire facility (https://baxtel.com/data-center/ovh-strasbourg-campus/photos https://baxtel.com/data-center/ovh-strasbourg-campus/photos), it seems an uncontrolled fire could take out all three buildings in short order. I’m very interested in seeing the damage and postmortem from this one. Also interesting they tout all sorts of sophistication (https://www.ovh.com/world/us/about-us/datacenters.xml https://www.ovh.com/world/us/about-us/datacenters.xml), but no mention of fire suppression
- cpncrunch 6y ago
  Fire extinction is mentioned here: https://us.ovhcloud.com/about/company/security https://us.ovhcloud.com/about/company/security Out of curiosity, which backups are these? Are they OVH backups? I just noticed that our OVH backup server is in the same region as our server (BHS).
  - crazyjncsu 6y ago
    S3 backups. It’s part of this pseudo-cloud orchestrator we wrote for moving around instances of our single-tenant app.
    - cpncrunch 6y ago
      Thx. S3 seems to be the best way to go.
AlbinoPeacock 6y ago
Latest update: https://twitter.com/olesovhcom/status/1369504527544705025 https://twitter.com/olesovhcom/status/1369504527544705025
SlothXC 6y ago
Another incident/maintenance ticket on their tracker mentions mass replacing PSU cabling in another datacenter (Limburg) for potential insulation defects, starting an hour ago. Make of that what you will... http://travaux.ovh.net/?do=details&id=49017& http://travaux.ovh.net/?do=details&id=49017&

27 points by Signez 6y ago

http://travaux.ovh.net/?do=details&id=49484 http://travaux.ovh.net/?do=details&id=49484 "We are currently facing a major incident in our DataCenter of Strasbourg with a fire declared in the building SBG2. Firefighters were immediately on the scene but could not control the fire in SBG2. The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4. If your production is in Strasbourg, we recommend to activate your Disaster Recovery Plan. All our teams are fully mobilized along with the firefighters. We will keep you updated as more information becomes available."

crazyjncsu 6y ago

We had a bunch of servers get taken out by this. We were able to failover with backups from the last 12 hours. The incident sounded suspicious from the start, but the wording “could not control the fire in SBG2” sounds very ominous ... Chernobyl-y even. And seeing the compactness of the entire facility (https://baxtel.com/data-center/ovh-strasbourg-campus/photos https://baxtel.com/data-center/ovh-strasbourg-campus/photos), it seems an uncontrolled fire could take out all three buildings in short order. I’m very interested in seeing the damage and postmortem from this one. Also interesting they tout all sorts of sophistication (https://www.ovh.com/world/us/about-us/datacenters.xml https://www.ovh.com/world/us/about-us/datacenters.xml), but no mention of fire suppression

cpncrunch 6y ago
Fire extinction is mentioned here: https://us.ovhcloud.com/about/company/security https://us.ovhcloud.com/about/company/security Out of curiosity, which backups are these? Are they OVH backups? I just noticed that our OVH backup server is in the same region as our server (BHS).
- crazyjncsu 6y ago
  S3 backups. It’s part of this pseudo-cloud orchestrator we wrote for moving around instances of our single-tenant app.
  - cpncrunch 6y ago
    Thx. S3 seems to be the best way to go.

AlbinoPeacock 6y ago

Latest update: https://twitter.com/olesovhcom/status/1369504527544705025 https://twitter.com/olesovhcom/status/1369504527544705025

SlothXC 6y ago

Another incident/maintenance ticket on their tracker mentions mass replacing PSU cabling in another datacenter (Limburg) for potential insulation defects, starting an hour ago. Make of that what you will... http://travaux.ovh.net/?do=details&id=49017& http://travaux.ovh.net/?do=details&id=49017&