64 ms·

Fire declared in OVH SBG2 datacentre building

• rctay89 6y ago
  Puzzles and puzzle storm is down on Lichess: > Due to a fire at one of our data centres, a few of our servers are down and may be down permanently. We are restoring these servers from backups and will enable puzzles and storm as soon as possible. > > We hope that everyone who is dealing with the fire is safe, including the firefighters and everyone at OVH. <3.
  • gnulinux 6y ago
    I think playing "from position" is also broken? I was playing chess with my friend and we usually play "from position" but it wasn't working just now so we're playing standard instead. It might be an unrelated bug.
  • ericra 6y ago
    I opened two tabs to relax tonight: Hacker News and Lichess. This was the top HN thread, and Lichess is having issues because of the fire. I didn't know what OVH was before 10 minutes ago, but this seems really impactful. I hope everyone there is safe and that the immediate disaster gets resolved quickly.
    • sofixa 6y ago
      Look them up, they're one of the biggest hosting providers in the world ( especially in France), and due to cheap prices are especially pop-up with smaller scale stuff.
      • Macha 6y ago
        Yeah, they scale down to their kimsufi line which used to have quite powerful dedicated servers for the price of basic VPSes from other providers. e.g. They have a 4core, 16gb ram server for $22/mo which is 25% of what my preferred provider, Linode, charges. Now, it comes with older consumer hardware (that one is a sandy bridge i5), and about as much support as the price tag suggests, as well as a dated management interface, but when I used to run a modded minecraft server as a college student, which needed excessive amounts of RAM and could be easily upset by load spikes on other clients, then it was a no-brainer, even if I would expect the modern-ish Xeons Linode uses to win on a core for core basis.
        • ev1 6y ago
          Dated? They're probably the only place that isn't $comedy "bare metal cloud" pricing that not only has an API for their $5/m servers but also the panel is a reasonably modern SPA that implements that API and uses OAuth for login
          • Macha 6y ago
            Has it been replaced since I last used it in ~2016? This is not the interface I had to use at all. This is the interface they had when I used them last: https://www.youtube.com/watch?v=h5-J_DO_FS0 https://www.youtube.com/watch?v=h5-J_DO_FS0
            • ev1 6y ago
              Yeah this is not at all what you use now. It's an Angular SPA. It is still a mess of separate accounts, but you can use email addr to log in instead of random-numbers generated handle. The OVHCloud US is completely separated for legal purposes, from what I remember. No account sharing, different staff, OVH EU cannot help you at all with US accounts. https://www.youtube.com/watch?v=I2G6TkKg0gQ https://www.youtube.com/watch?v=I2G6TkKg0gQ
            • littlestymaar 6y ago
              Yes it changed. It was replaced by a more modern version a few years ago (but the transition was painful, as not everything was implemented in the new version when they started to deploy it).
              • ev1 6y ago
                for me, at times you'd not bother trying to remember your NIC-handle and just curl the API instead out of laziness
• jbeales 6y ago
  > If your production is in Strasbourg, we recommend to activate your Disaster Recovery Plan. Ouf.
• drpgq 6y ago
  Like something out of Mr Robot
• nerdbaggy 6y ago
  Dang, I have a lot of respect for Octave and what he has created. https://twitter.com/olesovhcom?s=21 https://twitter.com/olesovhcom?s=21
  • de6u99er 6y ago
    After the total loss of one data-center I would tend to disagree with this statement.
• lgleason 6y ago
  https://www.youtube.com/watch?v=1EBfxjSFAxQ https://www.youtube.com/watch?v=1EBfxjSFAxQ
  • mhh__ 6y ago
    I'm not sure if it's because my tolerance of Graham Linehan has snapped or not, but I barely laugh at the IT Crowd any more. As with other GL shows I find it's just mostly held together but the cast's delivery and such The laugh track and the writing is honestly dated even by the standards of Dads Army.
    • muglug 6y ago
      The IT Crowd's comedy became dated incredibly quickly, just like Father Ted's. Comedies that came later ditched the laugh track. They had to work harder to get viewers at home to laugh, but ultimately a bunch of them (starting with The UK Office) hold up much better as a result.
    • jrockway 6y ago
      I don't remember the details, but I think that season 2 kind of retroactively ruined season 1. They used to have all those O'Reilly and EFF stickers, and working at a help desk at the time, it felt very authentic. Then everything got super nice in season 2 -- leather couches, people were dressing nicely, etc. It kind of lost its charm. You can't rewatch it because you know Denholm is just going to randomly jump out of a window. (Having said that, I think "Fire" was a memorable episode that is still amusing. The 0118911881999119 song, "it's off, so I'll turn it on... AND JUST WALK AWAY".) It might have been ahead of its time. Silicon Valley was well received and is as nerdy and intricately detailed as Season 1 of the IT Crowd. "Normal people" thought it was far out and zany. People that work in tech have been to all those meetings. And, a major character was named PG!
      • kuschku 6y ago
        I only just realized the Paul Graham/Peter Gabriel easter egg...
• bigiain 6y ago
  There is no cloud, there is just other people’s computers. And they’re on fire.
  • cookiengineer 6y ago
    Ah, that explains why there are so many dark patterns in the cloud! /s
  • Fordec 6y ago
    Your data was in the cloud, now it's in the smoke.
    • herpderperator 6y ago
      ...which has now become an actual cloud.
    • Rapzid 6y ago
      You might even say, it's gone with the wind... Yeeeeeahhhhhhhhhh!!!!!
  • quickthrower2 6y ago
    This is a literal kubectl delete pod
    • cutemonster 6y ago
      kubectl delete dc
• deleted 6y ago
  [deleted]
  • deleted 6y ago
    [deleted]
• geocrasher 6y ago
  The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2, and I also have a dedicated server (also OVH/Kimsufi) that gets the backups. I can redeploy in less than a day on fresh hardware, and that's good enough for my purposes. What's YOUR Disaster Recovery Plan?
  • tialaramex 6y ago
    If you are a corporate entity of some kind, the final layer of your plan should always be "Go bankrupt". You can't successfully recover from every possible disaster and you shouldn't try to. In the event of a sufficiently unlikely event, your business fails and every penny spent attempting the impossible will be wasted, move on and let professional administrators salvage what they can for your creditors. Lots of people plan for specific elements they can imagine and forget other equally or even more important things they are going to need in a disaster. Check out how many organisations that doubtless have 24/7 IT support in case a web server goes down somehow had no plan for what happens if it's unsafe for their 500 call centre employees to sit in tiny cubicles answering phones all day even though pandemic respiratory viruses are so famously likely that Gates listed them consistently as the #1 threat.
    • jfrunyon 6y ago
      IMHO, the part they had no plan for was being unable to just require their employees to come in anyway...
      • osmano807 6y ago
        Just lobby the government to put call centers in "essential services". In my state they are open even with a partial lockdown.
      • 6y ago
• worldofmatthew 6y ago
  https://twitter.com/olesovhcom/status/1369504527544705025 https://twitter.com/olesovhcom/status/1369504527544705025 "Update 5:20pm. Everybody is safe. Fire destroyed SBG2. A part of SBG1 is destroyed. Firefighters are protecting SBG3. no impact SBG4."
  • jen20 6y ago
    An easy mistake (in the quoted tweet) to make given the circumstances, but it probably should be 5:20am rather than pm.
  • akamaka 6y ago
    For some context: “SBG1, the first Strasbourg data center, consisting of twelve containers, came online in 2012. The 12 containers had a capacity of 12,000 servers. SBG2 is a non-container data center in 2016 using its “Tower” design with a capacity of 30,000 servers. SBG3 tower was built in 2017 with a capacity of 30,000 servers. SBG4 was built in 2013 as several containers to augment capacity, but was decommissioned in 2018 and moved to SB3” https://baxtel.com/data-center/ovh-strasbourg-campus https://baxtel.com/data-center/ovh-strasbourg-campus
    • Scoundreller 6y ago
      and a 2017 (!) article where they were planning to remove 1 and 2 because of power issues! > OVH to Disassemble Container Data Centers after Epic Outage in Europe > “This is probably the worst-case scenario that could have happened to us.” > OVH [...] is planning to shut down and disassemble two of the three data centers on its campus in Strasbourg, France, following a power outage that brought down the entire campus Friday, causing prolonged disruption to customer applications that lasted throughout the day and well into the evening. https://www.datacenterknowledge.com/uptime/ovh-disassemble-container-data-centers-after-epic-outage-europe https://www.datacenterknowledge.com/uptime/ovh-disassemble-c...
      • Macha 6y ago
        It sounds like the plan was to shutdown 1 and 4, the latter of which happened and the former which did not.
• lovedswain 6y ago
  He posted an update, seems SBG2 is totally destroyed. Ouch https://twitter.com/Onepamopa/status/1369484420982407173 https://twitter.com/Onepamopa/status/1369484420982407173
  • meepmorp 6y ago
    > DID ANYTHING SURVIVE? ANYTHING AT ALL?????? OUR DATA IS IN SBG2. WHAT DO WE DO NOW ?!?! Double check those backups, folks.
    • monkeybutton 6y ago
      Trust but verify. As a developer it doesn't matter what sysadmins or anyone else says about backups of your data; if you haven't run your DR plan and verified the results, it doesn't exist.
      • mhh__ 6y ago
        > Trust but verify. If it's business critical should you even trust at all?
        • casi 6y ago
          yeah backups are a case of "don't trust, verify"
    • mhh__ 6y ago
      One of the things I'm still extremely grateful for is that I learnt the basics of computer science from an ex-oracle guy turned secondary school teacher, who wasn't the best programmer let's say but who absolutely drilled into us (I was probably the only one listening but still) the importance of code quality, backups, information security etc. Nothing fancy, but it's the kind of bread and butter intuition you need to avoid walking straight off a cliff. He also let me sit at the back writing a compiler instead of learning VB.Net, top dude
    • ikiris 6y ago
• pm90 6y ago
  Unfortunately A lot of people are going to find out the hard way today why AWS/GCP/Big Expensive Cloud is so expensive (Hint: they have redundancy and failover procedures which drive up costs). Keep in mind I’m talking not of “downtime” but of actual data loss which might affect business continuity. This is really tragic. I’m hoping they have some kind of multi regional backup/replication and not just multi zones (although from the twitters it appears that only one of the zones was destroyed however the others don’t seem to be operational atm).
  • hashhar 6y ago
    AWS and GCP are also prone to same kind of data loss if the AZ you are operating in goes down. They don't automatically geo-replicate things. You still need a backup for the torched EC2 instance to be able to relaunch in another AZ/region.
    • nnx 6y ago
      That's true, but it seems whole of SBG region for OVH is within same disaster radius for one fire... with SBG2 destroyed and SBG1 partly damaged. "The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4. " Wonder if those SBGx were advertised as being the same as "Availability Zones" - when other cloud providers ensure zones are distanced enough from each other (~1km at least) to likely survive events such as fire.
      • hashhar 6y ago
        Thats a fair point. If OVH does market them as AZs then it's disingenuous and liable to suits IMO.
        • cbg0 6y ago
          No, it isn't, as there's no clear cut definition of what an availability zone is.
      • tyingq 6y ago
• deleted 6y ago
  [deleted]
• redisman 6y ago
  Wow. That equipment is going to be very hard to replace right now too.
• nerdbaggy 6y ago
  Here is some pics of what it looked like before, spg 1-4 , and it’s history https://baxtel.com/data-center/ovh-strasbourg-campus https://baxtel.com/data-center/ovh-strasbourg-campus
  • gameshot911 6y ago
    Is it just my laptop or is only ~40% of vertical screen real-estate dedicated to the actual content? :<
    • softblush 6y ago
      It's just you
    • aetherspawn 6y ago
      I have the same issue on my XPS 13 (4K screen), the header takes up a good 30% or so of the height of the screen and it's like reading through a mailbox slit.
    • AnssiH 6y ago
      Nope, exactly the same here (on landscape phone).
  • eb0la 6y ago
    I'm surprised to see how close the DCs are to the river. Fortunately it's in the high part of the river, less prone to overflow.
    • RantyDave 6y ago
      Maybe quite handy for water cooling?
  • bmurray7jhu 6y ago
    SBG3 is almost adjacent to SBG2. It is impressive that the firefighters saved SGB3 with minimal damage.
• MattGaiser 6y ago
  > We recommend to activate your Disaster Recovery Plan. What percentage of organizations have these?
  • dylan604 6y ago
    Short answer: not enough.
  • ikiris 6y ago
    Some plans have a single step: 1) panic.
  • RantyDave 6y ago
    I guess we're about to find out (or, rather, they are).
• yalogin 6y ago
  I don’t know what ovh is and going to the site point me to a speed test with no information.
  • MattGaiser 6y ago
    French AWS/GCP is my understanding.
    • tyingq 6y ago
      They are really more like Hetzner. They have "cloud", but most of the business is dedicated servers. They also operate kimsufi.com and soyoustart.com. They do have APAC, Canadian, and US data centers as well.
  • beckler 6y ago
    They’re a web/hosting service provider, like AWS or GCP, but with less services. They’re much more popular in European countries.
  • kuschku 6y ago
    Imagine AWS, with less features, but 10x-100x lower prices. And now you know why until a few years ago they were larger in traffic, customers, and number of servers than even AWS.
  • tecleandor 6y ago
    You went to ovh.net instead of ovh.com. Largest hosting provider in Europe, probably top10 or top5 in the world.
• Shank 6y ago
  A status update on the OVH tracker for a different datacenter (LIM-1 / Limburg) says "We are going to intervene in the rack to replace a large number of power supply cables that could have an insulation defect." [0][1] The same type of issue is "planned" in BHS [3] and GRA [2]. Eerie timing: do they possibly suspect some bad cables? [0]: http://travaux.ovh.net/?do=details&id=49016 http://travaux.ovh.net/?do=details&id=49016 [1]: http://travaux.ovh.net/?do=details&id=49017 http://travaux.ovh.net/?do=details&id=49017 [3]: http://travaux.ovh.net/?do=details&id=49462 http://travaux.ovh.net/?do=details&id=49462 [2]: http://travaux.ovh.net/?do=details&id=49465 http://travaux.ovh.net/?do=details&id=49465
  • dylan604 6y ago
    >Eerie timing: do they possibly suspect some bad cables? Why not? Cables with ratings lower than the load they are carrying is a prime cause for electrical fires. If the load is too high for long enough, the shielding melts away, and if it is close enough for other material to catch fire then that's the ball game. It's a common cause for home electrical fires. Some lamp with poor wiring catches the drapes on fire, etc. Wouldn't think a data center would have flammable curtains though.
    • varispeed 6y ago
      This definitely could be probable cause. When I was a teen I witnessed such fire once. Basically a friend had a heater but couldn't find any mains cable and eventually decided to disconnect mains cable from the radio and use that. After few minutes the isolation from the mains cable melted away and the cable turned glowing red and it started burning the table it was on. Fortunately we were not asleep and got it under control quickly. Lesson learned.
  • jfrunyon 6y ago
    They're waiting an awful long time to do the one at BHS-7 if so: 14 days from now?
  • terom 6y ago
    https://www.google.com/search?q=site%3Ahttp%3A%2F%2Ftravaux.ovh.net+%22+Replacement+of+the+C13%2FC14+power+supply+cables%22 https://www.google.com/search?q=site%3Ahttp%3A%2F%2Ftravaux.... there's quite a few of these http://travaux.ovh.net/?do=details&id=47840 http://travaux.ovh.net/?do=details&id=47840 earliest one that I found was back in December
• neo34 6y ago
  "Everyone is safe. Fire has destroyed SBG2. A part of SBG1 is destroyed. The firefighters are protecting SBG3. No impact on SBG4". Tweet from Octave Klaba, founder of OVHcloud. "All our clients on this site are possibly impacted" Tout le monde est sain et sauf. Le feu a détruit SBG2. Une partie de SBG1 est détruite. Les pompiers protègent actuellement SBG3. Pas d’impact sur SBG4 », a tweeté Octave Klaba, le fondateur d’OVHcloud, en désignant les différentes parties du site. « Tous nos clients sur ce centre de données sont susceptibles d’être impactés », a précisé l’entreprise sur Twitter.
• walrus01 6y ago
  Reminder to not only have backups, but also have some periodic OFFLINE backups. If your primary is set up with credentials to automatically transfer a copy to the backup destination over the network, what happens if your primary gets pwned and the access is used to encrypt or delete the backup? Secondly, test doing restores of your backups, and have methods/procedures in place for exactly what a restore looks like.
  • deleted 6y ago
    [deleted]
  • qeternity 6y ago
    > what happens if your primary gets pwned and the access is used to encrypt or delete the backup? Append-only permissions. We do this in S3 for that specific reason. S3 lifecycle rules take care of pruning old backups. You can also build a pull-based system where auth resides on the backup system, not the production system.
• rsync 6y ago
  This is a literal nightmare for me. I can remember several San Diego fires that threatened the original JohnCompanies datacenter[1] circa mid 2000s and thinking about all of the assets and invested time and care that went into every rack in the facility. Very interested to read the post-mortem here ... even more interested in any actionable takeaways from what is a very rare event ... [1] Castle Access, as it was known, at the intersection of Aero Dr. and the 15 ... was later bought by Redit, then Kio ...
  • deleted 6y ago
    [deleted]
  • sebmellen 6y ago
    Man, 2006 was a freaky time to be in San Diego.
• ogaj 6y ago
  Wow. Weren't they just angling for an IPO? I wonder how much of this was insured and what the impact is to their overall operations.
  • moooo99 6y ago
    I think they announced their IP plans yesterday [1], which is probably the worst timing one can have (if there even is a good timing for a datacenter burning down, probably there isn't). If they have a good insurance I'm confident this will have little impact on their operations, I really hope they do. I host a few components on OVH/SoYouStart dedicated servers, luckily not mission critical, but still had rather good experience with them, especially in terms of price to performance. [1] https://www.reuters.com/article/amp/idUSKBN2B01FM https://www.reuters.com/article/amp/idUSKBN2B01FM
    • toyg 6y ago
      The publicity damage alone will be on par with (if not bigger) their replacement costs. I wouldn’t be surprised if they had to rebrand. The timing is so bad that it becomes almost suspicious. When is the best time to sabotage your competitor? When they are the most visible.
      • karambahh 6y ago
        SBG is a small part of their operations. I doubt it will have a lasting issue on their image
      • brmgb 6y ago
        > The publicity damage alone will be on par with (if not bigger) their replacement costs. I wouldn’t be surprised if they had to rebrand. Honest question: which publicity damage? A fire in a datacenter is very much part of the things you should expect to see happen when you operate a large number of datacenters and will obviously cause some disruption to your customers hosting physical servers there. Provided the disruption doesn't significantly extend to their cloud customers and doesn't affect people paying for guaranteed availability (which it shouldn't - OVH operates datacenters throughout the world), this seems to me to be an unfortunate incident but not a business threatening one.
• ohnonotagain9 6y ago
  (site a)---[replicate local LUNs/shares to remote storage arrays]--->(site b), (site a)---[replicates local VMs to remote HCI]--->(site b), (site a)---[local backups to local data archive]--->(site a), (site a)---[local data archive replicates to remote data archive]--->(site b), (site b)---[remote data archive replicates to remote air gapped data archive]--->(site b), (site a)---[replicates to cold storage on aws/gcp/azure]--->(site c), (site c)---[replicate to another geo site on cloud]--->(site d) scenario 1: site a is down plan: recover to site b by most convinent means scenario 2: site b is down plan: restore services, operate without redundancy out of site a scenario 3: site c is down plan: restore services, catch up later. continue operating out of site a scenario 4: site b and c down plan: restore services, operate without redundancy out of site a scenario 5: site a and b down plan: cross fingers, restore to new site from cold storage on expensive cloud VM instances scenario 6: data archive corrupted ransomware plan: restore from air gapped data archive, hope ransomware was identified within 90 days scenario 7: site b and c down, then site a down plan: quit scenario 8: staff hates job and all quit plan: outsource scenario 9: and so on...
• NorwegianDude 6y ago
  I can't see anything about a fire suppression system mentioned? Doesn't OVH have one, except for colocation datacenters? A fire detection system using eg. lasers and Inergen(or Argonite) for putting the fire out is commonly used in datacenters. The gas fills the room and reduces the amount of oxygen in the room so most fires are put out within a minute. The cool thing is that the gas is designed to be used in rooms with people, so that is can be triggered any time. It is however quite loud, and some setups have been known to be too loud, even destroying harddrives.
  • MattGaiser 6y ago
    Are fires common in data centres? Specialized fire suppression tech seems to indicate that they are.
    • jhugo 6y ago
      The decision to design and implement specialised tech comes from a combination of how likely the risk is and the magnitude of the potential loss. Fires are not that common in DCs, but the potential loss can be enormous (as OVH is currently finding out).
    • jabroni_salad 6y ago
      They aren't super common, but halon and other gas systems are just the right tool for the job. It can get inside the server chassis and doesn't damage equipment like a chemical application would. We won't know what went wrong at OVH until a proper post mortem comes out. These systems work by suppressing the flame reaction, but if the actual source was not addressed, it could reignite after awhile.
    • perlgeek 6y ago
      Yes, mostly due to two reasons: * overall high energy density (lots of current flowing everywhere) * the batteries for backup power are dangerous, and can easily(ish) overheat when activated.
  • jhugo 6y ago
    At a datacentre I used to visit years ago, part of the site induction was learning that if the fire suppression alarm went off, you had a certain amount of time to get out of the room before the argon would deploy, so you should always have a path to the nearest exit in mind. The implication was that it wasn't safe to be in the room once it deployed, but I don't know for sure.
• qwertykb 6y ago
  Honestly just for the memes. https://isovhonfire.com https://isovhonfire.com
  • donatj 6y ago
    Did this already exist or was this thrown up insanely fast.
    • qwertykb 6y ago
      Threw it up really quick.
    • navanchauhan 6y ago
      whois[0] shows it was registered on March 10, so thrown up insanely fast. [0] https://who.is/whois/isovhonfire.com https://who.is/whois/isovhonfire.com
  • bithaze 6y ago
    Probably not a good look for this to be the first thing another hosting company thinks to put up in response.
    • qwertykb 6y ago
      /shrug/ We have our infrastructure partially in OVH, I see it as a friendly jab at them and a way to get updates without having to navigate to twitter.
    • sofixa 6y ago
      OVH aren't nice in that regard, and have trolled competitors to "leave it to the pros" before when there were serious incidents ( of which OVH have had their fair share), so it's not surprising.
• cbg0 6y ago
  This event reminded me of the fire at The Planet a while back: https://www.datacenterknowledge.com/archives/2008/06/01/explosion-at-the-planet-causes-major-outage https://www.datacenterknowledge.com/archives/2008/06/01/expl...
• plasma 6y ago
  “ Update 7:20am Fire is over. Firefighters continue to cool the buildings with the water. We don’t have the access to the site. That is why SBG1, SBG3, SBG4 won’t be restarted today.” https://mobile.twitter.com/olesovhcom/status/1369535787570724864 https://mobile.twitter.com/olesovhcom/status/136953578757072...
  • rgj 6y ago
    With water, they said.
    • optimalsolver 6y ago
      Liquid cooling.
      • rgj 6y ago
        No, as in "firehose". https://mobile.twitter.com/abonin_DNA/status/1369538028243456000 https://mobile.twitter.com/abonin_DNA/status/136953802824345...
• pmontra 6y ago
  Some pictures of the building with firefighters at work https://www.dna.fr/faits-divers-justice/2021/03/10/strasbourg-important-incendie-dans-une-entreprise-situee-sur-un-site-seveso-au-port-du-rhin https://www.dna.fr/faits-divers-justice/2021/03/10/strasbour... Edit: Video at https://www.youtube.com/watch?v=a9jL_THG58U https://www.youtube.com/watch?v=a9jL_THG58U Satellite view of the site on Google Maps https://goo.gl/maps/L2T6YNFCtiyDdiNv7 https://goo.gl/maps/L2T6YNFCtiyDdiNv7
  • ilkkao 6y ago
    Easy to see now that a lightly constructed five story cube might not be a fully fire proof.
    • Biganon 6y ago
      How do you see that?
    • mike_d 6y ago
      Would you humor us with a link to a fully fire proof datacenter?
      • verytrivial 6y ago
        From the outside (and from a position lacking all inside knowledge) it looks highly interconnected and very well ventilated. I'm not sure where you'd put a inert gas supression system or beefy firewalls to slow the fire progress.
      • dividedbyzero 6y ago
        I think Microsoft had some experimental datacenter containers they submerged in the northern Atlantic for passive cooling, and I believe those were filled with an inert gas as well. I guess that would come very close to an actual fireproof datacenter.
        • reasonabl_human 6y ago
• rgj 6y ago
  Video here. This will be Disaster Recovery 101 material. https://mobile.twitter.com/abonin_DNA/status/1369538028243456000 https://mobile.twitter.com/abonin_DNA/status/136953802824345...
  • raverbashing 6y ago
    Ouch, that's not pretty... but it seems that the fire was constrained to 1 or 2 sectors (inside the building) - per their updates Not sure how good were the fire suppression systems of the building.
    • baggy_trough 6y ago
      Looks like a lot of those containers/compartments are melted.
  • Rapzid 6y ago
    Or disaster prevention. I'm curious how their fire suppression system failed so spectacularly..
    • rgj 6y ago
      I think they failed to install it. They only mention smoke detection: https://www.ovh.com/world/us/about-us/datacenters.xml https://www.ovh.com/world/us/about-us/datacenters.xml
      • lorenzhs 6y ago
        In French but with pictures: https://lafibre.info/ovh-datacenter/ovh-et-la-protection-incendie https://lafibre.info/ovh-datacenter/ovh-et-la-protection-inc... - probably a different OVH data centre, but they clearly have sprinklers there. The argument they made against gas extinguishers is by the time they have to use it, the data is gone anyway, and it's only going to trigger in the affected areas. It's also far safer for the people working there.
        • 6y ago
• brainzap 6y ago
  my VPS is gone, hmm
• deleted 6y ago
  [deleted]
• mot2ba 6y ago
  Ovh needed a better firewall :(
• deleted 6y ago
  [deleted]
• anilakar 6y ago
  Finally a legitimate reason for BSD sysadmins to run poweroff -n!
• Tepix 6y ago
  Last update https://twitter.com/olesovhcom/status/1369535787570724864?s=21 https://twitter.com/olesovhcom/status/1369535787570724864?s=... "Update 7:20am Fire is over. Firefighters continue to cool the buildings with the water. We don’t have the access to the site. That is why SBG1, SBG3, SBG4 won’t be restarted today."
• trailmonster 6y ago
  As an industry we like to think we're transparent, honest, and perhaps even based on merit, but I can't find any of that in this messaging. "We are currently facing a major incident in our Strasbourg datacentre, with a fire declared in the SBG2 building. Firefighters intervened immediately on the spot but were unable to control the SBG2 fire. As a precautionary measure, the electricity was cut off on the whole site, which impacts all our services at SBG1, SBG2, SBG3 and SBG4. If your production is in Strasbourg, we recommend that you activate your Business Recovery Plan." Incidents are faced, not caused. It's made clear that they called the fire department as soon as they should have, and they did what they could as well, but in reality, your disaster recovery plan and how well you implemented it is what's really the question now, isn't it? I think it's far easier to ask whether you've done the right thing, user, than it is to ask why a fire managed to take out an entire facility that was designed to prevent that exact scenario, but only if you're OVH.
• 112233 6y ago
  Photos and video from the scene: https://www.dna.fr/faits-divers-justice/2021/03/10/strasbourg-important-incendie-dans-une-entreprise-situee-sur-un-site-seveso-au-port-du-rhin https://www.dna.fr/faits-divers-justice/2021/03/10/strasbour...
• deleted 6y ago
  [deleted]
• rarefied_tomato 6y ago
  Was the building made from stacked shipping containers? Containers are such a budget-friendly and trendy structural building block these days. They even click with the software engineers - "Hey, it's like Docker". Containers would seem to be at a disadvantage when it comes to dissipating, rather than containing, heat. I hope improved thermal management and fire suppression designs can be implemented.
  • qeternity 6y ago
    OVH use a custom water cooling solution they claim enables these niche designs and increases rack density. As another comment says, that density probably exacerbated the issue here.
  • AnssiH 6y ago
    I could be wrong, but I understood from other comments that only SBG1 used containers.
• crubier 6y ago
  The key differentiator of OVH is the very compact datacenters they achieve thanks to water cooling. Some OVH exec were touting about that in a recent podcast. Interestingly in this case, having a very compact data center was probably an aggravating factor. This shows how complex these technical choices are, you have to think of operating savings, with a trade off on the gravity of black swan events...
  • nnx 6y ago
    Interesting. That said, the technique is not the issue here, losing a whole datacenter can always happen. This event would have been much less serious if all the four SBG* datacenters were not all so close to each other on the same plot of land. They are so close to each other that they are basically the same physical datacenter with 4 logical partitions.
    • delfinom 6y ago
      They are all annexes built up over time and a victim of their own success (the site was meant to be more of a edge node in size). The container based annexes were meant to be dismantled 3 years ago but profit probably got in the way
• nikanj 6y ago
  According to the official status page, the whole datacenter is still green http://status.ovh.com/vms/index_sbg2.html http://status.ovh.com/vms/index_sbg2.html
  • aetherspawn 6y ago
    You're completely right. IMO this is the best comment in this whole thread. Their status page must be broken, or it's a lie.
    • bootloop 6y ago
      If there is a SLA with consequences associated with it every status page is going to be a lie.
      • aetherspawn 6y ago
        Well, it sucks to catch fire and I care for the employees and the firemen, but if their status page is a lie then I have a whole lot less sympathy for the business. That's shady business and they should feel bad. I can appreciate an honest mistake though, like the status page server cron is hosted in the same cluster that caught fire and hence it burnt down and can't update the page anymore.
        • Ploskin 6y ago
          Is the status page relevant though? At the very least, OVH immediately made a status announcement on their support page and they've been active on Twitter. I don't see anything shady here. From their support page: > The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4. If your production is in Strasbourg, we recommend to activate your Disaster Recovery Plan What more could you want?
          • PudgePacket 6y ago
            > Is the status page relevant though? What's the point of a status page then if it does not show you the status? I don't want to be chasing down twitter handles and support pages during an outage.
• aw4y 6y ago
  elliot..is it you?
• k_sze 6y ago
  I sent this story to my colleagues and one of them asked "where is the FM200?" I don't really know how FM200 systems work in data centres, but I'm guessing that if the fire didn't start from within the actual server room, FM200 might not save you? e.g. if a fire started elsewhere and went out of control, it would be able to burn through the walls/ceiling/floor of the server room, in which case no amount of FM200 gas can save you, right? Another possibility, of course, is that the FM200 system simply failed to trigger even though the fire started from within the server room. There is no published investigation details about this incident yet, I believe. Can somebody chime in about past incidents where FM200 failed to save the day?
  • etiennemarcel 6y ago
    I think most of these gases are or will eventually be banned in Europe because of their impact on the environment. I've seen newer datacenters use water mist sprays.
    • mike_d 6y ago
      Nitrogen makes up 78% of the atmosphere, so I doubt it will be banned. Most datacenters don't actually use halocarbons despite the common "FM200" name.
    • exikyut 6y ago
      How... what. What if the fire is electrical? You can't just go "well the triple interlocked electrical isolation will trip and cut the current" if a random fully-charged UPS decides to get angry...
    • divingdragon 6y ago
      You might be thinking of Halons, which are CFCs that depletes the ozone layer? They are mostly phased out worldwide but existing installations might still be in use. FM200 is something else that is often used in modern builds (not just datacenters).
      • etiennemarcel 6y ago
• awalias 6y ago
  not to be a conspiracist, but are they still hosting wikileaks data? https://en.wikipedia.org/wiki/OVH#WikiLeaks https://en.wikipedia.org/wiki/OVH#WikiLeaks
• deleted 6y ago
  [deleted]
• simplecto 6y ago
  This is horrible and a sobering reminder to do the things we don't enjoy or consider -- disaster recovery. How many of us here plan Fire Drills within our teams and larger organizations?
• emersion 6y ago
  Picture of the fire from last night: https://twitter.com/BobZeHareng/status/1369563084277374980 https://twitter.com/BobZeHareng/status/1369563084277374980
• phtrivier 6y ago
  Seems like data.gouv.fr [1], the government platform for open data is impacted ; we might not get the nice COVID-19 graphs from non-governemental sites ([2], [3]) today. I can't wait for the conspiracy theories about how the fire is a "cover up" to "hide" bad COVID-19 numbers... [1] https://twitter.com/GuillaumeRozier/status/1369572490599690240 https://twitter.com/GuillaumeRozier/status/13695724905996902... [2] https://covidtracker.fr/ https://covidtracker.fr/ [3] https://www.meteo-covid.com/trouillocarte https://www.meteo-covid.com/trouillocarte (Just wanted to share the "trouillocarte" - which roughly translates to "'how badly is shit hitting the fan today' map" ;) )
  • Shadonototro 6y ago
    it has nothing to do with covid: - https://www.usine-digitale.fr/article/le-health-data-hub-heberge-par-microsoft-se-penche-sur-les-effets-de-l-invalidation-du-privacy-shield.N1056354 https://www.usine-digitale.fr/article/le-health-data-hub-heb... - https://www.genethique.org/la-cnam-refuse-le-transfert-global-de-donnees-vers-le-health-data-hub/ https://www.genethique.org/la-cnam-refuse-le-transfert-globa... - https://france3-regions.francetvinfo.fr/bretagne/donnees-medicales-decouvrez-si-vous-etes-concernes-par-la-fuite-de-donnees-des-laboratoires-bretons-1972174.html https://france3-regions.francetvinfo.fr/bretagne/donnees-med... Here is more plausible conspiracy theory: USA/microsoft is behind the cyberattacks and the fire
• tester34 6y ago
  holy shit my 3$ VPS!! nvm not this dc!
• zoobab 6y ago
  To my experience, backups in companies are barely done. Companies want quick money, they push people to skip important IT operations, like disaster recovery plans. And backups are the least monitored systems.
• Psype 6y ago
  Been there, done that. The 2nd worst thing, IF you happen to catch it soon and control it, is that the temp raise triggers many alerts and automatic controls. So when it's controlled, it's still a real nightmare. Here, firemen could not even control it...
• jjjeii3 6y ago
  White collar jobs are not paid well in the EU. If a saleswoman from a nearby shop has the same salary, why should I care about the quality of my work at all? The entire data center burned down? Fine! There are a lot of other places with the same tiny salary. As you can see below, a developer expert is only making 24K-63K Euro per year at OVH (in US dollar it's almost the same amount): https://www.glassdoor.com/Salary/OVHcloud-France-Salaries-EI_IE523060.0,8_IL.9,15_IN86_IP3.htm https://www.glassdoor.com/Salary/OVHcloud-France-Salaries-EI... after paying taxes you will only get a half of that amount.
• benlumen 6y ago
  Someone told me that there were a lot of warez sites hosted with OVH. Anyone know of any casualties?
• odiroot 6y ago
  Interesting! I got the news from my local package courier website. They warn, their services can be unreliable due to the fire at OVH. It's all connected.
• thbb21 6y ago
  My VPS in SBG3 stopped pinging around 9am. My impression is that they tried very hard to maintain uptime, which was probably a bad idea when we see the extent of the damages. This VPS just hosts external facing services and is easy to set back up.
• Aldipower 6y ago
  Looks a little bit like Fukushima. I hope the clean up doesn't take that long though..
• IceWreck 6y ago
  Always, always keep local backups folks.
  • coolspot 6y ago
    “Local” meaning on the same server, just in another folder, right?
    • IceWreck 6y ago
      Sorry, I should've phrased it better. Local as in your home/office. While your application may run in AWS or whatever remote server, its necessary to have copies of your data that you can physically touch and access. One main deployment, one remote backup and one onsite physically accessible backup.
• termau 6y ago
  Most modern data centres wouldn't have had this issue, at least in in Australia they use Argonite suppression systems, these work by using a gas that is a mixture of argon and nitrogen that suppresses fire by depleting oxygen in the the data hall.
  • exikyut 6y ago
    I'm seeing quite a lot of repeated sentiment throughout the comments that Halon is illegal and is no longer used. Is the situation "Halon is legal in Australia" or "Halon isn't actually illegal per se if you don't use a lot of it"?
    • aaronmdjones 6y ago
      Halon and Argonite are unrelated.
    • divingdragon 6y ago
      Halons are being phased out because, being CFCs, they deplete the ozone layer. Other gases like argon or FM200 are not Halons.
  • LIV2 6y ago
    I don't think Aussie dc's are all the same. Globalswitch in Sydney uses inergen but Equinix is using water at least at SY1 but I'm reasonably sure that GS is much older
• dbrgn 6y ago
  Wow, SBG3 seems to be OK: «Update 11:20am: All servers in SBG3 are okey. They are off, but not impacted. We create a plan how to restart them and connect to the network. no ETA. Now, we will verify SBG1.» https://twitter.com/olesovhcom/status/1369592437585412097 https://twitter.com/olesovhcom/status/1369592437585412097
• molszanski 6y ago
  Is that a New Serverless Platform everyone is talking about recently?
• londons_explore 6y ago
  Did OVH claim that SBG1 and SBG2 were isolated failure domains? Despite them just being different rooms in the same building?
  • kuschku 6y ago
    They're separate buildings, with separate power systems, just standing next to another. Next to SBG2 are also SBG3 and SBG4.
    • londons_explore 6y ago
      Those buildings weren't quite as far apart as they should have been if a fire in one requires all 4 of the others to be turned off...
      • Leo_Verto 6y ago
        Not disagreeing with you but the firefighters probably shut down all power onsite as soon as they arrived.
      • jiofih 6y ago
        Apparently hey are more like (temporary) extensions of the main building than separate DCs.
      • jiofih 6y ago
        Apparently they are more like (temporary) extensions of the main building than separate DCs.
  • ev1 6y ago
    They are different physical buildings. OVH does not generally claim anything regarding AZ or distance. SBG1, 2, 3, etc is just denoting the building your server is in - they are not like AWS style AZ or similar, quite literally just building addresses. I have used them for years and I don't believe they've ever said anything like deploy in both SBG1 and SBG2 for safety or availability, because you don't get that choice. When you provision a machine (eg via API) they tell you "SBG 5 min, LON 5 min, BHS 72h" and you pick SBG and get assigned first-available. There is no "I want to be in SBG4" generally.
• mwcampbell 6y ago
  I just recently started moving some services for my business to one of OVH's US-based data centers. Should I take this fire as evidence that OVH is incompetent and get out? I really don't want AWS, or the big three hyperscalers in general, to be the only option.
  • lenartowski 6y ago
    IMO you should take this fire as evidence, that you need to have (working!) backups wherever you host your data. AWS, GCP Azure are not fire resistant, same as OVH. I don't know if OVH is more or less competent than big three, I choose to trust no one.
    • ghosty141 6y ago
      I read multiple times that they didn't even have sprinklers, only smoke detectors in their EU datacenter(s). I'm 100% sure AWS, Azure and Google have better fire prevention.
      • Symbiote 6y ago
        This thread has people saying they have sprinklers, don't have sprinklers, have / don't have gas suppression, and have puppies / actually have toilets. Wait for the misinformation hose to dry up, and decide in a few weeks. https://us.ovhcloud.com/about/company/security https://us.ovhcloud.com/about/company/security
• manishsharan 6y ago
  And this is why the big 3 will continue to dominate. AWS, Microsoft and Google can throw in a lot more money at their phyiscal infrastructure than any other cloud provider. After this sorry episode, I dont think any CTO or CIO of any public company will be able to even consider using the other guys. edit: I am not implying that we put all eggs in one basket with no failover and dr. I am implying the big cos will pay 2x premium on infrastructure to project reliability.
  • fooyc 6y ago
    I could replicate my whole infrastructure on 3 different OVH datacenters, with enough provision to support twice the peak load - it would still be cheaper than a single infrastructure at AWS, and I would get a better uptime than AWS: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=%22aws%20is%20down%22&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
    • manishsharan 6y ago
      I agree with you 100%. However, I did state * any CTO or CIO of any public company* ... the executives don't worry about costs , they worry about being able to * project reliability * .
      • fooyc 6y ago
        Executives that worry about reliability would insist on deploying on multiple data centers, which would make the project more reliable than any single AWS availability zone. Also, cost matters if the AWS bill is one of the company's top expenses.
    • joshuamorton 6y ago
      reserved a1.large instances are about half the price of OVH's b2-7 instance. a1.xlarge are still cheaper (and larger). So you get more raw compute per dollar on AWS. What?
      • 6y ago
• broknbottle 6y ago
  It’s a Thermal Event not a fire
• canadianfella 6y ago
  I hereby declare this to be a fire.
• KirillPanov 6y ago
  The cloud is on fire.
• betamaxthetape 6y ago
  This is certainly a good reminder to have regular backups. I have (had?) a VPS in SBG1, the shipping-container-based data centre in the Strasbourg site, and the latest I know is that out of the 12 containers, 8 are OK but 4 are destroyed [1]. Regardless, I imagine it will be weeks / probably months before even the OK servers can be brought back online. Naturally, I didn't do regular backups. Most of the data I have from a 2019 backup, but there's a metadata database that I don't have a copy of, and will need to reconstruct from scratch. Thankfully for my case reconstructing will be possible - I know that's not the case for everyone. Right now I'm feeling pretty stupid, but I only have myself to blame. For their part, OVH have been really good at keeping everyone updated (particularly the Founder and CEO, Octave Klaba). I believe that when I signed up back in 2017, the Strasbourg location was advertised as one of the cheapest, so I can imagine a lot of people with a ~$4 / month OVH VPS are in the same situation, desperately scrambling to find a backup. (For those that have a OVH VPS that's down right now, you can find what location it is in by logging onto the OVH control panel.) [1] https://twitter.com/olesovhcom/status/1369598998441558016 https://twitter.com/olesovhcom/status/1369598998441558016
• r4ibOm 6y ago
  I see several people talking about advanced backup systems for businesses. I do not have a company, I work as a freelancer, I am Brazilian and the current 70 euros of tuition I was paying was already compromising my income, since my local currency is quite devalued against the dollar. Then imagine the situation. my websites are down, the only backup I have was a copy of the vps that i made in november last year because i was with the intention of creating a server at my house, since it was getting expensive to maintain this server at OVH. It would be unacceptable for a company of this size not to have its servers backed up or to keep them in the same location as the incident, since their networks are all connected. I hope you have a satisfactory and quick solution to this problem.
  • MegaThorx 6y ago
    I had my server in SBG2 and sadly the backup failed since the end of january. Yep it is my mistake for not checking the backups. Now I lost about 1 month of data. The only good thing is that my backup was offsite. Does OVH offer automatic snapshots for VPS? I know Hetzner does it for 20% addidation of the cost of the server. If they do the next question would be if they are destroyed too?
    • kuschku 6y ago
      They do offer automatic backups, and they're offsite (RBX in this case). Here's more information on them: https://docs.ovh.com/gb/en/vps/using-automated-backups-on-a-vps/ https://docs.ovh.com/gb/en/vps/using-automated-backups-on-a-... With OVH the price can depending on the situation double your cost, if you've got a 3€ VPS and a 3€ backup configuration (as the price depends on size)
  • Symbiote 6y ago
    If OVH backed up everything, the cost of the service would be double. Many customers don't need a backup, so it's up to each customer to arrange their own backups — perhaps with tools and services provided by the hosting company, or their own solution. Running a company with no backup (for cost or any other reason) is very risky, as some people will have found out today.
  • 6y ago
• Pick-A-Hill2019 6y ago
  For the curious - this is what it looks like when a fire suppression system activates in a (small) server room. https://youtu.be/DrDU4UQUwKg?t=60 https://youtu.be/DrDU4UQUwKg?t=60
• martinald 6y ago
  Interesting. I wonder if the cladding was a major problem here? It looks like it has all burnt out and could have had the fire spread extremely rapidly on the outside.
  • jiofih 6y ago
    The cladding was metal so very unlikely it contributed to the fire spreading.
    • martinald 6y ago
      ACM was the reason the grenfell disaster happened in the UK.
      • jiofih 6y ago
        “ACM” is a plastic sheet with aluminum skins. It would be plain idiotic to cover your DC in that (and residences too, but I digress). You can see from pictures of the OVH fire aftermath that the outer plates have melted holes in them characteristic of metal.
• lovedswain 6y ago
  > In the case of Roubaix 4, the Datacenter is made with a lot of wood: > Finally, we have other photos of the floor of the OVH "Roubaix 4" tower. It is clearly wood! Hope it's fireproof wood! A wooden datacenter ... is still original, we must admit. > In France, data centers are mainly regulated by the labor code, by ICPE recommendations (with authorization or declaration) and by insurers. At the purely regulatory level, the only things that are required are: > - Mechanical or natural smoke extraction for blind premises or those covering more than 300m2 > - The fire compartmentalization beyond a certain volume / m2 > - Emergency exits accessible with a certain width > - Ventilation giving a minimum of fresh air per occupant > - Access to the firefighter from the facade for the premises are the low floor of the last level is more than 8 meters > - 1 toilet for 10 people (occupying a position considered "fixed") https://lafibre.info/ovh-datacenter/ovh-et-la-protection-incendie/msg80372/?PHPSESSID=dp2nmlbd994ebb28nrv608ebr7#msg80372 https://lafibre.info/ovh-datacenter/ovh-et-la-protection-inc...
  • malobre 6y ago
    "1 chiotte" -> "1 toilet" not puppy
    • 4ggr0 6y ago
      A puppy would be called a "chiot", if someone wonders.
  • MayeulC 6y ago
    Ah, interesting pics and discussions (in French) in the neighboring thread: https://lafibre.info/datacenter/incendie-sur-un-site-ovh-a-strasbourg/ https://lafibre.info/datacenter/incendie-sur-un-site-ovh-a-s...
• bdz 6y ago
  Rust (the video game) lost all EU server data w/o restore https://twitter.com/playrust/status/1369611688539009025 https://twitter.com/playrust/status/1369611688539009025
  • rplnt 6y ago
    Now that is weird. I fully understand not having a backup service in place, but no data backup either?
    • Voloskaya 6y ago
      TBF "data" here means the state of the gameworld which anyway resets entirely every 2 weeks or every month, so it's not exactly a big deal, everyone constantly start over in Rust.
      • rplnt 6y ago
        Ah, I see. That makes sense then. I thought it's an everlasting game world, thanks for clarification.
  • chrisandchris 6y ago
    Someone forgot the 3-2-1 rule for backups. I don‘t get why people don‘t do offsite backup today as it‘s basically for free. AWS Glacier basically costs nothing at all.
• ricardobeat 6y ago
  > At 00:47 on Wednesday, March 10, 2021, a fire broke out in a room in one of our 4 datacenters in Strasbourg, SBG2. Please note that the site is not classified as a Seveso site. > Firefighters immediately intervened to protect our teams and prevent the spread of the fire. At 2:54 am they isolated the site and closed off its perimeter. > By 4:09 am, the fire had destroyed SBG2 and continued to present risks to the nearby datacenters until the fire brigade brought the fire under control. > From 5:30 am, the site has been unavailable to our teams for obvious security reasons, under the direction of the prefecture. The fire is now contained.
• jeffrallen 6y ago
  Hugops to OVH folks, hang in there, you're my favorite European data center operator.
• curiousgal 6y ago
  Came across this on r/France https://i.imgur.com/epj1Lue.png https://i.imgur.com/epj1Lue.png Translation : We lost our Gitlab and backups... And the automatic backups that had been put in place no longer worked so a priori we lost everything...
• archit3cture 6y ago
  in case there were not posted before, here are pictures of SBG2 in flames taken by the firemen. https://twitter.com/xgarreau/status/1369559995491172354 https://twitter.com/xgarreau/status/1369559995491172354 This puts an image on the sentence "SBG2 is destroyed". Do not expect any recovery from SBG2.
  • benlumen 6y ago
    Holy hell. Are these "datacenters" really just shipping containers? That's what it looks like.
    • tantalor 6y ago
      https://en.wikipedia.org/wiki/Modular_data_center https://en.wikipedia.org/wiki/Modular_data_center
      • quickthrower2 6y ago
        Insert docker joke...
    • switch007 6y ago
      Aren't they a bargain-basement provider? You get what you pay for I guess?
  • lucb1e 6y ago
    Status shows greens across the board http://status.ovh.com/vms/index_sbg2.html http://status.ovh.com/vms/index_sbg2.html Am I looking at the wrong thing or am I right to wonder why we still bother with public status pages if it never shows the real status? Edit: nvm just saw another comment pointing out the same further down the thread (I randomly came across this page while looking for the physical location of another DC)
• refraincomment 6y ago
  I will never gonna financially recover from this..
• SilasX 6y ago
  Late to the party, but ... context? Everyone is talking like OVH or the SBG2 buildings are well-known and common knowledge.
  • Symbiote 6y ago
    https://letmegooglethat.com/?q=OVH https://letmegooglethat.com/?q=OVH
• TrueDuality 6y ago
  There have been a handful of talks at computer security conferences talking about setting up physical traps in server chassis (such as this one: https://www.youtube.com/watch?v=XrzIjxO8MOs https://www.youtube.com/watch?v=XrzIjxO8MOs). Since seeing those I've been waiting for some idiot to try something like that in a physical server and burn down a data center. There is NO evidence that is what happened here, and I don't think OVH allows customers to bring their own equipment making even less likely. Still I wait and hope to hear a root cause from this one.
• MrQuimico 6y ago
  I have a VPS in SBG1. So far I haven't got any communication from OVH alerting me about this. I think that's the first thing they should do, alerting their customers that something it's happening. Anyway I was running a service that I was about to close, so this may be it. I do have a recovery plan, but I don't know it it is worth it at this point. I'm never using OVH again. The fire can happen, but don't ask me about my recovery plan, what about yours?
  • Symbiote 6y ago
    There's a big link from their homepage: https://www.ovh.ie/news/press/cpl1786.fire-our-strasbourg-site https://www.ovh.ie/news/press/cpl1786.fire-our-strasbourg-si... It says they've alerted customers, but I expect some have been missed, through inaccurate email records, email hosted on the systems that have been destroyed etc.
• Diederich 6y ago
  Back in the late 90s, I implemented the first systematic monitoring of WalMart Store's global network, including all of the store routers, hubs (not switches yet!) and 900mhz access points. Did you know that WalMart had some stores in Indonesia? They did until 1998. So when https://en.wikipedia.org/wiki/May_1998_riots_of_Indonesia https://en.wikipedia.org/wiki/May_1998_riots_of_Indonesia started happening, we heard some harrowing stories of US employees being abducted, among other things. Around that same time, the equipment in the Jakarta store started sending high temperature alerts prior to going offline. Our NOC wasn't able to reach anyone in the store. The alerts were quite accurate: that was one of the many buildings that had been burned down in the riots. I guess it's slightly surprising that electrical power to the equipment in question lasted long enough to allow temperature alerting. Most of our stores back then used satellite for their permanent network connection, so it's possible telcom died prior to the fire reaching the UPC office. In a couple of prominent places in the home office, there were large cutouts of all of the countries WalMart was in at the time up on the walls. A couple of weeks after this event, the Indonesia one was taken down over the weekend and the others re-arranged.
  • 1MachineElf 6y ago
    Thanks for sharing this interesting story. Part of my family immigrated from Indonesia due to those riots, but I was unaware up until today of the details covered by the Wikipedia article you linked. I remember during the 2000s and 2010s that WalMart in the USA earned a reputation for it's inventories primarily consisting of Chinese-made goods. I'm not sure if that reputation goes all the way back to 1998, but it makes me wonder if WalMart was especially targeted by the anti-Chinese element of the Indonesian riots because it.
    • Diederich 6y ago
      What the hell, here's another story. The summary to catch your attention: in the early 2000s, I first became aware of WalMart's full scale switch to product sourcing from China by noting some very unusual automated network to site mappings. Part of what my team (Network Management) did was write code and tools to automate all of the various things that needed to be done with networking gear. A big piece of that was automatically discovering the network. Prior to our auto discovery work, there was no good data source for or inventory of the routers, hubs, switches, cache engines, access points, load balancers, VOIP controllers...you name it. On the surface, it seems scandalous that we didn't know what was on our own network, but in reality, short of comprehensive and accurate auto discovery, there was no way to keep track of everything, for a number of reasons. First was the staggering scope: when I left the team, there were 180,000 network devices handling the traffic for tens of millions of end nodes across nearly 5,000 stores, hundreds of distribution centers and hundreds of home office sites/buildings in well over a dozen countries. The main US Home Office in Bentonville, Arkansas was responsible for managing all of this gear, even as many of the international home offices were responsible for buying and scheduling the installation of the same gear. At any given time, there were a dozen store network equipment rollouts ongoing, where a 'rollout' is having people visit some large percentage of stores intending to make some kind of physical change: installing new hardware, removing old equipment, adding cards to existing gear, etc. If store 1234 in Lexington, Kentucky (I remember because it was my favorite unofficial 'test' store :) was to get some new switches installed, we would probably not know what day or time the tech to do the work was going to arrive. ANYway...all that adds up to thousands of people coming in and messing with our physical network, at all hours of the day and night, all over the world, constantly. Robust and automated discovery of the network was a must, and my team implemented that. The raw network discovery tool was called Drake, named after this guy: https://en.wikipedia.org/wiki/Francis_Drake https://en.wikipedia.org/wiki/Francis_Drake and the tool that used many automatic and manual rules and heuristics to map the discovered networking devices to logical sites (ie, Store 1234, US) was called Atlas, named after this guy: https://en.wikipedia.org/wiki/Atlas_(mythology) https://en.wikipedia.org/wiki/Atlas_(mythology) All of that background aside, the interesting story. In the late 90s and early 2000s, Drake and Atlas were doing their thing, generally quite well and with only a fairly small amount of care and feeding required. I was snooping around and noticed that a particular site of type International Home Office had grown enormously over the course of a few years. When I looked, it had hundreds of network devices and tens of thousands of nodes. This was around 2001 or 2002, and at that time, I knew that only US Home Office sites should have that many devices, and thought it likely that Atlas had a 'leak'. That is, as Atlas did its recursive site mapping work, sometimes the recursion would expand much further than it should, and incorrectly map things. After looking at the data, it all seemed fine. So I made some inquiries, and lo and behold, that particular international home office site had indeed been growing explosively. The site's mapped name was completely unfamiliar to me, at the time at least. You might have heard of it: https://en.wikipedia.org/wiki/Shenzhen https://en.wikipedia.org/wiki/Shenzhen I was seeing fingerprints in our network of WalMart's whole scale switch to sourcing from China.
• pwned1 6y ago
  To be honest, this gives me a little schadenfreude. OVH is the most notorious host that refuses to act on abuse complaints for phishing sites.
• rjsw 6y ago
  The photos look like the building had external cladding, wonder if that contributed to the size of the blaze [1]. [1] https://en.wikipedia.org/wiki/Grenfell_Tower_fire https://en.wikipedia.org/wiki/Grenfell_Tower_fire
• giis 6y ago
  I don't know whether my 10+ year side project with 225,000 users(www.webminal.org) gone forever! :( I have backup snapshots but its stored in ovh itself :( hoping for a miracle!
  • XCSme 6y ago
    What was the reason of storing the backup on the same server? To allow for rollbacks in case of data corruption or some changes gone wrong?
    • Operyl 6y ago
      I think they’re talking about the backup services provided by OVH, in which I believe they’re stored in RBX.
    • giis 6y ago
      Yes, I thought rollbacks will be much easier in case of data loss. :sob:
  • quickthrower2 6y ago
    Really sorry to hear, hope you get it restored. I cannot judge - I use the default backup options in Azure and hope they store it in another data centre but never though to check too hard. This is very bad luck. Hopefully you had the code in GitHub but that still leaves the DB. It looks like your has something to do with command line or Linux lessons so not sure how much user data is critical? Maybe you can get this up and running again to some extent.
  • kuschku 6y ago
    If your backup snapshots are stored through OVH's normal backup functionality, then create a new server at e.g. RBX now, and restore from those backups. That'll take a few hours and it'll all be up again quickly.
• Kye 6y ago
  The best time to test your backups is before the production server dies in a fire.
• kzrdude 6y ago
  Lichess was affected by this fire: https://twitter.com/lichess/status/1369543554255757314 https://twitter.com/lichess/status/1369543554255757314 But they seem to be back up
• deleted 6y ago
  [deleted]
• Gengis 6y ago
  Hi, Does anyone recommend a mail provider that implements IMAP failover and replication across different sites ? My mail account is hosted by OVH, it has been down for hours and from what I read I may have to wait for another 1 or 2 days. Thanks !
• SamLicious 6y ago
  This is such a an eye opening story, didn't know about any of this. Thank you for sharing!
• helge9210 6y ago
  Almost 11 years ago (March 27 2010) in Ukraine datacenter of company Hosting.ua went in flames as clients were watching their systems go unresponsive at various rows of the datacenter. Anti-fire systems didn't kick-in. The reason? For couple of days the system was detecting a little bit of smoke from one of the devices. Operators weren't able to pinpoint exact location, considered it a false alarm and manually switched anti-fire system off.
  • jaza 6y ago
    Sounds disturbingly similar to the story of how Chernobyl went into meltdown. Also in Ukraine!
• airhead969 6y ago
  Don't they build datacenters with FM200 anymore? Even the antique Stanford datacenter I did some work in had bottles of it all over.