7 ms·

OVH CEO Octave Klaba apologizes and provides updates about SBG2

148 points by netgusto 6y ago

progval 6y ago
duplicate of https://news.ycombinator.com/item?id=26427960 https://news.ycombinator.com/item?id=26427960
deleted 6y ago
[deleted]
imbusy111 6y ago
The worst thing, in my opinion, is that there has been no communication from OVH directly to their customers. I am impacted and I have not received a single e-mail about the situation even now.
- ognarb 6y ago
  We received a mail. Fortunately we were using the managed database service from ovh and could quickly restore our websites from a backup.
- puxedo 6y ago
  Same in my case. No communication. Generally they are good with any networking issues, attacks, etc. In this case nothing apart from the banner on the dashboard.
- cpncrunch 6y ago
  Octave has been putting out daily messages on twitter to customers explaining the situation.
  - duxup 6y ago
    Man I hate how that works. I'll want to know what is going on with a thing and find out "oh they mostly post that kind of news on twitter" or "oh this person uses instagram", or "that news goes out via email", or youtube ... or ... who knows where ... Like guises come on, where am I supposed to get information when you're using like 8 different channels and not consistently...
    - ARandomerDude 6y ago
      This raises the question, "How could Octave be so tone-deaf?"
      - 1123581321 6y ago
        Only for people who want this incident to be part of a grand story arc about the character of OVH. The communication has been fine.
        ARandomerDude 6y ago
        It was a bad pun.
        1123581321 6y ago
        Ah! I’m very sorry. (Doubly sorry? I know being misunderstood hertz?)
    - fshr 6y ago
      Should they be sending emails? Yes, of course. But, you’re being hyperbolic. It’s always, otherwise, Twitter or a status page on their website.
  - aaomidi 6y ago
    Did the fire take out their database of customer emails too?
- Nextgrid 6y ago
  Their status page was also showing everything being fine in the affected datacenter at least until the next day.
- rubenbe 6y ago
  I did get an email yesterday around 16h00 GMT. It was pretty generic, the only new thing it contained (for me at least) was the FAQ regarding this incident. https://help.ovhcloud.com/en-gb/faq/strasbourg-incident/?xtor=ES-120-[fr_int_2021_ovh_multiuniverse_multiproduct_Incident_SBG_statement_11march_awareness_reach]-20210311-[FAQ_EN] https://help.ovhcloud.com/en-gb/faq/strasbourg-incident/?xto...
  - karmakaze 6y ago
    Following some links, I saw SBG1 and SBG4: we are planning to bring these datacentres and the network room onlinenext week. SBG3: we are planning to bring this datacentre online in 1-2 weeks. I'm not an OVH customer, are these building names? It seems a bit confusing that a datacentre doesn't correspond to an availability zone or is this typical at large scale?
    - throwaway81523 6y ago
      SBG is OVH Strasbourg, a complex of data centres that are adjacent to each other, but which were not built at the same time and are architecturally not the same. Yes, SBG1-4 are the different buildings. Some of them are made from stacked shipping containers, but SBG2 was an actual building. SBG2 basically burned to the ground. The fire also spread into SBG1 if I remember correctly: parts of SBG1 were destroyed but some parts are intact. SBG3 and 4 are also intact. Under the circumstances the entire complex is offline at the moment though, which is not surprising given the scale of the fire and the firefighter response. The photos reminded me of the WTC attack, which shut down lower Manhattan for months. SBG2 was built like a vertical tunnel, a quadrangle of server rooms with a huge air shaft in the middle, intended to convection-cool the servers through the natural motion of the rising air through it as the air was warmed by the servers. In this case it may have resulted in something like a firestorm, where the fire maintained its own airflow through that column. I'd be interested to know if there is any hope and/or intention in this situation to recover possibly-operable hard drives from the charred servers for data recovery purposes. I couldn't believe the complaints about the audio in that mp4. I could understand him perfectly well except a few times his accent got in the way. Better audio wouldn't have mattered much.
- remram 6y ago
  I am not directly impacted (not running anything that is advertised as physically there) but emails have been taking multiple hours to arrive for days. It would be nice to have a way of knowing whether I should change my MX servers or not. I think it's fine for them to take some time to fix their infrastructure, but if they could let me know, I can easily work around it. Incoming email is kind of difficult to monitor and needs to work... [edit] 3 separate tickets in 3 days... it seems dishonest to keep opening/closing them rather than let us know the infra is unreliable for the foreseeable future http://travaux.ovh.net/?do=details&id=49451& http://travaux.ovh.net/?do=details&id=49451& http://travaux.ovh.net/?do=details&id=49504& http://travaux.ovh.net/?do=details&id=49504& http://travaux.ovh.net/?do=details&id=49530& http://travaux.ovh.net/?do=details&id=49530&
- codezero 6y ago
  maybe all their email systems were in that building, not even joking
deleted 6y ago
[deleted]
aritmo 6y ago
The video was badly prepared. He was talking about datacentre names as if the audience are his own employees and are fully aware of the different buildings. He could at least have some graphics.
- lovedswain 6y ago
  As a customer I understood the video perfectly. Those names appear on every invoice and during every order
  - lenartowski 6y ago
    And probably you as a customer are a main target of such video, but the fire was such a high-profile case (I think) that this should be more understandable for general audience. edit: not sure why I got downvoted (and I don't really care that much), I just think that communication from CEO of big company after such event should be clear for everyone, not only for those directly affected.
    - Macha 6y ago
      Honestly, nobody lost their lives, the impact is pretty much limited to OVH (who need to find a new building), their customers (who need to find new servers and restore backups), and the city of strasbourg (who run the emergency services and presumably will have some involvement in the cleanup). While we're all interested in a "Oh, are we prepared if that happened to our provider?" sense, I don't see why OVH is obliged to provide communications to us, rather than to their customers.
    - longcommonname 6y ago
      I would expect the communication to be directly targeted to customers not the general public.
    - Ploskin 6y ago
      But ... the video isn't for a general audience. He doesn't have any reason to apologize to a general audience. His customers are the only people that matter in this situation and this video is for them.
cyberlab 6y ago
I'm glad they compartment their data centers. Having one data center for everything would be a big fail. I say this because you would be surprised at how many cloud services centralize everything in one spot.
- Cthulhu_ 6y ago
  They shouldn't advertise as 'cloud' then. I know it's a marketing buzzword and no guarantee for redundancy or anything, but still. a 'cloud' provider should have, at the very least, multiple redundant datacenters in one availability zone.
  - Clewza313 6y ago
    Literally no cloud does that. An availability zone is pretty much by definition a single point of failure, and you need to architect accordingly.
  - marcinzm 6y ago
    That's why you have multiple availability zones. Those are the physically separate locations that you have visibility into and can spread your infrastructure across. Having multiple data centers in a single availability without visibility into where your systems are is fairly useless. And if you do have visibility then you now simply have more availability zones so why bother renaming things.
- remram 6y ago
  Most cloud providers have availability regions and zones, with some services specifically having multi-region/multi-zone options (usually with extra cost). Is there any provider that hides this?
mooneater 6y ago
He looks as exactly as stressed out and sleep deprived as you might expect given the circumstances.
SirSavary 6y ago
OVH wiped my server a couple weeks ago for non-payment while I was in the middle of a support ticket to sort out a billing issue with my invoice. Was asked for more information from a support rep. Server was auto-wiped less than an hour later. Ended an eleven year relationship and after this fire I can't recommend them in good faith to anyone.
- serverlessbird 6y ago
  Had the same shit with them few months ago and also have troubles with load balancing. I am on the step of choosing a new provider. You still with OVH? Maybe you could recommend something you really confident in. I am choosing between Constellix (https://constellix.com/ https://constellix.com/) and Azure (https://azure.microsoft.com/ https://azure.microsoft.com/) now... Pros, cons?

148 points by netgusto 6y ago

progval 6y ago

duplicate of https://news.ycombinator.com/item?id=26427960 https://news.ycombinator.com/item?id=26427960

deleted 6y ago

[deleted]

imbusy111 6y ago

The worst thing, in my opinion, is that there has been no communication from OVH directly to their customers. I am impacted and I have not received a single e-mail about the situation even now.

ognarb 6y ago
We received a mail. Fortunately we were using the managed database service from ovh and could quickly restore our websites from a backup.
puxedo 6y ago
Same in my case. No communication. Generally they are good with any networking issues, attacks, etc. In this case nothing apart from the banner on the dashboard.
cpncrunch 6y ago
Octave has been putting out daily messages on twitter to customers explaining the situation.
- duxup 6y ago
  Man I hate how that works. I'll want to know what is going on with a thing and find out "oh they mostly post that kind of news on twitter" or "oh this person uses instagram", or "that news goes out via email", or youtube ... or ... who knows where ... Like guises come on, where am I supposed to get information when you're using like 8 different channels and not consistently...
  - ARandomerDude 6y ago
    This raises the question, "How could Octave be so tone-deaf?"
    - 1123581321 6y ago
      Only for people who want this incident to be part of a grand story arc about the character of OVH. The communication has been fine.
      - ARandomerDude 6y ago
        It was a bad pun.
        1123581321 6y ago
        Ah! I’m very sorry. (Doubly sorry? I know being misunderstood hertz?)
  - fshr 6y ago
    Should they be sending emails? Yes, of course. But, you’re being hyperbolic. It’s always, otherwise, Twitter or a status page on their website.
- aaomidi 6y ago
  Did the fire take out their database of customer emails too?
Nextgrid 6y ago
Their status page was also showing everything being fine in the affected datacenter at least until the next day.
rubenbe 6y ago
I did get an email yesterday around 16h00 GMT. It was pretty generic, the only new thing it contained (for me at least) was the FAQ regarding this incident. https://help.ovhcloud.com/en-gb/faq/strasbourg-incident/?xtor=ES-120-[fr_int_2021_ovh_multiuniverse_multiproduct_Incident_SBG_statement_11march_awareness_reach]-20210311-[FAQ_EN] https://help.ovhcloud.com/en-gb/faq/strasbourg-incident/?xto...
- karmakaze 6y ago
  Following some links, I saw SBG1 and SBG4: we are planning to bring these datacentres and the network room onlinenext week. SBG3: we are planning to bring this datacentre online in 1-2 weeks. I'm not an OVH customer, are these building names? It seems a bit confusing that a datacentre doesn't correspond to an availability zone or is this typical at large scale?
  - throwaway81523 6y ago
    SBG is OVH Strasbourg, a complex of data centres that are adjacent to each other, but which were not built at the same time and are architecturally not the same. Yes, SBG1-4 are the different buildings. Some of them are made from stacked shipping containers, but SBG2 was an actual building. SBG2 basically burned to the ground. The fire also spread into SBG1 if I remember correctly: parts of SBG1 were destroyed but some parts are intact. SBG3 and 4 are also intact. Under the circumstances the entire complex is offline at the moment though, which is not surprising given the scale of the fire and the firefighter response. The photos reminded me of the WTC attack, which shut down lower Manhattan for months. SBG2 was built like a vertical tunnel, a quadrangle of server rooms with a huge air shaft in the middle, intended to convection-cool the servers through the natural motion of the rising air through it as the air was warmed by the servers. In this case it may have resulted in something like a firestorm, where the fire maintained its own airflow through that column. I'd be interested to know if there is any hope and/or intention in this situation to recover possibly-operable hard drives from the charred servers for data recovery purposes. I couldn't believe the complaints about the audio in that mp4. I could understand him perfectly well except a few times his accent got in the way. Better audio wouldn't have mattered much.
remram 6y ago
I am not directly impacted (not running anything that is advertised as physically there) but emails have been taking multiple hours to arrive for days. It would be nice to have a way of knowing whether I should change my MX servers or not. I think it's fine for them to take some time to fix their infrastructure, but if they could let me know, I can easily work around it. Incoming email is kind of difficult to monitor and needs to work... [edit] 3 separate tickets in 3 days... it seems dishonest to keep opening/closing them rather than let us know the infra is unreliable for the foreseeable future http://travaux.ovh.net/?do=details&id=49451& http://travaux.ovh.net/?do=details&id=49451& http://travaux.ovh.net/?do=details&id=49504& http://travaux.ovh.net/?do=details&id=49504& http://travaux.ovh.net/?do=details&id=49530& http://travaux.ovh.net/?do=details&id=49530&
codezero 6y ago
maybe all their email systems were in that building, not even joking

deleted 6y ago

[deleted]

aritmo 6y ago

The video was badly prepared. He was talking about datacentre names as if the audience are his own employees and are fully aware of the different buildings. He could at least have some graphics.

lovedswain 6y ago
As a customer I understood the video perfectly. Those names appear on every invoice and during every order
- lenartowski 6y ago
  And probably you as a customer are a main target of such video, but the fire was such a high-profile case (I think) that this should be more understandable for general audience. edit: not sure why I got downvoted (and I don't really care that much), I just think that communication from CEO of big company after such event should be clear for everyone, not only for those directly affected.
  - Macha 6y ago
    Honestly, nobody lost their lives, the impact is pretty much limited to OVH (who need to find a new building), their customers (who need to find new servers and restore backups), and the city of strasbourg (who run the emergency services and presumably will have some involvement in the cleanup). While we're all interested in a "Oh, are we prepared if that happened to our provider?" sense, I don't see why OVH is obliged to provide communications to us, rather than to their customers.
  - longcommonname 6y ago
    I would expect the communication to be directly targeted to customers not the general public.
  - Ploskin 6y ago
    But ... the video isn't for a general audience. He doesn't have any reason to apologize to a general audience. His customers are the only people that matter in this situation and this video is for them.

cyberlab 6y ago

I'm glad they compartment their data centers. Having one data center for everything would be a big fail. I say this because you would be surprised at how many cloud services centralize everything in one spot.

Cthulhu_ 6y ago
They shouldn't advertise as 'cloud' then. I know it's a marketing buzzword and no guarantee for redundancy or anything, but still. a 'cloud' provider should have, at the very least, multiple redundant datacenters in one availability zone.
- Clewza313 6y ago
  Literally no cloud does that. An availability zone is pretty much by definition a single point of failure, and you need to architect accordingly.
- marcinzm 6y ago
  That's why you have multiple availability zones. Those are the physically separate locations that you have visibility into and can spread your infrastructure across. Having multiple data centers in a single availability without visibility into where your systems are is fairly useless. And if you do have visibility then you now simply have more availability zones so why bother renaming things.
remram 6y ago
Most cloud providers have availability regions and zones, with some services specifically having multi-region/multi-zone options (usually with extra cost). Is there any provider that hides this?

mooneater 6y ago

He looks as exactly as stressed out and sleep deprived as you might expect given the circumstances.

SirSavary 6y ago

OVH wiped my server a couple weeks ago for non-payment while I was in the middle of a support ticket to sort out a billing issue with my invoice. Was asked for more information from a support rep. Server was auto-wiped less than an hour later. Ended an eleven year relationship and after this fire I can't recommend them in good faith to anyone.

serverlessbird 6y ago
Had the same shit with them few months ago and also have troubles with load balancing. I am on the step of choosing a new provider. You still with OVH? Maybe you could recommend something you really confident in. I am choosing between Constellix (https://constellix.com/ https://constellix.com/) and Azure (https://azure.microsoft.com/ https://azure.microsoft.com/) now... Pros, cons?