5 ms·

Unfortunately A lot of people are going to find out the hard way today why AWS/GCP/Big Expensive Cloud is so expensive (Hint: they have redundancy and failover

by pm90 6y ago

Unfortunately A lot of people are going to find out the hard way today why AWS/GCP/Big Expensive Cloud is so expensive (Hint: they have redundancy and failover procedures which drive up costs). Keep in mind I’m talking not of “downtime” but of actual data loss which might affect business continuity. This is really tragic. I’m hoping they have some kind of multi regional backup/replication and not just multi zones (although from the twitters it appears that only one of the zones was destroyed however the others don’t seem to be operational atm).

hashhar 6y ago
AWS and GCP are also prone to same kind of data loss if the AZ you are operating in goes down. They don't automatically geo-replicate things. You still need a backup for the torched EC2 instance to be able to relaunch in another AZ/region.
- nnx 6y ago
  That's true, but it seems whole of SBG region for OVH is within same disaster radius for one fire... with SBG2 destroyed and SBG1 partly damaged. "The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4. " Wonder if those SBGx were advertised as being the same as "Availability Zones" - when other cloud providers ensure zones are distanced enough from each other (~1km at least) to likely survive events such as fire.
  - hashhar 6y ago
    Thats a fair point. If OVH does market them as AZs then it's disingenuous and liable to suits IMO.
    - cbg0 6y ago
      No, it isn't, as there's no clear cut definition of what an availability zone is.
  - tyingq 6y ago
    The buildings are VERY close to one another. https://cdn.baxtel.com/data-center/ovh-strasbourg-campus/photos/flickity_display_OVH-Strasbourg-Campus-OVH-Strasbourg-Campus-showing-all-3-Buildings-2019.png https://cdn.baxtel.com/data-center/ovh-strasbourg-campus/pho...
    - Zevis 6y ago
      That seems, uh, problematic.
      - ZWoz 6y ago
        Why? They aren't making any sort guarantees with those locations, those aren't advertised as separated fault zones. Those buildings are meant as expansions for existing site.
  - brmgb 6y ago
    > but it seems whole of SBG region for OVH is within same disaster radius for one fire SBG is for Strasbourg. That's not a region. It's a city. Obviously, SBG1 to 4 are in the radius of one fire. It's four different buildings on the same site.
  - ev1 6y ago
    They were not and never advertised as anything similar to AZ. You could not deploy in SBG1,2,3, etc. You only pick city = Strasbourg at deploy time. It's merely a building marker.
leajkinUnk 6y ago
“Big cloud” has had fires take out clusters, and somehow they manage to keep it out of the news. In spite of the redundancy and failover procedures, keeping your data centers running when one of the clusters was recently *on fire* is something that is often only possible due to heroic efforts. When I say “heroic efforts”, that’s in contrast to “ordinary error recovery and failover”, which is the way you’d want to handle a DC fire, because DC fires happen often enough. The thing is, while these big companies have a much larger base of expertise to draw on and simply more staff time to throw at problems, there are factors which incentivize these employees to *increase risk* rather than reduce it. These big companies put pressure on all their engineers to figure out ways to drive down costs. So, while a big cloud provider won’t make a rookie mistake—they won’t forget to run disaster recovery drills, they won’t forget to make backups and run test restores—they *will* do a bunch of calculations to figure out how close to disaster they can run in order to save money. The real disaster will then reveal some false, hidden assumption in their error recovery models. Or in other words, the big companies solve all the easy problems and then create new, hard problems.
- deleted 6y ago
  [deleted]
- pm90 6y ago
  You know, those are excellent observations. But they don’t change the decision calculus in this case. Using bigger cloud providers doesn’t eliminate all risk, it just creates a different kind of risk. What we call “progress” in humanity is just putting our best efforts into reducing or eliminating the problems we know how to solve without realizing the problems they may create further down the line. The only way to know for sure is to try it, see how it goes, and then re-evaluate later. California had issues with many forest fires. They put out all fires. Turns out, that solution creates a bigger problem down the line with humongous uncontrollable fires which would not have happened if the smaller fires had not been put out so frequently. Oops.
- exikyut 6y ago
  I'm curious what references or leads I might follow to learn more about these fires and other events you mention.
the_duke 6y ago
I encourage you to have a look at the operating income that AWS rakes in. Sure, the amount of expertise, redundancy and breadth of service offerings they provide is worth a markup, but they are also significantly more expensive than they need to be. Thanks to being the leader in an oligopoly, and due to patterns like making network egress unjustifiably expensive to keep you (/your data) from leaving.
- pm90 6y ago
  I think the question here, then is of subjective value. AWS may charge more for egress, but that’s not high enough for it to be a concern for most clients. A bigger, independent concern is probably that there should be sufficient redundancy, backups and such that allows for business continuity. (Note again that I’m not saying that all companies make full use of these features, but those that care for such things do. Additionally, I’ve honestly never heard of an AWS DC burning down. Either it doesn’t happen frequently or it doesn’t have enough of an effect on regular customers, both of situations are equivalent for my case). Most businesses choose to prioritize the second aspect. Even if they have to pay extra for egress sometimes, it’s just not big enough of a concern as compared to businesses continuity.
  - hhw 6y ago
    I've never heard of any data centre burning down (and I work in this industry), so never hearing of an AWS DC burning down isn't really saying anything about AWS.
    - ev1 6y ago
      I remember hosting.ua DC mysteriously "catch fire"
  - opsunit 6y ago
    An availability zone (AZ) in AWS eu-west-2 was flooded by a fire protection system going off within the last year. It absolutely did affect workloads in that AZ. That shouldn't have had a large impact on their customers since AWS promote and make as trivial as is viable multi-AZ architectures. Put another way: one is guided towards making operational good choices rather than being left to discover them yourself. This is a value proposition of public clouds since it commoditises that specialist knowledge.
x3sphere 6y ago
If the data is that critical, surely you would be backing it up frequently and also mirror it on at least one geographically separate server? I use a single server at OVH, and I'm not in the affected DC, but if this DID happen to me I could get back up and running fairly quickly. All our data is mirrored on S3 and off site backups are made frequently enough it wouldn't be an issue. Plus, you still need to plan for a scenario like this even with AWS or any other cloud provider. It is less likely to happen with those, given the redundancy, but there is still a chance you lose it all without a backup plan.
zelly 6y ago
Yup, I've never heard of a fire taking out a Big Cloud DC. They actually know what they're doing and don't put server racks in shipping containers stacked on top of each other. If you want quality in life, sometimes you have to pay for it. Personally I'll continue to use these third world cloud providers. But I like to live on the edge.
- jiofih 6y ago
  Apple fire: https://www.datacenterdynamics.com/en/news/fire-rages-through-apples-data-center-hq-in-mesa-arizona/ https://www.datacenterdynamics.com/en/news/fire-rages-throug... Google fire: https://www.google.nl/amp/s/gigazine.net/amp/en/20060313_google_fire https://www.google.nl/amp/s/gigazine.net/amp/en/20060313_goo... AWS fire: https://money.cnn.com/2015/01/09/technology/amazon-data-center-fire/ https://money.cnn.com/2015/01/09/technology/amazon-data-cent...
  - zelly 6y ago
    > Apple fire Their solar panels on the roof caught on fire. Presumably there was no actual disruption in service computing-wise. > Google fire 2006, still a young company at that point. Also this was way before GCP where they were responsible for other people's services. > AWS fire Data center caught on fire while it was under construction. Drunk construction workers messed up, not AWS. Presumably no computers were even inside yet. This is completely different from OVH completely burning down while all their customers' data went up in flames. If these are the best examples of Big Cloud messing up, this is quite the endorsement. It's funny how the OVH CEO tells everyone to go activate their DRPs while the company didn't have enough foresight to install fire suppression systems.
    - jiofih 6y ago
      These are just the first examples I could find. Fires account for 20%+ of all DC outages in the industry. But taking the time to find a reason to invalidate each one, it seems like you have your mind set.

by pm90 6y ago

hashhar 6y ago

AWS and GCP are also prone to same kind of data loss if the AZ you are operating in goes down. They don't automatically geo-replicate things. You still need a backup for the torched EC2 instance to be able to relaunch in another AZ/region.

nnx 6y ago
That's true, but it seems whole of SBG region for OVH is within same disaster radius for one fire... with SBG2 destroyed and SBG1 partly damaged. "The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4. " Wonder if those SBGx were advertised as being the same as "Availability Zones" - when other cloud providers ensure zones are distanced enough from each other (~1km at least) to likely survive events such as fire.
- hashhar 6y ago
  Thats a fair point. If OVH does market them as AZs then it's disingenuous and liable to suits IMO.
  - cbg0 6y ago
    No, it isn't, as there's no clear cut definition of what an availability zone is.
- tyingq 6y ago
  The buildings are VERY close to one another. https://cdn.baxtel.com/data-center/ovh-strasbourg-campus/photos/flickity_display_OVH-Strasbourg-Campus-OVH-Strasbourg-Campus-showing-all-3-Buildings-2019.png https://cdn.baxtel.com/data-center/ovh-strasbourg-campus/pho...
  - Zevis 6y ago
    That seems, uh, problematic.
    - ZWoz 6y ago
      Why? They aren't making any sort guarantees with those locations, those aren't advertised as separated fault zones. Those buildings are meant as expansions for existing site.
- brmgb 6y ago
  > but it seems whole of SBG region for OVH is within same disaster radius for one fire SBG is for Strasbourg. That's not a region. It's a city. Obviously, SBG1 to 4 are in the radius of one fire. It's four different buildings on the same site.
- ev1 6y ago
  They were not and never advertised as anything similar to AZ. You could not deploy in SBG1,2,3, etc. You only pick city = Strasbourg at deploy time. It's merely a building marker.

leajkinUnk 6y ago

“Big cloud” has had fires take out clusters, and somehow they manage to keep it out of the news. In spite of the redundancy and failover procedures, keeping your data centers running when one of the clusters was recently *on fire* is something that is often only possible due to heroic efforts. When I say “heroic efforts”, that’s in contrast to “ordinary error recovery and failover”, which is the way you’d want to handle a DC fire, because DC fires happen often enough. The thing is, while these big companies have a much larger base of expertise to draw on and simply more staff time to throw at problems, there are factors which incentivize these employees to *increase risk* rather than reduce it. These big companies put pressure on all their engineers to figure out ways to drive down costs. So, while a big cloud provider won’t make a rookie mistake—they won’t forget to run disaster recovery drills, they won’t forget to make backups and run test restores—they *will* do a bunch of calculations to figure out how close to disaster they can run in order to save money. The real disaster will then reveal some false, hidden assumption in their error recovery models. Or in other words, the big companies solve all the easy problems and then create new, hard problems.

deleted 6y ago
[deleted]
pm90 6y ago
You know, those are excellent observations. But they don’t change the decision calculus in this case. Using bigger cloud providers doesn’t eliminate all risk, it just creates a different kind of risk. What we call “progress” in humanity is just putting our best efforts into reducing or eliminating the problems we know how to solve without realizing the problems they may create further down the line. The only way to know for sure is to try it, see how it goes, and then re-evaluate later. California had issues with many forest fires. They put out all fires. Turns out, that solution creates a bigger problem down the line with humongous uncontrollable fires which would not have happened if the smaller fires had not been put out so frequently. Oops.
exikyut 6y ago
I'm curious what references or leads I might follow to learn more about these fires and other events you mention.

the_duke 6y ago

I encourage you to have a look at the operating income that AWS rakes in. Sure, the amount of expertise, redundancy and breadth of service offerings they provide is worth a markup, but they are also significantly more expensive than they need to be. Thanks to being the leader in an oligopoly, and due to patterns like making network egress unjustifiably expensive to keep you (/your data) from leaving.

pm90 6y ago
I think the question here, then is of subjective value. AWS may charge more for egress, but that’s not high enough for it to be a concern for most clients. A bigger, independent concern is probably that there should be sufficient redundancy, backups and such that allows for business continuity. (Note again that I’m not saying that all companies make full use of these features, but those that care for such things do. Additionally, I’ve honestly never heard of an AWS DC burning down. Either it doesn’t happen frequently or it doesn’t have enough of an effect on regular customers, both of situations are equivalent for my case). Most businesses choose to prioritize the second aspect. Even if they have to pay extra for egress sometimes, it’s just not big enough of a concern as compared to businesses continuity.
- hhw 6y ago
  I've never heard of any data centre burning down (and I work in this industry), so never hearing of an AWS DC burning down isn't really saying anything about AWS.
  - ev1 6y ago
    I remember hosting.ua DC mysteriously "catch fire"
- opsunit 6y ago
  An availability zone (AZ) in AWS eu-west-2 was flooded by a fire protection system going off within the last year. It absolutely did affect workloads in that AZ. That shouldn't have had a large impact on their customers since AWS promote and make as trivial as is viable multi-AZ architectures. Put another way: one is guided towards making operational good choices rather than being left to discover them yourself. This is a value proposition of public clouds since it commoditises that specialist knowledge.

x3sphere 6y ago

If the data is that critical, surely you would be backing it up frequently and also mirror it on at least one geographically separate server? I use a single server at OVH, and I'm not in the affected DC, but if this DID happen to me I could get back up and running fairly quickly. All our data is mirrored on S3 and off site backups are made frequently enough it wouldn't be an issue. Plus, you still need to plan for a scenario like this even with AWS or any other cloud provider. It is less likely to happen with those, given the redundancy, but there is still a chance you lose it all without a backup plan.

zelly 6y ago

Yup, I've never heard of a fire taking out a Big Cloud DC. They actually know what they're doing and don't put server racks in shipping containers stacked on top of each other. If you want quality in life, sometimes you have to pay for it. Personally I'll continue to use these third world cloud providers. But I like to live on the edge.

jiofih 6y ago
Apple fire: https://www.datacenterdynamics.com/en/news/fire-rages-through-apples-data-center-hq-in-mesa-arizona/ https://www.datacenterdynamics.com/en/news/fire-rages-throug... Google fire: https://www.google.nl/amp/s/gigazine.net/amp/en/20060313_google_fire https://www.google.nl/amp/s/gigazine.net/amp/en/20060313_goo... AWS fire: https://money.cnn.com/2015/01/09/technology/amazon-data-center-fire/ https://money.cnn.com/2015/01/09/technology/amazon-data-cent...
- zelly 6y ago
  > Apple fire Their solar panels on the roof caught on fire. Presumably there was no actual disruption in service computing-wise. > Google fire 2006, still a young company at that point. Also this was way before GCP where they were responsible for other people's services. > AWS fire Data center caught on fire while it was under construction. Drunk construction workers messed up, not AWS. Presumably no computers were even inside yet. This is completely different from OVH completely burning down while all their customers' data went up in flames. If these are the best examples of Big Cloud messing up, this is quite the endorsement. It's funny how the OVH CEO tells everyone to go activate their DRPs while the company didn't have enough foresight to install fire suppression systems.
  - jiofih 6y ago
    These are just the first examples I could find. Fires account for 20%+ of all DC outages in the industry. But taking the time to find a reason to invalidate each one, it seems like you have your mind set.