7 ms·

Every time a major cloud provider has an outage, Infra people and execs cry foul and say we need to move to <the other one>. But does anyone really have an obje

Every time a major cloud provider has an outage, Infra people and execs cry foul and say we need to move to <the other one>. But does anyone really have an objective measure of how clouds stack up reliability-wise? I doubt it, since outages and their effects are nuanced. The other move is that they want to go multi-cloud... But I’ve been involved in enough multi-cloud initiatives to know how much time and effort those soak up, not to mention the overhead costs of maintaining two sets of infra sub-optimally. I would say that for most businesses, these costs far exceed that occasional six-hour-long outage.

metadat 5y ago
I know the Oracle OCI cloud has a reputation for never going hard-down, but also realize HN seems to loathe Big Red (understandably, to a degree, though OCI is pretty nice IME and _very_ predictable).
- SixDouble5321 5y ago
  I don't think it's unfair. They aren't the worst villain, but they are up there.
mongrelion 5y ago
I agree with you. I think that having multi-AZ is the first thing to figure out before wanting to do multi-cloud, which is just another buzzword taken out of management's bullshit bucket :)
- Hippocrates 5y ago
  Agree, and multi AZ is usually easy. IME with AWS and GCP the control plane is the same, the scaling works across AZ, bandwidth is free and latency is near zero. The level of effort to do that is simply ticking the right boxes at setup time IME.
  - Jweb_Guru 5y ago
    Cross-AZ bandwidth is far from free and the biggest reason companies avoid it (IMO). Also latency is not near zero but I don't think that's the primary reason.
    - mongrelion 5y ago
      That's true. The moment your data leaves the region you start paying for egress and that can get expensive quickly. Still beats the crap out of multicloud, though :)
- jtc331 5y ago
  I’ve seen at least half a dozen full region AWS issues in the past 8 months. You really need multi-region and also not be relying on any AWS service that’s located only in us-east-1 (including everything from creating new S3 buckets to IAM’s STS).
sdevonoes 5y ago
Perhaps is us, the customers (and our customers, and the customers of our customers, ...), the ones who should get used to the status of "things can go wrong"? Except for some specific scenarios (medical-related stuff, for instance), if my favourite online shopping place is down, well, it's down, I'll buy later.
mijoharas 5y ago
I mean from the explanation[0], assuming that is correct (I don't have evidence to suggest it's false) - you don't need to be multi-cloud, and you don't even need to be multi-region. As long as you're spread out over multiple availability zones in a region you should be resilient to this failure. Somewhat surprising to see how many things are failing though, which implies, either that a lot of services aren't able to fail-over to a different availability zone, or there is something else going wrong. [0] https://news.ycombinator.com/item?id=29648992 https://news.ycombinator.com/item?id=29648992
- zeckalpha 5y ago
  That’s true for this failure but the prior two for AWS were region wide and the one for GCP last month was global.
- omh2 5y ago
  AWS doesn't follow their own advice about hosting multi-regional so every time us-east-1 has significant issues pretty much every AZ and region is affected. Specifically large parts of the management API, and IAM service are seemingly centrally hosted in us-east-1. So called Global endpoints are also dependent on us-east-1 and parts of AWS' internal event queues (eg. event bridge triggers) If your infrastructure is static you'll largely avoid the fallout, but if you rely on API calls or dynamically created resources you can get caught in the blast regardless of region
  - spmurrayzzz 5y ago
    Your last comment is really important, I think. I have always petitioned for "passive over active" design in distributed cloud systems. The recent outages, and also ones from the past, demonstrate why. The fewer API calls you need to make in-band with whatever throughput is generated via your customer demand, the better. Related to that, I have been critical of lambda/FaaS/serverless infrastructure patterns for similar reasons. Always felt like a brittle house of cards to me (N.B. I do still use aws lambda, but keep it constrained to non-critical workloads).
    - 5y ago
indigomm 5y ago
> I doubt it, since outages and their effects are nuanced. Your point here deserves highlighting. A failure such as a zone failing is nowadays a relatively simple problem to have. But cloud services do have bugs, internal limits or partial failures that are much more complex. They often require support assistance, which is where the expertise of their staff comes into play. Having a single provider that you know well and trust is better than having multiple providers where you need to keep track of disparate issues.
sfoley 5y ago
Who says this? I have literally never once seen this.

metadat 5y ago

I know the Oracle OCI cloud has a reputation for never going hard-down, but also realize HN seems to loathe Big Red (understandably, to a degree, though OCI is pretty nice IME and _very_ predictable).

SixDouble5321 5y ago
I don't think it's unfair. They aren't the worst villain, but they are up there.

mongrelion 5y ago

I agree with you. I think that having multi-AZ is the first thing to figure out before wanting to do multi-cloud, which is just another buzzword taken out of management's bullshit bucket :)

Hippocrates 5y ago
Agree, and multi AZ is usually easy. IME with AWS and GCP the control plane is the same, the scaling works across AZ, bandwidth is free and latency is near zero. The level of effort to do that is simply ticking the right boxes at setup time IME.
- Jweb_Guru 5y ago
  Cross-AZ bandwidth is far from free and the biggest reason companies avoid it (IMO). Also latency is not near zero but I don't think that's the primary reason.
  - mongrelion 5y ago
    That's true. The moment your data leaves the region you start paying for egress and that can get expensive quickly. Still beats the crap out of multicloud, though :)
jtc331 5y ago
I’ve seen at least half a dozen full region AWS issues in the past 8 months. You really need multi-region and also not be relying on any AWS service that’s located only in us-east-1 (including everything from creating new S3 buckets to IAM’s STS).

sdevonoes 5y ago

Perhaps is us, the customers (and our customers, and the customers of our customers, ...), the ones who should get used to the status of "things can go wrong"? Except for some specific scenarios (medical-related stuff, for instance), if my favourite online shopping place is down, well, it's down, I'll buy later.

mijoharas 5y ago

I mean from the explanation[0], assuming that is correct (I don't have evidence to suggest it's false) - you don't need to be multi-cloud, and you don't even need to be multi-region. As long as you're spread out over multiple availability zones in a region you should be resilient to this failure. Somewhat surprising to see how many things are failing though, which implies, either that a lot of services aren't able to fail-over to a different availability zone, or there is something else going wrong. [0] https://news.ycombinator.com/item?id=29648992 https://news.ycombinator.com/item?id=29648992

zeckalpha 5y ago
That’s true for this failure but the prior two for AWS were region wide and the one for GCP last month was global.
omh2 5y ago
AWS doesn't follow their own advice about hosting multi-regional so every time us-east-1 has significant issues pretty much every AZ and region is affected. Specifically large parts of the management API, and IAM service are seemingly centrally hosted in us-east-1. So called Global endpoints are also dependent on us-east-1 and parts of AWS' internal event queues (eg. event bridge triggers) If your infrastructure is static you'll largely avoid the fallout, but if you rely on API calls or dynamically created resources you can get caught in the blast regardless of region
- spmurrayzzz 5y ago
  Your last comment is really important, I think. I have always petitioned for "passive over active" design in distributed cloud systems. The recent outages, and also ones from the past, demonstrate why. The fewer API calls you need to make in-band with whatever throughput is generated via your customer demand, the better. Related to that, I have been critical of lambda/FaaS/serverless infrastructure patterns for similar reasons. Always felt like a brittle house of cards to me (N.B. I do still use aws lambda, but keep it constrained to non-critical workloads).
  - 5y ago

indigomm 5y ago

> I doubt it, since outages and their effects are nuanced. Your point here deserves highlighting. A failure such as a zone failing is nowadays a relatively simple problem to have. But cloud services do have bugs, internal limits or partial failures that are much more complex. They often require support assistance, which is where the expertise of their staff comes into play. Having a single provider that you know well and trust is better than having multiple providers where you need to keep track of disparate issues.

sfoley 5y ago

Who says this? I have literally never once seen this.