3 ms·

There's a major element missing from these discussions, and I'm stunned I don't see it in the comments either. And that is, ...what happens when things go wron

There's a major element missing from these discussions, and I'm stunned I don't see it in the comments either. And that is, ...what happens when things go wrong? One of the major advantages of cloud, be it AWS, Azure, GCP, etc, is the ability to bridge across failure domains. Properly architected, a workload can be resilient in the face of data center outages and circuit breaks. To me, these savings from operating in a single colo are living on borrowed time. Because they're one data center fire or cable break away from failure. Can the workload survive a sudden outage for hours? A day? Will it survive a loss of the data center and implementation of the business continuity plan? (There is a plan, right?) Every organization is different of course, and not everyone needs active-active failover, or 4-nines of availability, etc. But when figures of $400m start getting thrown around, I can't imagine it's in a state where "oh, we'll just rebuild it elsewhere" is the plan. Show me where they achieve these savings and still have a viable DR/COOP, and I'll be impressed. Disclaimer: I'm a TAM for AWS

jjav 4y ago
> ...what happens when things go wrong? The same that happens when things go wrong with AWS (just cheaper). One can of course engineer for any level of redundancy the business needs, whether on colo or cloud. Or not, as in most cases. We had four multi-hour outages due to AWS being down last year.
Melatonic 4y ago
You can also use more than one COLO or have your DR stuff located in the cloud provider of your choice :-)
dboreham 4y ago
You use more than one colo. Nothing about not using AWS says you need to not care about HA.
weberer 4y ago
>But when figures of $400m start getting thrown around, I can't imagine it's in a state where "oh, we'll just rebuild it elsewhere" is the plan. If you're operating at that scale, you're likely to be operating in multiple data centers with load balancers already.
q1w2 4y ago
I appreciate this argument, and let me give the counter-example. 99.5% uptime is good enough for nearly all businesses. I run a small IT service company - we provide a hosted service in the insurance industry. We host on-premise, in our physical office (well, across 2 offices for DR). I personally have a history in infra management, so I generally enjoy it. You're right that sometimes things go wrong. Last year, we were down for about 3 hours. The year before that, there was a 6 hour outage. We've had failures in power, ISP, networking, server hardware, software, etc... but we've matured to a point that we can handle them quickly. Most importantly, these outages are understood by the business and the users. They know going in that they get a credit for outages, and that it won't exceed 1 business day per year (usually much less). Because we accept this 0.5% downtime (it's actually much lower in reality), we pay 1/10th what I've been quoted by cloud services - probably even less.
- leviathant 4y ago
  >99.5% uptime is good enough for nearly all businesses. I don't know what your business is, but as a presales engineer in the ecommerce field, if our SLA was 99.5% we wouldn't even make it past the first round of vendor selection.
  - jjav 4y ago
    > I don't know what your business is, but as a presales engineer in the ecommerce field, if our SLA was 99.5% we wouldn't even make it past the first round of vendor selection. As you know, the SLA isn't the same as the downtime. It just means the threshhold where you'd start paying back some credits for missing the SLA (only to customers who notice and ask). Which is often cheaper than building the infrastructure to actually support the promised SLA.
    - stcroixx 4y ago
      Our customer contracts obligate us to inform all customers about any outages. We can’t just hope they don’t notice and only pay the ones that happen to.
pritambarhate 4y ago
Meanwhile AWS keeps having region wide outages almost every year which last for many hours - https://awsmaniac.com/aws-outages/ https://awsmaniac.com/aws-outages/ Architecting a multi-region app on AWS is not trivial. All basic building blocks AWS provides makes running MultiAZ very easy, however very few AWS services support multi region deployments and failover out of the box.

jjav 4y ago

> ...what happens when things go wrong? The same that happens when things go wrong with AWS (just cheaper). One can of course engineer for any level of redundancy the business needs, whether on colo or cloud. Or not, as in most cases. We had four multi-hour outages due to AWS being down last year.

Melatonic 4y ago

You can also use more than one COLO or have your DR stuff located in the cloud provider of your choice :-)

dboreham 4y ago

You use more than one colo. Nothing about not using AWS says you need to not care about HA.

weberer 4y ago

>But when figures of $400m start getting thrown around, I can't imagine it's in a state where "oh, we'll just rebuild it elsewhere" is the plan. If you're operating at that scale, you're likely to be operating in multiple data centers with load balancers already.

q1w2 4y ago

I appreciate this argument, and let me give the counter-example. 99.5% uptime is good enough for nearly all businesses. I run a small IT service company - we provide a hosted service in the insurance industry. We host on-premise, in our physical office (well, across 2 offices for DR). I personally have a history in infra management, so I generally enjoy it. You're right that sometimes things go wrong. Last year, we were down for about 3 hours. The year before that, there was a 6 hour outage. We've had failures in power, ISP, networking, server hardware, software, etc... but we've matured to a point that we can handle them quickly. Most importantly, these outages are understood by the business and the users. They know going in that they get a credit for outages, and that it won't exceed 1 business day per year (usually much less). Because we accept this 0.5% downtime (it's actually much lower in reality), we pay 1/10th what I've been quoted by cloud services - probably even less.

leviathant 4y ago
>99.5% uptime is good enough for nearly all businesses. I don't know what your business is, but as a presales engineer in the ecommerce field, if our SLA was 99.5% we wouldn't even make it past the first round of vendor selection.
- jjav 4y ago
  > I don't know what your business is, but as a presales engineer in the ecommerce field, if our SLA was 99.5% we wouldn't even make it past the first round of vendor selection. As you know, the SLA isn't the same as the downtime. It just means the threshhold where you'd start paying back some credits for missing the SLA (only to customers who notice and ask). Which is often cheaper than building the infrastructure to actually support the promised SLA.
  - stcroixx 4y ago
    Our customer contracts obligate us to inform all customers about any outages. We can’t just hope they don’t notice and only pay the ones that happen to.

pritambarhate 4y ago

Meanwhile AWS keeps having region wide outages almost every year which last for many hours - https://awsmaniac.com/aws-outages/ https://awsmaniac.com/aws-outages/ Architecting a multi-region app on AWS is not trivial. All basic building blocks AWS provides makes running MultiAZ very easy, however very few AWS services support multi region deployments and failover out of the box.