5 ms·

Whatever happened to 5-nines uptime? It seems no cloud service provider these days is able to offer what was considered an industry standard. AWS even have doc

Whatever happened to 5-nines uptime? It seems no cloud service provider these days is able to offer what was considered an industry standard. AWS even have documents telling people how to achieve exactly this! https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/welcome.html https://docs.aws.amazon.com/wellarchitected/latest/reliabili... Why don't "premium" service providers like heroku, etc, do this?

ranguna 5y ago
99.999% uptime still mean around 7 and a half hours of downtime per year.
- rafBM 5y ago
  99.999% uptime is 5m 15s per year: https://uptime.is/99.999 https://uptime.is/99.999
  - deleted 5y ago
    [deleted]
- bobviolier 5y ago
  No, just 5 minutes https://uptime.is/99.999 https://uptime.is/99.999
- ranguna 5y ago
  Ups, quick maths == wrong maths.
makeitdouble 5y ago
TBF there is very few real world services that offer customers and non giant size companies 5-nines of uptime. E.g. my electricity provider doesn't.
- supermatt 5y ago
  Services providers such as Heroku should be easily able to have five-nines uptime. They ONLY offer fully managed services, which can be backed by the multi-cloud, multi-AZ setup I refer to - but instead a single product outage from a single upstream provider in a single datacenter is affecting all their clients. This is a regular occurrence for Heroku - and they charge a substantial premium for their "service".
  - danjac 5y ago
    https://floridanewstimes.com/the-decline-of-heroku-infoworld/208803/ https://floridanewstimes.com/the-decline-of-heroku-infoworld...
zokier 5y ago
> It seems no cloud service provider these days is able to offer what was considered an industry standard. I wonder how many services really had 5 nines availability pre-cloud era either. Somehow I feel your view of it being "industry standard" might be slightly rose tinted
- RantyDave 5y ago
  Plenty. But in most cases there was luck involved.
- lbriner 5y ago
  The issue is more like the guarantee is not worth anything. AWS/Azure/whomever "promise" 5 9s uptime. Something goes wrong, you don't get 5 9s, and what do you get? A system that went down for 4 hours and a $50 rebate on your next bill!
  - sciurus 5y ago
    Your point about the credit s stands, but could providers don't even offer 5 9 SLAs. E.G. https://aws.amazon.com/compute/sla/ https://aws.amazon.com/compute/sla/
  - supermatt 5y ago
    its not about AWS/Azure etc. They are providing IaaS. Literally compute services littered around the globe. It is up to these so-called cloud service providers, like heroku, to utilize that infrastructure to achieve 99.999%. I even gave a link in my comment to what AWS say about this. Are people downvoting me because they dont read, or what?
    - weird-eye-issue 5y ago
      You make it sound like AWS has 100% uptime and services built on top of them are completely to blame. And for something like Heroku's managed DBs you can't just achieve 99.99999% availability on a DB without making certain sacrifices. Availability isn't everything past a certain point
CodesInChaos 5y ago
For non trivial services (in particular ones that need consistency), I'm skeptical that it's realistic to achieve 5-nines at competitive cost. You'll probably achieve it for several years, and then you run into a complex failure which takes 1h to fix, blowing through the downtime budget of a decade.
- supermatt 5y ago
  In herokus case: Their "dynos" are ephemeral. They could literally deploy the images to a backup environment hosted elsewhere. Their data services could all be synchronously replicated to that backup environment. And thats it - they dont offer any other core services (and their other services run on the same platform.) So for (at most) double their infrastructure cost they have another network they can immediately switch over to. And herokus already soooo expensive. Even if you used a 1-to-1 mapping for ec2 to heroku dynos (which they dont - its multiple dynos per backing instance), you would be looking at 5-10x markup using on-demand instances! Reserved instances are even less expensive. Spot instances can be 5x less again! I think they could retain their current pricing model and still offer this kind of resiliency - at a minimum. Fly.io is making strides in this direction, distributing the VMs across multiple availability zones, and routing traffic internally from their multiple geographically distributed POPs - but you need to roll your own DB VMs for multi-az synchronization.. EDIT: seems they do provide managed postgres with synchronized replication now (in beta), neat!
busterarm 5y ago
AWS doesn't offer 5-nines uptime on Compute. It's S3 that is 5-nines availability. AWS's published SLA for Compute (which includes EBS) is 4-nines. https://aws.amazon.com/compute/sla/ https://aws.amazon.com/compute/sla/
- supermatt 5y ago
  I know they dont, but in the link I gave they tell you how to achieve 5-nines via redundancy - something these cloud service providers (like heroku) neglect to implement.
- terom 5y ago
  It's worth noting that the AWS EC2 99.99% SLA is a regional SLA, i.e. it only covers a situation where multiple AZs are down simultaneously. One AZ going down is not covered by the 99.99% SLA. AFAIK there isn't any per-AZ SLA, only a single-instance SLA of 99.5%. The effective per-AZ SLA is going to be somewhere between the two.
  - busterarm 5y ago
    Design accordingly. :)

ranguna 5y ago

99.999% uptime still mean around 7 and a half hours of downtime per year.

rafBM 5y ago
99.999% uptime is 5m 15s per year: https://uptime.is/99.999 https://uptime.is/99.999
- deleted 5y ago
  [deleted]
bobviolier 5y ago
No, just 5 minutes https://uptime.is/99.999 https://uptime.is/99.999
ranguna 5y ago
Ups, quick maths == wrong maths.

makeitdouble 5y ago

TBF there is very few real world services that offer customers and non giant size companies 5-nines of uptime. E.g. my electricity provider doesn't.

supermatt 5y ago
Services providers such as Heroku should be easily able to have five-nines uptime. They ONLY offer fully managed services, which can be backed by the multi-cloud, multi-AZ setup I refer to - but instead a single product outage from a single upstream provider in a single datacenter is affecting all their clients. This is a regular occurrence for Heroku - and they charge a substantial premium for their "service".
- danjac 5y ago
  https://floridanewstimes.com/the-decline-of-heroku-infoworld/208803/ https://floridanewstimes.com/the-decline-of-heroku-infoworld...

zokier 5y ago

> It seems no cloud service provider these days is able to offer what was considered an industry standard. I wonder how many services really had 5 nines availability pre-cloud era either. Somehow I feel your view of it being "industry standard" might be slightly rose tinted

RantyDave 5y ago
Plenty. But in most cases there was luck involved.
lbriner 5y ago
The issue is more like the guarantee is not worth anything. AWS/Azure/whomever "promise" 5 9s uptime. Something goes wrong, you don't get 5 9s, and what do you get? A system that went down for 4 hours and a $50 rebate on your next bill!
- sciurus 5y ago
  Your point about the credit s stands, but could providers don't even offer 5 9 SLAs. E.G. https://aws.amazon.com/compute/sla/ https://aws.amazon.com/compute/sla/
- supermatt 5y ago
  its not about AWS/Azure etc. They are providing IaaS. Literally compute services littered around the globe. It is up to these so-called cloud service providers, like heroku, to utilize that infrastructure to achieve 99.999%. I even gave a link in my comment to what AWS say about this. Are people downvoting me because they dont read, or what?
  - weird-eye-issue 5y ago
    You make it sound like AWS has 100% uptime and services built on top of them are completely to blame. And for something like Heroku's managed DBs you can't just achieve 99.99999% availability on a DB without making certain sacrifices. Availability isn't everything past a certain point

CodesInChaos 5y ago

For non trivial services (in particular ones that need consistency), I'm skeptical that it's realistic to achieve 5-nines at competitive cost. You'll probably achieve it for several years, and then you run into a complex failure which takes 1h to fix, blowing through the downtime budget of a decade.

supermatt 5y ago
In herokus case: Their "dynos" are ephemeral. They could literally deploy the images to a backup environment hosted elsewhere. Their data services could all be synchronously replicated to that backup environment. And thats it - they dont offer any other core services (and their other services run on the same platform.) So for (at most) double their infrastructure cost they have another network they can immediately switch over to. And herokus already soooo expensive. Even if you used a 1-to-1 mapping for ec2 to heroku dynos (which they dont - its multiple dynos per backing instance), you would be looking at 5-10x markup using on-demand instances! Reserved instances are even less expensive. Spot instances can be 5x less again! I think they could retain their current pricing model and still offer this kind of resiliency - at a minimum. Fly.io is making strides in this direction, distributing the VMs across multiple availability zones, and routing traffic internally from their multiple geographically distributed POPs - but you need to roll your own DB VMs for multi-az synchronization.. EDIT: seems they do provide managed postgres with synchronized replication now (in beta), neat!

busterarm 5y ago

AWS doesn't offer 5-nines uptime on Compute. It's S3 that is 5-nines availability. AWS's published SLA for Compute (which includes EBS) is 4-nines. https://aws.amazon.com/compute/sla/ https://aws.amazon.com/compute/sla/

supermatt 5y ago
I know they dont, but in the link I gave they tell you how to achieve 5-nines via redundancy - something these cloud service providers (like heroku) neglect to implement.
terom 5y ago
It's worth noting that the AWS EC2 99.99% SLA is a regional SLA, i.e. it only covers a situation where multiple AZs are down simultaneously. One AZ going down is not covered by the 99.99% SLA. AFAIK there isn't any per-AZ SLA, only a single-instance SLA of 99.5%. The effective per-AZ SLA is going to be somewhere between the two.
- busterarm 5y ago
  Design accordingly. :)