6 ms·

All the big clouds regularly have outages.

by zie 8y ago

outworlder 8y ago
In _all_ regions? No.
- NicoJuicy 8y ago
  Amazon AWS had/has a dependency on one of their first datacenters last year for example
  - syntheticcdo 8y ago
    The us-east-1 outage of S3 in spring of 2017 comes to mind. The AWS status page went down and no one could figure out what was going on because the status page had a hard dependency on us-east-1 S3.
  - cthalupa 8y ago
    To my knowledge, there has never been a global outage for any AWS service, and I've been using them for a long time. There was the status page, but S3 being down in us-east-1 didn't effect S3 in ap-southeast-1, etc. The big DynamoDB outage a few years back also was limited to us-east-1
- deathanatos 8y ago
  It's DNS, so it is somewhat inherently global. Route53 isn't region specific either, so I could see an issue with that having a global effect, too.
  - aioprisan 8y ago
    Sure, but that's hypothetical, and I don't recall AWS having any such issue in recent history.
  - y0y 8y ago
    DNS is also inherently distributed. This should make it resilient to all of the most common outage scenarios, and is likely why AWS offers a 100% uptime SLA for Route 53. I'll be interested in the post-mortem from Azure on this one.
    - el_duderino 8y ago
      Do they typically provide a postmortem?
      - crankylinuxuser 8y ago
        It's Microsoft. I'm sure they just rebooted it! (I had to, see username!) edit: seriously,-3 ? it was a joke.
    - deathanatos 8y ago
      > likely why AWS offers a 100% uptime SLA for Route 53 Well, that's interesting. We occasionally see getaddrinfo() calls fail claiming domains that we know exist at the failure time (b/c the records are completely static) don't exist. (We've not got a reproducible case for this yet, and it's incredibly rare for any given VM/service. But across our fleet, it crops up fairly regularly.)
      - cthalupa 8y ago
        > We occasionally see getaddrinfo() calls fail claiming domains that we know exist at the failure time (b/c the records are completely static) don't exist. That could be whatever resolvers you're hitting failing rather than an issue with Route 53 authoritative nameservers, though. The resolving DNS servers in EC2 are not actually part of Route 53, for example.
        deathanatos 8y ago
        I'd think that would correspond to EAI_AGAIN or EAI_FAIL, whereas I'm pretty sure we're getting a EAI_NONAME.
- zie 8y ago
  in _all_ regions, regularly, Probably not... Depends on how you define "regularly".. but even one or two a year could be counted as regular for some definition of regular. * Azure: 12 user-facing outages this year so far. source: https://azure.microsoft.com/en-us/status/history/ https://azure.microsoft.com/en-us/status/history/ * Amazon: They don't make their status history public it seems, but I see 5 easily searchable outages from last year. I would venture to bet their actual outage history is not any better than Azure or Google. * Google: 23 from this year so far source: https://status.cloud.google.com/summary https://status.cloud.google.com/summary These numbers are from when I counted in March 2019, so ~ 3 months, their outages are in double digits. That's TERRIBLE. Obviously the above numbers are not across _all_ regions, they are just a total count across all regions.
  - cheeze 8y ago
    Eh, just counting outages doesn't do anything imo. Most providers consider something like < 99% availability for > 5 minutes an outage. Outages like that are expected in any single region and product. If I really need perfect availability (I probably don't), then I can leverage the multiple regions and build a failovers strategy. I do not expect global outages which affect multiple regions at the same time. That's just amateur hour when it is a common occurrence.
    - zie 8y ago
      Of course there are solutions to outages. Multiple regions, failover, multiple cloud providers, or just don't put all your eggs in $CLOUD, and do some on-prem as well, etc, etc, etc. There are many ways to handle outages. I'm just saying if you are averaging 4+ outages PER MONTH across all of your services, you are probably doing something wrong. These providers are on schedule to pass 50 outages this year! All the places I've ever worked, if we can count our customer facing outages per year past the number of planets humans live on(1) we consider ourselves a total failure that year and re-work our planning. We haven't had to re-work our plans in ~ 5 years. Our longest outage in the past 5 years? 1 hr long. Granted I'm not the scale of $CLOUD, but there is zero reason they can't get serious about uptime, they just don't bother. People are still flocking to $CLOUD in droves, despite their crappy uptimes. It's not rocket science to make stuff work, you just have to get Boring(tm). Where Boring here means using standard, boring, well understood technology that everyone knows and understands and not $NEWHOTNESS. $NEWHOTNESS is guaranteed to break in new and exciting ways.
420codebro 8y ago
Not globally. Most clouds architect each region as stand-alone. The logic being if a change that begins in region A causes an issue, you have a feedback-loop to decide to either move forward or to halt the roll-out to further regions.

All the big clouds regularly have outages.

by zie 8y ago

All the big clouds regularly have outages.

outworlder 8y ago

In _all_ regions? No.

NicoJuicy 8y ago
Amazon AWS had/has a dependency on one of their first datacenters last year for example
- syntheticcdo 8y ago
  The us-east-1 outage of S3 in spring of 2017 comes to mind. The AWS status page went down and no one could figure out what was going on because the status page had a hard dependency on us-east-1 S3.
- cthalupa 8y ago
  To my knowledge, there has never been a global outage for any AWS service, and I've been using them for a long time. There was the status page, but S3 being down in us-east-1 didn't effect S3 in ap-southeast-1, etc. The big DynamoDB outage a few years back also was limited to us-east-1
deathanatos 8y ago
It's DNS, so it is somewhat inherently global. Route53 isn't region specific either, so I could see an issue with that having a global effect, too.
- aioprisan 8y ago
  Sure, but that's hypothetical, and I don't recall AWS having any such issue in recent history.
- y0y 8y ago
  DNS is also inherently distributed. This should make it resilient to all of the most common outage scenarios, and is likely why AWS offers a 100% uptime SLA for Route 53. I'll be interested in the post-mortem from Azure on this one.
  - el_duderino 8y ago
    Do they typically provide a postmortem?
    - crankylinuxuser 8y ago
      It's Microsoft. I'm sure they just rebooted it! (I had to, see username!) edit: seriously,-3 ? it was a joke.
  - deathanatos 8y ago
    > likely why AWS offers a 100% uptime SLA for Route 53 Well, that's interesting. We occasionally see getaddrinfo() calls fail claiming domains that we know exist at the failure time (b/c the records are completely static) don't exist. (We've not got a reproducible case for this yet, and it's incredibly rare for any given VM/service. But across our fleet, it crops up fairly regularly.)
    - cthalupa 8y ago
      > We occasionally see getaddrinfo() calls fail claiming domains that we know exist at the failure time (b/c the records are completely static) don't exist. That could be whatever resolvers you're hitting failing rather than an issue with Route 53 authoritative nameservers, though. The resolving DNS servers in EC2 are not actually part of Route 53, for example.
      - deathanatos 8y ago
        I'd think that would correspond to EAI_AGAIN or EAI_FAIL, whereas I'm pretty sure we're getting a EAI_NONAME.
zie 8y ago
in _all_ regions, regularly, Probably not... Depends on how you define "regularly".. but even one or two a year could be counted as regular for some definition of regular. * Azure: 12 user-facing outages this year so far. source: https://azure.microsoft.com/en-us/status/history/ https://azure.microsoft.com/en-us/status/history/ * Amazon: They don't make their status history public it seems, but I see 5 easily searchable outages from last year. I would venture to bet their actual outage history is not any better than Azure or Google. * Google: 23 from this year so far source: https://status.cloud.google.com/summary https://status.cloud.google.com/summary These numbers are from when I counted in March 2019, so ~ 3 months, their outages are in double digits. That's TERRIBLE. Obviously the above numbers are not across _all_ regions, they are just a total count across all regions.
- cheeze 8y ago
  Eh, just counting outages doesn't do anything imo. Most providers consider something like < 99% availability for > 5 minutes an outage. Outages like that are expected in any single region and product. If I really need perfect availability (I probably don't), then I can leverage the multiple regions and build a failovers strategy. I do not expect global outages which affect multiple regions at the same time. That's just amateur hour when it is a common occurrence.
  - zie 8y ago
    Of course there are solutions to outages. Multiple regions, failover, multiple cloud providers, or just don't put all your eggs in $CLOUD, and do some on-prem as well, etc, etc, etc. There are many ways to handle outages. I'm just saying if you are averaging 4+ outages PER MONTH across all of your services, you are probably doing something wrong. These providers are on schedule to pass 50 outages this year! All the places I've ever worked, if we can count our customer facing outages per year past the number of planets humans live on(1) we consider ourselves a total failure that year and re-work our planning. We haven't had to re-work our plans in ~ 5 years. Our longest outage in the past 5 years? 1 hr long. Granted I'm not the scale of $CLOUD, but there is zero reason they can't get serious about uptime, they just don't bother. People are still flocking to $CLOUD in droves, despite their crappy uptimes. It's not rocket science to make stuff work, you just have to get Boring(tm). Where Boring here means using standard, boring, well understood technology that everyone knows and understands and not $NEWHOTNESS. $NEWHOTNESS is guaranteed to break in new and exciting ways.

420codebro 8y ago

Not globally. Most clouds architect each region as stand-alone. The logic being if a change that begins in region A causes an issue, you have a feedback-loop to decide to either move forward or to halt the roll-out to further regions.