10 ms·

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time th

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by their "green board" is meaningless. During the event they were late and reluctant to report it and its significance. My point is that they are wrongly incentivized to keep the board green at all costs.

tootie 5y ago
Second hand info but supposedly when an outage hits they go all hands on resolving it and no one who knows what's going on has time to update the status board which is why it's always behind.
- voidfunc 5y ago
  Not AWS, but Azure: highly doubt. At least at Azure the moment you declare an outage there is a incident manager to handle customer communication. Bullshit someone at Amazon doesn’t have time to update the status.
amalter 5y ago
I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.
- dijit 5y ago
  Once is a mistake. Twice is a coincidence. Three times is a pattern. But this… This is every time.
  - doctor_eval 5y ago
    Four times is a policy.
- s_dev 5y ago
  >You can take it on bad faith It's smart politics -- I don't blame them but I don't trust the dashboard either. There's established patterns now of the AWS dashboard being useless. If I want to check if Amazon is down I'm checking Twitter and HN. Not bad faith -- no faith.
  - sorry_outta_gas 5y ago
    That's only useful when it's an entire region, there are minor issues in smaller services that cause problems for a lot of people they don't reflect in their status board; and not everyone checks twitter or HN all the time while at work it's a bullshit board used fudge numbers when negoaiting SLAs like I don't care that much, hell my company does the same thing; but let's not get defensive over it
  - toss1 5y ago
    >>It's smart politics -- I don't blame them Um, so you think straight-up lying is good politics? Any 7-year old knows that telling a lie when you broke something makes you look better superficially, especially if you get away with it. That does not mean that we should think it is a good idea to tell lies when you break things. It sure as hell isn't smart politics in my book. It is straight-up disqualifying to do business with them. If they are not honest about the status or amount of service they are providing, how is that different than lying about your prices? Would you go to a petrol station that posted $x.00/gallon, but only delivered 3 quarts for each gallon shown on the pump? We're being shortchanged and lied to. Fascinating that you think it is good politics on their part.
    - efitz 5y ago
      You don’t know what you’re talking about. AWS spends a lot of time thinking about this problem in service to their customers. How do you reduce the status of millions of machines, the software they run, and the interconnected-ness of those systems to a single graphical indicator? It would be dumb and useless to turn something red every single time anything had a problem. Literally there are hundreds of things broken every minute of every day. On-call engineers are working around the clock on these problems. Most of the problems either don’t affect anyone due to redundancy or affect only a tiny number of customers- a failed memory module or top-of-rack switch or a random bit flip in one host for one service. Would it help anyone to tell everyone about all these problems? People would quickly learn to ignore it as it had no bearing on their experience. What you’re really arguing is that you don’t like the thresholds they’ve chosen. That’s fine, everyone has an opinion. The purpose of health dashboards like these are mostly so that customers can quickly get an answer to “is it them or me” when there’s a problem. As others on this thread have pointed out, AWS has done a pretty good job of making the SHD align with the subjective experience of most customers. They also have personal health dashboards unique to each customer, but I assume thresholding is still involved.
- systemvoltage 5y ago
  People of HN has been extremely unprofessional with regards to AWS's downtime. Some kind of a massive zeitgeist against Amazon, like a giant hive mind that spews hate. Why are we doing this folks? What's making you so angry and contemptful? Literally try searching the history of downtimes and it was always professional and respectful. Yesterday, my comment was fricking flagged for asking people to be nice to which people responded "Professionals recognize other professionals lying". Completely baseless and hate spewing comments like this is ruining HN.
  - NicoJuicy 5y ago
    I think the biggest issue is about the status dashboard that always stays green. I haven't seen much else, no? It seems that degraded seems down in most cases. Since authorization of managers is required.
  - jiggawatts 5y ago
    AWS as a business has an enormous (multi-billion-dollar) moral hazard: they have a fantastically strong disincentive to update their status dashboard to accurately reflect the true nature of an ongoing outage. They use weasel words like "some customers may be seeing elevated errors", which we all know translates to "almost all customers are seeing 99.99% failure rates." They have a strong incentive to lie, and they're doing it. This makes people dependent upon the truth for refunds understandably angry.
  - ProAm 5y ago
    > Why are we doing this folks? What's making you so angry and contemptful? Because Amazon kills industries. Takes job. They do this because they promise they hire the best people that can do this better than you and for cheaper. And it's rarely true. And then they lie about it when things hit the fan. If you're going to be the best you need to act like the best, and execute like the best. Not build a walled garden that people cant see into, and hard to leave.
  - edoceo 5y ago
    All too often folk conflate frustration with anger or hate. The comments are frustrated users. Not hateful.
- luhn 5y ago
  Off the top of my head, this is the third time they've had a major outage where they've been unable to properly update the status page. First we had the S3 outage, where the yellow and red icons were hosted in S3 and unable to be accessed. Second we had the Kinesis outage, which snowballed into a Cognito outage, so they were unable to login into the status page CMS. Now this. They "own up to it" in their postmortems, but after multiple failures they're still unwilling to implement the obvious solution and what is widely regarded as best practice: host the status page on a different platform.
  - koheripbal 5y ago
    This challenge is not specific to Amazon. Being able to automatically detect system health is a non-trivial effort.
    - bob778 5y ago
      That’s not what’s being asked though - in all 3 events, they couldn’t manually update it. It’s clearly not a priority to fix it for even manual alerts.
    - blackearl 5y ago
      Why automatic? Surely someone could have the responsibility to do it manually.
      - geenew 5y ago
        Or override the autogenerated values
    - jjoonathan 5y ago
      They had all day to do it manually.
    - saagarjha 5y ago
      As others mention, you can do it manually. But it’s also not that hard to do automatically: literally just spin up a “client” of your service and make sure it works.
- tw04 5y ago
  Multiple AWS employees have acknowledged it takes VP approval to change the status color of the dashboard. That is absurd and it tells you everything you need to know. The status page isn't about accurate information, it's about plausible deniability and keeping AWS out of the news cycle.
  - dylan604 5y ago
    >it's about plausible deniability and keeping AWS out of the news cycle. How'd that work out for them? https://duckduckgo.com/?q=AWS+outage+news+coverage&t=h_&ia=web https://duckduckgo.com/?q=AWS+outage+news+coverage&t=h_&ia=w...
    - tw04 5y ago
      When is the last time they had a single service outage in a single region? How about in a single AZ in a single region? Struggling to find a lot of headline stories? I'm willing to bet it's happened in the last 2 years and yet I don't see many news articles about it... so I'd say if the only thing that hits the front page is a complete region outage for 6+ hours, it's working out pretty well for them.
      - grumple 5y ago
        Last year's Thanksgiving outage and this one are the two biggest. They've been pretty reliable. That's still 99.7% uptime.
  - cookie_monsta 5y ago
    I am so naive. I honestly thought those things were automated.
- discodave 5y ago
  The AWS summary says: "As the impact to services during this event all stemmed from a single root cause, we opted to provide updates via a global banner on the Service Health Dashboard, which we have since learned makes it difficult for some customers to find information about this issue" This seems like bad faith to me based on my experience when I worked for AWS. As they repeated many times at Re:Invent last week, they've been doing this for 15+ years. I distinctly remember seeing banners like "Don't update the dashboard without approval from <importantSVP>" on various service team runbooks. They tried not to say it out loud, but there was very much a top-down mandate for service teams to make the dashboard "look green" by: 1. Actually improving availability (this one is fair). 2. Using the "Green-I" icon rather than the blue, orange, or red icons whenever possible. 3. They built out the "Personal Health Dashboard" so they can post about many issues in there, without having to acknowledge it publicly.
  - res0nat0r 5y ago
    Eh I mean at least when DeSantis was lower on the food chain then he is now, the normal directive was that ec2 status wasn't updated unless a certain X percent of hosts were affected. Which is reasonable because a single rack going down isn't relevant enough to constitute a massive problem with ec2 as a whole.
- ProAm 5y ago
  > You can take it on bad faith, but they own up to the dashboard being completely useless during the incident. Let's not act like this is the first time this has happened. It's bad faith that they do not change when their promise is they hire the best to handle infrastructure so you don't have to. It's clearly not the case. Between this and billing I we can easily lay blame and acknowledge lies.
- nwallin 5y ago
  So -- ctrl-f "Dash" only produces four results and it's hidden away in the bottom of the page. It's false to claim that even 20% of the post mortem is addressing the failure of the dashboard. The problem is that the dashboard requires VP approval to be updated. Which is broken. The dashboard should be automatic. The dashboard should update before even a single member of the AWS team knows there's something wrong.
  - hunter2_ 5y ago
    Is it typical for orgs (the whole spectrum: IT departments everywhere, telecom, SaaS, maybe even status of non-technical services) to have automatic downtime messaging that doesn't need a human set of eyes to approve it first?
Clubber 5y ago
Yes, it's a conflict of interest. They have a guarantee on uptime and they decide what their actual uptime is. There's a lot of that now. Most insurances comes to mind.
deleted 5y ago
[deleted]
notimetorelax 5y ago
If you were deployed in 2 regions would it alleviate the impact?
- ransom1538 5y ago
  Yes. Exactly. Pay double. That is what all the blogs say. But no, when a region goes down everything is hosed. Give it a shot! Next time an entire region is down try out your apis or give AWS support a call.
- hvgk 5y ago
  No. We don't have an active deployment in that region at all. It killed our build pipeline as ECR was down globally so we had nowhere to push images. Also there was a massive risk as our target environments are EKS so any node failures or scaling events had nowhere to pull images from while ECR was down. Edit: not to mention APIGW and Cloudwatch APIs were down too.
- multipassnetwrk 5y ago
  Depends. If your failover to another region required changing DNS and your DNS was using Route 53, you would have problems.
electroly 5y ago
> The entire time their outage board was solid green Unless you're talking about some board other than the Service Health Dashboard, this isn't true. They dropped EC2 down to degraded pretty early on. I bemusedly noted in our corporate Slack that every time I refreshed the SHD, another service was listed as degraded. Then they added the giant banner at the top. Their slight delay in updating the SHD at the beginning of the outage is mentioned in the article. It was absolutely not all green for the duration of the outage.
- logical_proof 5y ago
  That is not true. There was hours before they started annotating any kind of service issues. Maybe from when you noticed there was a problem it appeared to be quick, but the board remained green for a large portion of the outtage.
  - acdha 5y ago
    We saw the timing described where the dashboard updates started about an hour after the problem began (which we noticed immediately since 7:30AM Pacific is in the middle of the day for those of us in Eastern time). I don't know if there was an issue with browser caching or similar but once the updates started everyone here had no trouble seeing them and my RSS feed monitor picked them up around that time as well.
  - electroly 5y ago
    No, it was about an hour. We were aware from the very moment EC2 API error rates began to elevate, around 10:30 Eastern. By 11:30 the dashboard was updating. This timing is mentioned in the article, and it all happened in the middle of our workday on the east coast. The outage then continued for about 7 hours with SHD updates. I suspect we actually both agree on how long it took them to start updating, but I conclude that 1 hour wasn't so bad.
    - gkop 5y ago
      At the large platform company where I work, our policy is if the customer reported the issue before our internal monitoring caught it, we have failed. Give 5 minutes for alerting lag, 10 minutes to evaluate the magnitude of impact, 10 minutes to craft the content and get it approved, 5 minutes to execute the update, adds up to 30 minutes end to end with healthy buffer at each step. 1 hour (52 minutes according to the article) sounds meh. I wonder what their error rate and latency graphs look like from that day.
ALittleLight 5y ago
I worked at Amazon. While my boss was on vacation I took over for him in the "Launch readiness" meeting for our team's component of our project. Basically, you go to this meeting with the big decision makers and business people once a week and tell them what your status is on deliverables. You are supposed to sum up your status as "Green/Yellow/Red" and then write (or update last week's document) to explain your status. My boss had not given me any special directions here so I assumed I was supposed to do this honestly. I set our status as "Red" and then listed out what were, I felt, quite compelling reasons to think we were Red. The gist of it was that our velocity was negative. More work items were getting created and assigned to us than we closed, and we still had high priority items open from previous dates. There was zero chance, in my estimation, that we would meet our deadlines, so I called us Red. This did not go over well. Everyone at the Launch Readiness meeting got mad at me for declaring Red. Our VP scolded me in front of the entire meeting and lectured me about how I could not unilaterally declare our team red. Her logic was, if our team was Red, that meant the entire project was Red, and I was in no position to make that call. Other managers at the meeting got mad at me too because they felt my call made them look bad. For the rest of my manager's absence I had to first check in with a different manager and show him my Launch Readiness status and get him to approve my update before I was allowed to show it to the rest of the group. For the rest of the time that I went to Launch Readiness I was forbidden from declaring Red regardless of what our metrics said. Our team was Yellow or Green, period. Naturally, we wound up being over a year late on the deadlines, because, despite what they compelled us to say in those meetings, we weren't actually getting the needed work done. Constant "schedule slips" and adjustments. Endless wasted time in meetings trying to rework schedules that would instantly get blown up again. Hugely frustrating. Still slightly bitter about it. Anyway, I guess all this is to say that it doesn't surprise me that Amazon is bad about declaring Red, Yellow, or Green in other places too. Probably there is a guy in charge of updating those dashboards who is forbidden from changing them unless they get approval from some high level person and that person will categorically refuse regardless of the evidence because they want the indicators to be Green.
- transcriptase 5y ago
  This explicitly supports what most of us assume is going on. I wont be surprised if someone with a (un)vested interest will be along shortly to say that their experience is the opposite and that on their team, making people look bad by telling the truth is expected and praised.
hvgk 5y ago
This. We're under NDA too on internal support. Our customers know we use AWS and they go and check the AWS status dashboards and tell us there's nothing wrong so the inevitable vitriol is always directed at us which we then have to defend.
- macintux 5y ago
  I guess you have to hope that every outage that impacts you is big enough to make the news.
SkyPuncher 5y ago
My company isn't big enough for us to have any pull but this communication is _significantly_ downplaying the impact of this issue. One of our auxiliary services that's basically a pass through to AWS was offline nearly the entire day. Yet, this communication doesn't even mention that fact. In fact, it almost tries to suggest the opposite. Likewise, AWS is reporting S3 didn't have issues. Yet, for a period of time, S3 was erroring out frequently because it was responding so slowly.
jiggawatts 5y ago
SLAs with self-reported outage periods are worthless. SLAs that refund only the cost of the individual service that was down is worthless. SLAs that require separate proof and refund requests for each and every service that was affected are nearly worthless. There needs to be an independent body set up by a large cloud customers to monitor availability and enforce refunds.
codegeek 5y ago
"Our Support Contact Center also relies on the internal AWS network, so the ability to create support cases was impacted from 7:33 AM until 2:25 PM PST. " This to me is really bad. Even as a small company, we keep our support infrastructure separate. For a company of Amazon's size, this is a shitty excuse. If I cannot even reach you as a customer for almost 7 hours, that is just nuts. AWS must do better here. Also, is it true that the outage/status pages are manually updated ? If yes, there is no excuse why it was green for that long. If you are manually updating it, please update asap.
- anonu 5y ago
  Wasn't this the Bezos directive early on that created AWS? Anything that was created had to be a service with an API. Not allowed to recreate the wheel. So AWS depends on AWS.
  - jiggawatts 5y ago
    Dependency loops are such fun! My favourite is when some company migrates their physical servers to virtual machines, including the AD domain controllers. Then the next step is to use AD LDAP authentication for the VM management software. When there's a temporary outage and the VMs don't start up as expected, the admins can't log on and troubleshoot the platform because the logon system was running on it... but isn't now. The loop is closed. You see this all the time, especially with system-management software. They become dependent on the systems they're managing, and vice-versa. If you care about availability at all, make sure to have physical servers providing basic services like DNS, NTP, LDAP, RADIUS, etc...
    - nijave 5y ago
      Or even just have some non-federated/"local" accounts stored in a vault somewhere you can use when the centralized auth isn't working
- walrus01 5y ago
  I know a few tiny ISPs that host their voip server and email server outside of their own ASN so that in the event of a catastrophic network event, communications with customers is still possible... Not saying amazon should do the same, but the general principle isn't rocket science. there's such as thing as too much dogfooding.
soheil 5y ago
This was addressed at least 3 times during this post. I'm not defending them but you're just gaslighting. If you have something to add about the points they raised regarding the status page please do so.
codeduck 5y ago
carbon copy of our experience.
notyourday 5y ago
> The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. if ($pain > $gain) { move_your_shit_and_exit_aws(); } sub move_your_shit_and_exit_aws { printf("Dude. We have too much pain. Start moving\n"); printf("Yeah. That won't happen, so who cares\n"); exit(1); }
- secondcoming 5y ago
  Moving your shit from AWS can be really expensive, depending on how much shit you have. If you're nice, GCP may subsidise - or even cover - the costs!
tuananh 5y ago
even in the post mortem, they are reclutant to admit it > While AWS customer workloads were not directly impacted from the internal networking issues described above, the networking issues caused impact to a number of AWS Services which in turn impacted customers using these service capabilities. Because the main AWS network was not affected, some customer applications which did not rely on these capabilities only experienced minimal impact from this event.
- nijave 5y ago
  >some customer applications which did not rely on these capabilities only experienced minimal impact from this event Yeah so vanilla LB and EC2 with no autoscaling were fine. Anyone using "serverless" or managed services had a real bad day
eranation 5y ago
Obligatory mention to https://stop.lying.cloud https://stop.lying.cloud
jedberg 5y ago
Honestly, the should host that status page on CloudFlare or some completely separate infrastructure that they maintain in colo datacenters or something. The only time it really needs to be up is when their stuff isn't working.
steveBK123 5y ago
Exactly, we had the same thing almost exactly a year ago - https://www.dailymail.co.uk/sciencetech/article-8994907/Widespread-Amazon-cloud-service-outage-disables-Roombas-Ring-doorbells-Christmas-lights.html https://www.dailymail.co.uk/sciencetech/article-8994907/Wide... They are barely doing better than 2 9s.

tootie 5y ago

Second hand info but supposedly when an outage hits they go all hands on resolving it and no one who knows what's going on has time to update the status board which is why it's always behind.

voidfunc 5y ago
Not AWS, but Azure: highly doubt. At least at Azure the moment you declare an outage there is a incident manager to handle customer communication. Bullshit someone at Amazon doesn’t have time to update the status.

amalter 5y ago

I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.

dijit 5y ago
Once is a mistake. Twice is a coincidence. Three times is a pattern. But this… This is every time.
- doctor_eval 5y ago
  Four times is a policy.
s_dev 5y ago
>You can take it on bad faith It's smart politics -- I don't blame them but I don't trust the dashboard either. There's established patterns now of the AWS dashboard being useless. If I want to check if Amazon is down I'm checking Twitter and HN. Not bad faith -- no faith.
- sorry_outta_gas 5y ago
  That's only useful when it's an entire region, there are minor issues in smaller services that cause problems for a lot of people they don't reflect in their status board; and not everyone checks twitter or HN all the time while at work it's a bullshit board used fudge numbers when negoaiting SLAs like I don't care that much, hell my company does the same thing; but let's not get defensive over it
- toss1 5y ago
  >>It's smart politics -- I don't blame them Um, so you think straight-up lying is good politics? Any 7-year old knows that telling a lie when you broke something makes you look better superficially, especially if you get away with it. That does not mean that we should think it is a good idea to tell lies when you break things. It sure as hell isn't smart politics in my book. It is straight-up disqualifying to do business with them. If they are not honest about the status or amount of service they are providing, how is that different than lying about your prices? Would you go to a petrol station that posted $x.00/gallon, but only delivered 3 quarts for each gallon shown on the pump? We're being shortchanged and lied to. Fascinating that you think it is good politics on their part.
  - efitz 5y ago
    You don’t know what you’re talking about. AWS spends a lot of time thinking about this problem in service to their customers. How do you reduce the status of millions of machines, the software they run, and the interconnected-ness of those systems to a single graphical indicator? It would be dumb and useless to turn something red every single time anything had a problem. Literally there are hundreds of things broken every minute of every day. On-call engineers are working around the clock on these problems. Most of the problems either don’t affect anyone due to redundancy or affect only a tiny number of customers- a failed memory module or top-of-rack switch or a random bit flip in one host for one service. Would it help anyone to tell everyone about all these problems? People would quickly learn to ignore it as it had no bearing on their experience. What you’re really arguing is that you don’t like the thresholds they’ve chosen. That’s fine, everyone has an opinion. The purpose of health dashboards like these are mostly so that customers can quickly get an answer to “is it them or me” when there’s a problem. As others on this thread have pointed out, AWS has done a pretty good job of making the SHD align with the subjective experience of most customers. They also have personal health dashboards unique to each customer, but I assume thresholding is still involved.
systemvoltage 5y ago
People of HN has been extremely unprofessional with regards to AWS's downtime. Some kind of a massive zeitgeist against Amazon, like a giant hive mind that spews hate. Why are we doing this folks? What's making you so angry and contemptful? Literally try searching the history of downtimes and it was always professional and respectful. Yesterday, my comment was fricking flagged for asking people to be nice to which people responded "Professionals recognize other professionals lying". Completely baseless and hate spewing comments like this is ruining HN.
- NicoJuicy 5y ago
  I think the biggest issue is about the status dashboard that always stays green. I haven't seen much else, no? It seems that degraded seems down in most cases. Since authorization of managers is required.
- jiggawatts 5y ago
  AWS as a business has an enormous (multi-billion-dollar) moral hazard: they have a fantastically strong disincentive to update their status dashboard to accurately reflect the true nature of an ongoing outage. They use weasel words like "some customers may be seeing elevated errors", which we all know translates to "almost all customers are seeing 99.99% failure rates." They have a strong incentive to lie, and they're doing it. This makes people dependent upon the truth for refunds understandably angry.
- ProAm 5y ago
  > Why are we doing this folks? What's making you so angry and contemptful? Because Amazon kills industries. Takes job. They do this because they promise they hire the best people that can do this better than you and for cheaper. And it's rarely true. And then they lie about it when things hit the fan. If you're going to be the best you need to act like the best, and execute like the best. Not build a walled garden that people cant see into, and hard to leave.
- edoceo 5y ago
  All too often folk conflate frustration with anger or hate. The comments are frustrated users. Not hateful.
luhn 5y ago
Off the top of my head, this is the third time they've had a major outage where they've been unable to properly update the status page. First we had the S3 outage, where the yellow and red icons were hosted in S3 and unable to be accessed. Second we had the Kinesis outage, which snowballed into a Cognito outage, so they were unable to login into the status page CMS. Now this. They "own up to it" in their postmortems, but after multiple failures they're still unwilling to implement the obvious solution and what is widely regarded as best practice: host the status page on a different platform.
- koheripbal 5y ago
  This challenge is not specific to Amazon. Being able to automatically detect system health is a non-trivial effort.
  - bob778 5y ago
    That’s not what’s being asked though - in all 3 events, they couldn’t manually update it. It’s clearly not a priority to fix it for even manual alerts.
  - blackearl 5y ago
    Why automatic? Surely someone could have the responsibility to do it manually.
    - geenew 5y ago
      Or override the autogenerated values
  - jjoonathan 5y ago
    They had all day to do it manually.
  - saagarjha 5y ago
    As others mention, you can do it manually. But it’s also not that hard to do automatically: literally just spin up a “client” of your service and make sure it works.
tw04 5y ago
Multiple AWS employees have acknowledged it takes VP approval to change the status color of the dashboard. That is absurd and it tells you everything you need to know. The status page isn't about accurate information, it's about plausible deniability and keeping AWS out of the news cycle.
- dylan604 5y ago
  >it's about plausible deniability and keeping AWS out of the news cycle. How'd that work out for them? https://duckduckgo.com/?q=AWS+outage+news+coverage&t=h_&ia=web https://duckduckgo.com/?q=AWS+outage+news+coverage&t=h_&ia=w...
  - tw04 5y ago
    When is the last time they had a single service outage in a single region? How about in a single AZ in a single region? Struggling to find a lot of headline stories? I'm willing to bet it's happened in the last 2 years and yet I don't see many news articles about it... so I'd say if the only thing that hits the front page is a complete region outage for 6+ hours, it's working out pretty well for them.
    - grumple 5y ago
      Last year's Thanksgiving outage and this one are the two biggest. They've been pretty reliable. That's still 99.7% uptime.
- cookie_monsta 5y ago
  I am so naive. I honestly thought those things were automated.
discodave 5y ago
The AWS summary says: "As the impact to services during this event all stemmed from a single root cause, we opted to provide updates via a global banner on the Service Health Dashboard, which we have since learned makes it difficult for some customers to find information about this issue" This seems like bad faith to me based on my experience when I worked for AWS. As they repeated many times at Re:Invent last week, they've been doing this for 15+ years. I distinctly remember seeing banners like "Don't update the dashboard without approval from <importantSVP>" on various service team runbooks. They tried not to say it out loud, but there was very much a top-down mandate for service teams to make the dashboard "look green" by: 1. Actually improving availability (this one is fair). 2. Using the "Green-I" icon rather than the blue, orange, or red icons whenever possible. 3. They built out the "Personal Health Dashboard" so they can post about many issues in there, without having to acknowledge it publicly.
- res0nat0r 5y ago
  Eh I mean at least when DeSantis was lower on the food chain then he is now, the normal directive was that ec2 status wasn't updated unless a certain X percent of hosts were affected. Which is reasonable because a single rack going down isn't relevant enough to constitute a massive problem with ec2 as a whole.
ProAm 5y ago
> You can take it on bad faith, but they own up to the dashboard being completely useless during the incident. Let's not act like this is the first time this has happened. It's bad faith that they do not change when their promise is they hire the best to handle infrastructure so you don't have to. It's clearly not the case. Between this and billing I we can easily lay blame and acknowledge lies.
nwallin 5y ago
So -- ctrl-f "Dash" only produces four results and it's hidden away in the bottom of the page. It's false to claim that even 20% of the post mortem is addressing the failure of the dashboard. The problem is that the dashboard requires VP approval to be updated. Which is broken. The dashboard should be automatic. The dashboard should update before even a single member of the AWS team knows there's something wrong.
- hunter2_ 5y ago
  Is it typical for orgs (the whole spectrum: IT departments everywhere, telecom, SaaS, maybe even status of non-technical services) to have automatic downtime messaging that doesn't need a human set of eyes to approve it first?

Clubber 5y ago

Yes, it's a conflict of interest. They have a guarantee on uptime and they decide what their actual uptime is. There's a lot of that now. Most insurances comes to mind.

deleted 5y ago

[deleted]

notimetorelax 5y ago

If you were deployed in 2 regions would it alleviate the impact?

ransom1538 5y ago
Yes. Exactly. Pay double. That is what all the blogs say. But no, when a region goes down everything is hosed. Give it a shot! Next time an entire region is down try out your apis or give AWS support a call.
hvgk 5y ago
No. We don't have an active deployment in that region at all. It killed our build pipeline as ECR was down globally so we had nowhere to push images. Also there was a massive risk as our target environments are EKS so any node failures or scaling events had nowhere to pull images from while ECR was down. Edit: not to mention APIGW and Cloudwatch APIs were down too.
multipassnetwrk 5y ago
Depends. If your failover to another region required changing DNS and your DNS was using Route 53, you would have problems.

electroly 5y ago

> The entire time their outage board was solid green Unless you're talking about some board other than the Service Health Dashboard, this isn't true. They dropped EC2 down to degraded pretty early on. I bemusedly noted in our corporate Slack that every time I refreshed the SHD, another service was listed as degraded. Then they added the giant banner at the top. Their slight delay in updating the SHD at the beginning of the outage is mentioned in the article. It was absolutely not all green for the duration of the outage.

logical_proof 5y ago
That is not true. There was hours before they started annotating any kind of service issues. Maybe from when you noticed there was a problem it appeared to be quick, but the board remained green for a large portion of the outtage.
- acdha 5y ago
  We saw the timing described where the dashboard updates started about an hour after the problem began (which we noticed immediately since 7:30AM Pacific is in the middle of the day for those of us in Eastern time). I don't know if there was an issue with browser caching or similar but once the updates started everyone here had no trouble seeing them and my RSS feed monitor picked them up around that time as well.
- electroly 5y ago
  No, it was about an hour. We were aware from the very moment EC2 API error rates began to elevate, around 10:30 Eastern. By 11:30 the dashboard was updating. This timing is mentioned in the article, and it all happened in the middle of our workday on the east coast. The outage then continued for about 7 hours with SHD updates. I suspect we actually both agree on how long it took them to start updating, but I conclude that 1 hour wasn't so bad.
  - gkop 5y ago
    At the large platform company where I work, our policy is if the customer reported the issue before our internal monitoring caught it, we have failed. Give 5 minutes for alerting lag, 10 minutes to evaluate the magnitude of impact, 10 minutes to craft the content and get it approved, 5 minutes to execute the update, adds up to 30 minutes end to end with healthy buffer at each step. 1 hour (52 minutes according to the article) sounds meh. I wonder what their error rate and latency graphs look like from that day.

ALittleLight 5y ago

I worked at Amazon. While my boss was on vacation I took over for him in the "Launch readiness" meeting for our team's component of our project. Basically, you go to this meeting with the big decision makers and business people once a week and tell them what your status is on deliverables. You are supposed to sum up your status as "Green/Yellow/Red" and then write (or update last week's document) to explain your status. My boss had not given me any special directions here so I assumed I was supposed to do this honestly. I set our status as "Red" and then listed out what were, I felt, quite compelling reasons to think we were Red. The gist of it was that our velocity was negative. More work items were getting created and assigned to us than we closed, and we still had high priority items open from previous dates. There was zero chance, in my estimation, that we would meet our deadlines, so I called us Red. This did not go over well. Everyone at the Launch Readiness meeting got mad at me for declaring Red. Our VP scolded me in front of the entire meeting and lectured me about how I could not unilaterally declare our team red. Her logic was, if our team was Red, that meant the entire project was Red, and I was in no position to make that call. Other managers at the meeting got mad at me too because they felt my call made them look bad. For the rest of my manager's absence I had to first check in with a different manager and show him my Launch Readiness status and get him to approve my update before I was allowed to show it to the rest of the group. For the rest of the time that I went to Launch Readiness I was forbidden from declaring Red regardless of what our metrics said. Our team was Yellow or Green, period. Naturally, we wound up being over a year late on the deadlines, because, despite what they compelled us to say in those meetings, we weren't actually getting the needed work done. Constant "schedule slips" and adjustments. Endless wasted time in meetings trying to rework schedules that would instantly get blown up again. Hugely frustrating. Still slightly bitter about it. Anyway, I guess all this is to say that it doesn't surprise me that Amazon is bad about declaring Red, Yellow, or Green in other places too. Probably there is a guy in charge of updating those dashboards who is forbidden from changing them unless they get approval from some high level person and that person will categorically refuse regardless of the evidence because they want the indicators to be Green.

transcriptase 5y ago
This explicitly supports what most of us assume is going on. I wont be surprised if someone with a (un)vested interest will be along shortly to say that their experience is the opposite and that on their team, making people look bad by telling the truth is expected and praised.

hvgk 5y ago

This. We're under NDA too on internal support. Our customers know we use AWS and they go and check the AWS status dashboards and tell us there's nothing wrong so the inevitable vitriol is always directed at us which we then have to defend.

macintux 5y ago
I guess you have to hope that every outage that impacts you is big enough to make the news.

SkyPuncher 5y ago

My company isn't big enough for us to have any pull but this communication is _significantly_ downplaying the impact of this issue. One of our auxiliary services that's basically a pass through to AWS was offline nearly the entire day. Yet, this communication doesn't even mention that fact. In fact, it almost tries to suggest the opposite. Likewise, AWS is reporting S3 didn't have issues. Yet, for a period of time, S3 was erroring out frequently because it was responding so slowly.

jiggawatts 5y ago

SLAs with self-reported outage periods are worthless. SLAs that refund only the cost of the individual service that was down is worthless. SLAs that require separate proof and refund requests for each and every service that was affected are nearly worthless. There needs to be an independent body set up by a large cloud customers to monitor availability and enforce refunds.

codegeek 5y ago

"Our Support Contact Center also relies on the internal AWS network, so the ability to create support cases was impacted from 7:33 AM until 2:25 PM PST. " This to me is really bad. Even as a small company, we keep our support infrastructure separate. For a company of Amazon's size, this is a shitty excuse. If I cannot even reach you as a customer for almost 7 hours, that is just nuts. AWS must do better here. Also, is it true that the outage/status pages are manually updated ? If yes, there is no excuse why it was green for that long. If you are manually updating it, please update asap.

anonu 5y ago
Wasn't this the Bezos directive early on that created AWS? Anything that was created had to be a service with an API. Not allowed to recreate the wheel. So AWS depends on AWS.
- jiggawatts 5y ago
  Dependency loops are such fun! My favourite is when some company migrates their physical servers to virtual machines, including the AD domain controllers. Then the next step is to use AD LDAP authentication for the VM management software. When there's a temporary outage and the VMs don't start up as expected, the admins can't log on and troubleshoot the platform because the logon system was running on it... but isn't now. The loop is closed. You see this all the time, especially with system-management software. They become dependent on the systems they're managing, and vice-versa. If you care about availability at all, make sure to have physical servers providing basic services like DNS, NTP, LDAP, RADIUS, etc...
  - nijave 5y ago
    Or even just have some non-federated/"local" accounts stored in a vault somewhere you can use when the centralized auth isn't working
walrus01 5y ago
I know a few tiny ISPs that host their voip server and email server outside of their own ASN so that in the event of a catastrophic network event, communications with customers is still possible... Not saying amazon should do the same, but the general principle isn't rocket science. there's such as thing as too much dogfooding.

soheil 5y ago

This was addressed at least 3 times during this post. I'm not defending them but you're just gaslighting. If you have something to add about the points they raised regarding the status page please do so.

codeduck 5y ago

carbon copy of our experience.

notyourday 5y ago

> The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. if ($pain > $gain) { move_your_shit_and_exit_aws(); } sub move_your_shit_and_exit_aws { printf("Dude. We have too much pain. Start moving\n"); printf("Yeah. That won't happen, so who cares\n"); exit(1); }

secondcoming 5y ago
Moving your shit from AWS can be really expensive, depending on how much shit you have. If you're nice, GCP may subsidise - or even cover - the costs!

tuananh 5y ago

even in the post mortem, they are reclutant to admit it > While AWS customer workloads were not directly impacted from the internal networking issues described above, the networking issues caused impact to a number of AWS Services which in turn impacted customers using these service capabilities. Because the main AWS network was not affected, some customer applications which did not rely on these capabilities only experienced minimal impact from this event.

nijave 5y ago
>some customer applications which did not rely on these capabilities only experienced minimal impact from this event Yeah so vanilla LB and EC2 with no autoscaling were fine. Anyone using "serverless" or managed services had a real bad day

eranation 5y ago

Obligatory mention to https://stop.lying.cloud https://stop.lying.cloud

jedberg 5y ago

Honestly, the should host that status page on CloudFlare or some completely separate infrastructure that they maintain in colo datacenters or something. The only time it really needs to be up is when their stuff isn't working.

steveBK123 5y ago

Exactly, we had the same thing almost exactly a year ago - https://www.dailymail.co.uk/sciencetech/article-8994907/Widespread-Amazon-cloud-service-outage-disables-Roombas-Ring-doorbells-Christmas-lights.html https://www.dailymail.co.uk/sciencetech/article-8994907/Wide... They are barely doing better than 2 9s.