21 ms·

Summary of the Amazon DynamoDB Service Disruption

190 points by spew 11y ago

userbinator 11y ago
Unavailable servers continued to retry requests for membership data, maintaining high load on the metadata service. It sounds like the retries exacerbated the situation. I wonder if they use exponential backoff?
- BillinghamJ 11y ago
  It's amazing how frequently this seems to happen. A lot of the huge downtime events I've read about have occurred because of failures in the systems used to recover from failures. It seems that perhaps more time/effort needs to be spent testing the systems (and the use of those systems) which are critical for handling failure. While building them, it's easy to dismiss their scaling requirements on the basis that they shouldn't ever be under significant load.
  - antirez 11y ago
    This is a classic, even because those software parts usually are the least tested in a variety of conditions, even when they are relatively well tested. It's very hard to simulate real failures.
- bbrazil 11y ago
  > I wonder if they use exponential backoff? More importantly, did they use randomised exponential backoff? Having all the retires hitting at the same time can lead to a pulses of outages until things settle down.
  - jmason23 11y ago
    I have no doubt they would have been using randomised exponential backoff -- these issues are well recognised inside Amazon and best practices are well known. For example, here's a blog post from March this year from Marc Brooker on the topic: http://www.awsarchitectureblog.com/2015/03/backoff.html http://www.awsarchitectureblog.com/2015/03/backoff.html . It may not have been correctly tuned for this scenario however. I'd say the Dynamo team are well aware of what they should have been doing, and kicking themselves for not foreseeing this cascading-failure case. ouch!
- allengeorge 11y ago
  Most exponential backoff implementations I've seen have an upper limit, where the initial (fast) retries work around transient networking blips. My guess is that a case like this you'd hit the backoff's upper bound pretty quickly, and, given the large volume of servers hitting a comparatively small metadata pool, experience exactly the same failure mode.
arturhoo 11y ago
We did not have detailed enough monitoring for this dimension (membership size), and didn’t have enough capacity allocated to the metadata service to handle these much heavier requests. As much as I admire and rely on AWS' scale to build architectures and fault tolerant applications, it can't be ignored that the marketing towards going "full cloud" doesn't take into account how hard it is to build resilient architectures in the cloud. I see those disruptions events as stop signs: when the cloud itself fails to scale, I rethink a few decisions we all make when surfing those trends. http://yourdatafitsinram.com/ http://yourdatafitsinram.com/ also comes to mind.
- chillydawg 11y ago
  How is it any harder to do in the cloud than on a rack in a warehouse? At least you don't have to muck about with cables and phoning power companies up.
  - bbrazil 11y ago
    Given they had ~300 minutes of outage in 3 years, you're looking at ~99.98% reliable in just that region. That's pretty good for a stateful serving system, and indeed you'd be pushed to do better.
    - pfortuny 11y ago
      That is just time not traffic loss, it would be hard to get so much traffic lost by a single in-house service. Metrics depend.
  - antirez 11y ago
    Just an example: during the issue even people serving 10 ops/sec, but very important 10 ops/sec, were affected by a huge complessive load which was not their for most of the part. It's true that when you "go cloud" you don't have to manage your operations, but you are basically putting everything in the hands of other op people, and what happens to you is related to a more wide set of conditions. So managing your stuff is hard, but you are in control and can do things in a way you believe is completely safe for you. Or you at least may incur in the same events sometimes, but perhaps paying a lot less for the same services. Or you can create your deployment with characteristics which are often impossible (a lot of RAM for each server is an example) to be cost effective in the cloud. It's not stupid to use AWS services but is not stupid to manage your operations, either in your own hardware or at least using just bare metal and/or the virtual machines service certain providers give you, but still being in part accountable, responsabile, and in control, of your system software deployment and operations.
deskamess 11y ago
> but we should have posted the green-i to the dashboard sooner than we did Use full orange or even half orange circle or something else if you want to convey 'subset of customers may have issues'. If possible move 'having issue' items to the top - this way people do not have to scroll down to see whats broken.
- zwily 11y ago
  This was the first time we had actually seen red icons on the dashboard. We joked that they had to furiously photoshop them up Sunday morning. (I'm sure it had happened before, just the first time we had seen them.)
- deanCommie 11y ago
  Not enough granularity with a yellow icon. I feel like if only <1% of customers are affected, a yellow circle isn't warranted - that's for the 5-20% impact case
  - takeda 11y ago
    Unless you're one of the <1% of customers. The whole point of status page is to to help determine if the issue you're experiencing is on your side or the SaaS, and not to show as much green as possible. The way currently AWS status page works it simply fails to provide any functionality and might as well be shut down. The colors on it start to change when you already see tons of articles about AWS being down.
ghshephard 11y ago
A pretty good post mortem, but one worrisome final comment: "For us, availability is the most important feature of DynamoDB" I would think that durability should be the most important feature of DynamoDB; better to have intermittent outage, or reduced capacity, but not have data loss - but perhaps there is something about DynamoDB which suggests availability is rated more highly than durability?
- manuelflara 11y ago
  They probably said that because this was an availability outage. If this was a postmortem after some data had been lost, they would've ended it with "For us, durability is the most important feature of DynamoDB".
- stingraycharles 11y ago
  I think this is written from an ops point of view. Dynamo-based systems allow a client to wait until at least X servers have acknowledged the write, and use eventual consistency in case there is some service disruption. I personally didnt know they used meta data tables in Dynamo, and that sounds like they lost one of the most important traits which gave Dynamo its super-high availability abilities, which is having all the cluster state in the clients rather than in the servers.
  - sillypog 11y ago
    Thank you for saying that! I just read the Dynamo paper the other day and that was the first thing that jumped out at me when I read this outage description. Now I feel like I understood the paper.
    - zackelan 11y ago
      Despite the name, DynamoDB is not actually an implementation of the Dynamo paper. It's influenced by those ideas, as well as the issues with scaling SimpleDB, but other than that is a from-scratch implementation. http://www.allthingsdistributed.com/2012/01/amazon-dynamodb.html http://www.allthingsdistributed.com/2012/01/amazon-dynamodb....
beambot 11y ago
Looks like a fair number of key aws systems rely on DynamoDB -- and further, the same system used by customers. I wonder: do they have any inclination to decouple these dependencies to prevent correlated outages?
- Jgrubb 11y ago
  I don't think they rely on Dynamo, they rely on an internal metadata service that Dynamo just happened to overwhelm with too many large requests.
  - beambot 11y ago
    From the 2nd sentence: > ... subsequent impact to other AWS services that depend on DynamoDB ... And the metadata service is part of DynamoDB: > The membership of a set of table/partitions within a server is managed by DynamoDB’s internal metadata service.
    - ckozlowski 11y ago
      I just asked the question in a separate post before seeing yours, but I questioned this as well. Would it have been better to have a separate instance of DynamoDB independent from the one customers are writing to? I understand this is added cost/overhead, but it would have resulted in isolating the failure to the customer side only, I'd think.
  - threeseed 11y ago
    "Amazon SQS uses an internal table stored in DynamoDB to store information describing its queues" "EC2 Auto Scaling stores information about its groups and launch configurations in an internal table in DynamoDB" "CloudWatch uses an internal table stored in DynamoDB" "Customers attempting to log into the Console during this period saw much higher latency in the login process. This was due to a very long timeout being set on an API call that relied on DynamoDB" Seems like poor architectural design to have all of these storing state in the same instances of DynamoDB that are used by customers. If a new feature like GSI is added it should under no circumstances ever impact other services.
ckozlowski 11y ago
I noticed that in the "Impact on other services" bit, that CloudWatch and Console were affected, becasue they were dependent on DyanmoDB. Now, I don't pretend to know DyanmoDB to well, but it seems to me that having your monitoring application dependent on one of the things you would be monitoring is a strange circular dependency. Would it have been wiser for Amazon to implement a completely separate instance of DynamoDB for service offerings that depended on it? Or is this just simply cost ineffective? Help me understand, thanks. =)
- eli 11y ago
  And then what? Set up another set of monitoring tools to watch this new DynamoDB instance that's only used for monitoring and console?
- zwily 11y ago
  It sounds like most of the remediation is "we'll make sure this service keeps working even if Dynamo is down". It's telling that so many services are relying heavily on Dynamo now. If anything, this event just makes me like it more. (I have several things built on it).
- _alex_ 11y ago
  CloudWatch is a monitoring service that AWS offers publicly. It is different from the monitoring service that AWS services use internally. source: was on an AWS service team for several years
  - berkay 11y ago
    It's still a problem even if it's not the internal monitoring system. Customers use CloudWatch to detect problems with DynamoDB and get notified. This dependency means customers may not get notified if CloudWatch does not work as expected due to a DynamoDB problem.
    - sharpy 11y ago
      There are different forces at play. On one hand, one can design a service with minimum dependencies to survive other services' outages, but that means duplication of effort and increased cost, as well as slower delivery of features. Or one can focus on adding value. It seems though, from the update, that CloudWatch is going to have some sort of caching for most recent data.
deleted 11y ago
[deleted]
humanlike 11y ago
Very nice v1.11
bboreham 11y ago
> We did not have detailed enough monitoring for this dimension (membership size) It is often the case that you aren't monitoring the metric that would have told you the world was about to end. However, > their processing time was near the time limit for retrieval This is a specific response time with a known, fatal, limit. Not seeing that trend seems unfortunate.
djb_hackernews 11y ago
Hmm, we are seeing very high request latencies with DynamoDB in multiple accounts and can't even load the DynamoDB console...Anyone else?
- lostcolony 11y ago
  I can load the console, but it's very, very slow. And creating a new table takes forever, then spews error text over the interface in a way that makes it clear they never intended for an error to be displayed this way.
- RyanGWU82 11y ago
  Yes, we are seeing problems that started around 6:08 AM PDT. We contacted AWS Support and they confirmed they're investigating an incident.
- cddotdotslash 11y ago
  Same - the DynamoDB console is returning 500s.
humanlike 11y ago
Very nice.
jgalt212 11y ago
Does anyone keep stats on service outages for AWS, Axure, Google, et al?
- jaz 11y ago
  The closest I've seen to that is this page: https://cloudharmony.com/status-group-provider https://cloudharmony.com/status-group-provider
- colinbartlett 11y ago
  I currently track a lot of this through my project StatusGator which alerts you on outages: https://statusgator.io https://statusgator.io. Most of the historical data isn't shown publicly but I could probably do something with that.
  - jgalt212 11y ago
    The historical data would be quite useful for companies when deciding which cloud providers to contract with.
yandie 11y ago
As this post is on, Dynamo is experiencing error again: 6:33 AM PDT We are investigating increased error rates for API requests in the US-EAST-1 Region. Source: http://status.aws.amazon.com/ http://status.aws.amazon.com/
- colinbartlett 11y ago
  Some services are reporting problems now. Starting with the same set as last time such as Heroku and lots of services that depend on it: https://statusgator.io/events https://statusgator.io/events
  - toomuchtodo 11y ago
    DynamoDB, Cloudwatch, EC2, Scaling, and Elastic Beanstalk are all impacted currently.
- impostervt 11y ago
  Thank you for this. I was wondering why I was getting slow responses from their API Gateway. It's "green" on their status board, but based on what I saw Sunday, the two services are linked.
  - cddotdotslash 11y ago
    Pretty much the exact same thing that happened the other day. DynamoDB, then Lambda, now API Gateway. I can't update my Lambda functions, a small percentage are timing out, and API Gateway is sending back a "We're out of capacity" message.
- Someone1234 11y ago
  Their status screen is totally misleading and utterly pointless. Someone needs to get all of the RSS feeds (which are actually accurate) and create a new dash which is honest.
  - reustle 11y ago
felixgallo 11y ago
"It was impacting a relatively small number of customers, but we should have posted the green-i to the dashboard sooner than we did on Monday." Amazon. No. Please listen. You should never post a green-i. Green-i means nothing to anyone. It's a minimization of a problem. You should change the indicator to show that there is a problem. If there is a problem, that is what the indicator is for. Customers don't care about whatever weird internal political pressure causes you to want to show as little information as possible on that dashboard. Customers want to know that there is a problem, and we will totally be able to figure out if it is small or large on our own.
- deanCommie 11y ago
  What's with this all-or-nothing attitude? What is so wrong about different severity levels? Green-i: Some small percentage of customers are affected, but most of you have nothing to worry about. So if you are a customer having problems and see this notice, you are re-assured that you are not crazy. And if you are NOT seeing problems, you probably won't. Amazon has not always been the best at posting this information quickly enough, hence a commitment to be better at messaging for those customers Yellow: A significant amount of customers is affected. Potentially serious problems, and if you haven't seen them yet, don't be surprised if you will. Start monitoring your service health and preparing for a failover to another region. Red: FUBAR. Happens rarely, but when it does you'll probably know even before your application alarms - you'll notice when half the internet shuts down.
  - felixgallo 11y ago
    different severity levels are fine. That's what the indicator is for. Decorating the indicator, normally completely incorrectly, with a tiny extra indicator, is incomprehensible. That's why you've never seen a tiny red light in the upper corner of your green traffic light. Your 'check engine' light is not green with a tiny extra indicator in the corner. The metaphors go on and on. Customers are totally not interested in the fact that some other random customer may not be experiencing problems and that amazon has a wide variety of services, some of which may not be affected in certain geographies. Customers go to that panel with one express goal: to discover if the problems they are seeing with their servers could be related to them. It is a triage check. The question is not, is Amazon super great and are the availability engineers the bestest on average over time? The question is, what the fuck is going on? Amazon has suffered catastrophic unavailability, and the green-i has appeared belatedly, an hour or more into the problem. Because as revealed in this post, it is manually put up by engineers. Manually! Manually!
deleted 11y ago
[deleted]
- ceejayoz 11y ago
  I very much doubt that's an actual confession. The rest of their LinkedIn profile (https://www.linkedin.com/in/danveloper https://www.linkedin.com/in/danveloper) is a series of way-too-honest "I fucked up and got fired" jobs. > Although I signed up for a programming job, I was tasked with literally seeding clouds. I actually had to fly an airplane and dispel a solution of silver iodide into the air to -- quite literally -- "make it rain". I would've stayed here forever if the company hadn't gone under. Unfortunately, as it turned out, hiring programmers to manipulate the weather turned out to be a bad management policy. Oh well... ce la vie.
astral303 11y ago
"We did not realize soon enough that this low overall error rate was giving some customers disproportionately high error rates." Critical lesson for running multitenant SaaS services at scale: you need to monitor not just overall error rates, but individual tenant/customer error rates.

190 points by spew 11y ago

Unavailable servers continued to retry requests for membership data, maintaining high load on the metadata service. It sounds like the retries exacerbated the situation. I wonder if they use exponential backoff?

BillinghamJ 11y ago
It's amazing how frequently this seems to happen. A lot of the huge downtime events I've read about have occurred because of failures in the systems used to recover from failures. It seems that perhaps more time/effort needs to be spent testing the systems (and the use of those systems) which are critical for handling failure. While building them, it's easy to dismiss their scaling requirements on the basis that they shouldn't ever be under significant load.
- antirez 11y ago
  This is a classic, even because those software parts usually are the least tested in a variety of conditions, even when they are relatively well tested. It's very hard to simulate real failures.
bbrazil 11y ago
> I wonder if they use exponential backoff? More importantly, did they use randomised exponential backoff? Having all the retires hitting at the same time can lead to a pulses of outages until things settle down.
- jmason23 11y ago
  I have no doubt they would have been using randomised exponential backoff -- these issues are well recognised inside Amazon and best practices are well known. For example, here's a blog post from March this year from Marc Brooker on the topic: http://www.awsarchitectureblog.com/2015/03/backoff.html http://www.awsarchitectureblog.com/2015/03/backoff.html . It may not have been correctly tuned for this scenario however. I'd say the Dynamo team are well aware of what they should have been doing, and kicking themselves for not foreseeing this cascading-failure case. ouch!
allengeorge 11y ago
Most exponential backoff implementations I've seen have an upper limit, where the initial (fast) retries work around transient networking blips. My guess is that a case like this you'd hit the backoff's upper bound pretty quickly, and, given the large volume of servers hitting a comparatively small metadata pool, experience exactly the same failure mode.

arturhoo 11y ago

We did not have detailed enough monitoring for this dimension (membership size), and didn’t have enough capacity allocated to the metadata service to handle these much heavier requests. As much as I admire and rely on AWS' scale to build architectures and fault tolerant applications, it can't be ignored that the marketing towards going "full cloud" doesn't take into account how hard it is to build resilient architectures in the cloud. I see those disruptions events as stop signs: when the cloud itself fails to scale, I rethink a few decisions we all make when surfing those trends. http://yourdatafitsinram.com/ http://yourdatafitsinram.com/ also comes to mind.

chillydawg 11y ago
How is it any harder to do in the cloud than on a rack in a warehouse? At least you don't have to muck about with cables and phoning power companies up.
- bbrazil 11y ago
  Given they had ~300 minutes of outage in 3 years, you're looking at ~99.98% reliable in just that region. That's pretty good for a stateful serving system, and indeed you'd be pushed to do better.
  - pfortuny 11y ago
    That is just time not traffic loss, it would be hard to get so much traffic lost by a single in-house service. Metrics depend.
- antirez 11y ago
  Just an example: during the issue even people serving 10 ops/sec, but very important 10 ops/sec, were affected by a huge complessive load which was not their for most of the part. It's true that when you "go cloud" you don't have to manage your operations, but you are basically putting everything in the hands of other op people, and what happens to you is related to a more wide set of conditions. So managing your stuff is hard, but you are in control and can do things in a way you believe is completely safe for you. Or you at least may incur in the same events sometimes, but perhaps paying a lot less for the same services. Or you can create your deployment with characteristics which are often impossible (a lot of RAM for each server is an example) to be cost effective in the cloud. It's not stupid to use AWS services but is not stupid to manage your operations, either in your own hardware or at least using just bare metal and/or the virtual machines service certain providers give you, but still being in part accountable, responsabile, and in control, of your system software deployment and operations.

deskamess 11y ago

> but we should have posted the green-i to the dashboard sooner than we did Use full orange or even half orange circle or something else if you want to convey 'subset of customers may have issues'. If possible move 'having issue' items to the top - this way people do not have to scroll down to see whats broken.

zwily 11y ago
This was the first time we had actually seen red icons on the dashboard. We joked that they had to furiously photoshop them up Sunday morning. (I'm sure it had happened before, just the first time we had seen them.)
deanCommie 11y ago
Not enough granularity with a yellow icon. I feel like if only <1% of customers are affected, a yellow circle isn't warranted - that's for the 5-20% impact case
- takeda 11y ago
  Unless you're one of the <1% of customers. The whole point of status page is to to help determine if the issue you're experiencing is on your side or the SaaS, and not to show as much green as possible. The way currently AWS status page works it simply fails to provide any functionality and might as well be shut down. The colors on it start to change when you already see tons of articles about AWS being down.

ghshephard 11y ago

A pretty good post mortem, but one worrisome final comment: "For us, availability is the most important feature of DynamoDB" I would think that durability should be the most important feature of DynamoDB; better to have intermittent outage, or reduced capacity, but not have data loss - but perhaps there is something about DynamoDB which suggests availability is rated more highly than durability?

manuelflara 11y ago
They probably said that because this was an availability outage. If this was a postmortem after some data had been lost, they would've ended it with "For us, durability is the most important feature of DynamoDB".
stingraycharles 11y ago
I think this is written from an ops point of view. Dynamo-based systems allow a client to wait until at least X servers have acknowledged the write, and use eventual consistency in case there is some service disruption. I personally didnt know they used meta data tables in Dynamo, and that sounds like they lost one of the most important traits which gave Dynamo its super-high availability abilities, which is having all the cluster state in the clients rather than in the servers.
- sillypog 11y ago
  Thank you for saying that! I just read the Dynamo paper the other day and that was the first thing that jumped out at me when I read this outage description. Now I feel like I understood the paper.
  - zackelan 11y ago
    Despite the name, DynamoDB is not actually an implementation of the Dynamo paper. It's influenced by those ideas, as well as the issues with scaling SimpleDB, but other than that is a from-scratch implementation. http://www.allthingsdistributed.com/2012/01/amazon-dynamodb.html http://www.allthingsdistributed.com/2012/01/amazon-dynamodb....

beambot 11y ago

Looks like a fair number of key aws systems rely on DynamoDB -- and further, the same system used by customers. I wonder: do they have any inclination to decouple these dependencies to prevent correlated outages?

Jgrubb 11y ago
I don't think they rely on Dynamo, they rely on an internal metadata service that Dynamo just happened to overwhelm with too many large requests.
- beambot 11y ago
  From the 2nd sentence: > ... subsequent impact to other AWS services that depend on DynamoDB ... And the metadata service is part of DynamoDB: > The membership of a set of table/partitions within a server is managed by DynamoDB’s internal metadata service.
  - ckozlowski 11y ago
    I just asked the question in a separate post before seeing yours, but I questioned this as well. Would it have been better to have a separate instance of DynamoDB independent from the one customers are writing to? I understand this is added cost/overhead, but it would have resulted in isolating the failure to the customer side only, I'd think.
- threeseed 11y ago
  "Amazon SQS uses an internal table stored in DynamoDB to store information describing its queues" "EC2 Auto Scaling stores information about its groups and launch configurations in an internal table in DynamoDB" "CloudWatch uses an internal table stored in DynamoDB" "Customers attempting to log into the Console during this period saw much higher latency in the login process. This was due to a very long timeout being set on an API call that relied on DynamoDB" Seems like poor architectural design to have all of these storing state in the same instances of DynamoDB that are used by customers. If a new feature like GSI is added it should under no circumstances ever impact other services.

ckozlowski 11y ago

I noticed that in the "Impact on other services" bit, that CloudWatch and Console were affected, becasue they were dependent on DyanmoDB. Now, I don't pretend to know DyanmoDB to well, but it seems to me that having your monitoring application dependent on one of the things you would be monitoring is a strange circular dependency. Would it have been wiser for Amazon to implement a completely separate instance of DynamoDB for service offerings that depended on it? Or is this just simply cost ineffective? Help me understand, thanks. =)

eli 11y ago
And then what? Set up another set of monitoring tools to watch this new DynamoDB instance that's only used for monitoring and console?
zwily 11y ago
It sounds like most of the remediation is "we'll make sure this service keeps working even if Dynamo is down". It's telling that so many services are relying heavily on Dynamo now. If anything, this event just makes me like it more. (I have several things built on it).
_alex_ 11y ago
CloudWatch is a monitoring service that AWS offers publicly. It is different from the monitoring service that AWS services use internally. source: was on an AWS service team for several years
- berkay 11y ago
  It's still a problem even if it's not the internal monitoring system. Customers use CloudWatch to detect problems with DynamoDB and get notified. This dependency means customers may not get notified if CloudWatch does not work as expected due to a DynamoDB problem.
  - sharpy 11y ago
    There are different forces at play. On one hand, one can design a service with minimum dependencies to survive other services' outages, but that means duplication of effort and increased cost, as well as slower delivery of features. Or one can focus on adding value. It seems though, from the update, that CloudWatch is going to have some sort of caching for most recent data.

deleted 11y ago

[deleted]

humanlike 11y ago

Very nice v1.11

bboreham 11y ago

> We did not have detailed enough monitoring for this dimension (membership size) It is often the case that you aren't monitoring the metric that would have told you the world was about to end. However, > their processing time was near the time limit for retrieval This is a specific response time with a known, fatal, limit. Not seeing that trend seems unfortunate.

djb_hackernews 11y ago

Hmm, we are seeing very high request latencies with DynamoDB in multiple accounts and can't even load the DynamoDB console...Anyone else?

lostcolony 11y ago
I can load the console, but it's very, very slow. And creating a new table takes forever, then spews error text over the interface in a way that makes it clear they never intended for an error to be displayed this way.
RyanGWU82 11y ago
Yes, we are seeing problems that started around 6:08 AM PDT. We contacted AWS Support and they confirmed they're investigating an incident.
cddotdotslash 11y ago
Same - the DynamoDB console is returning 500s.

humanlike 11y ago

Very nice.

jgalt212 11y ago

Does anyone keep stats on service outages for AWS, Axure, Google, et al?

jaz 11y ago
The closest I've seen to that is this page: https://cloudharmony.com/status-group-provider https://cloudharmony.com/status-group-provider
colinbartlett 11y ago
I currently track a lot of this through my project StatusGator which alerts you on outages: https://statusgator.io https://statusgator.io. Most of the historical data isn't shown publicly but I could probably do something with that.
- jgalt212 11y ago
  The historical data would be quite useful for companies when deciding which cloud providers to contract with.

yandie 11y ago

As this post is on, Dynamo is experiencing error again: 6:33 AM PDT We are investigating increased error rates for API requests in the US-EAST-1 Region. Source: http://status.aws.amazon.com/ http://status.aws.amazon.com/

colinbartlett 11y ago
Some services are reporting problems now. Starting with the same set as last time such as Heroku and lots of services that depend on it: https://statusgator.io/events https://statusgator.io/events
- toomuchtodo 11y ago
  DynamoDB, Cloudwatch, EC2, Scaling, and Elastic Beanstalk are all impacted currently.
impostervt 11y ago
Thank you for this. I was wondering why I was getting slow responses from their API Gateway. It's "green" on their status board, but based on what I saw Sunday, the two services are linked.
- cddotdotslash 11y ago
  Pretty much the exact same thing that happened the other day. DynamoDB, then Lambda, now API Gateway. I can't update my Lambda functions, a small percentage are timing out, and API Gateway is sending back a "We're out of capacity" message.
Someone1234 11y ago
Their status screen is totally misleading and utterly pointless. Someone needs to get all of the RSS feeds (which are actually accurate) and create a new dash which is honest.
- reustle 11y ago

felixgallo 11y ago

"It was impacting a relatively small number of customers, but we should have posted the green-i to the dashboard sooner than we did on Monday." Amazon. No. Please listen. You should never post a green-i. Green-i means nothing to anyone. It's a minimization of a problem. You should change the indicator to show that there is a problem. If there is a problem, that is what the indicator is for. Customers don't care about whatever weird internal political pressure causes you to want to show as little information as possible on that dashboard. Customers want to know that there is a problem, and we will totally be able to figure out if it is small or large on our own.

deanCommie 11y ago
What's with this all-or-nothing attitude? What is so wrong about different severity levels? Green-i: Some small percentage of customers are affected, but most of you have nothing to worry about. So if you are a customer having problems and see this notice, you are re-assured that you are not crazy. And if you are NOT seeing problems, you probably won't. Amazon has not always been the best at posting this information quickly enough, hence a commitment to be better at messaging for those customers Yellow: A significant amount of customers is affected. Potentially serious problems, and if you haven't seen them yet, don't be surprised if you will. Start monitoring your service health and preparing for a failover to another region. Red: FUBAR. Happens rarely, but when it does you'll probably know even before your application alarms - you'll notice when half the internet shuts down.
- felixgallo 11y ago
  different severity levels are fine. That's what the indicator is for. Decorating the indicator, normally completely incorrectly, with a tiny extra indicator, is incomprehensible. That's why you've never seen a tiny red light in the upper corner of your green traffic light. Your 'check engine' light is not green with a tiny extra indicator in the corner. The metaphors go on and on. Customers are totally not interested in the fact that some other random customer may not be experiencing problems and that amazon has a wide variety of services, some of which may not be affected in certain geographies. Customers go to that panel with one express goal: to discover if the problems they are seeing with their servers could be related to them. It is a triage check. The question is not, is Amazon super great and are the availability engineers the bestest on average over time? The question is, what the fuck is going on? Amazon has suffered catastrophic unavailability, and the green-i has appeared belatedly, an hour or more into the problem. Because as revealed in this post, it is manually put up by engineers. Manually! Manually!

deleted 11y ago

[deleted]

ceejayoz 11y ago
I very much doubt that's an actual confession. The rest of their LinkedIn profile (https://www.linkedin.com/in/danveloper https://www.linkedin.com/in/danveloper) is a series of way-too-honest "I fucked up and got fired" jobs. > Although I signed up for a programming job, I was tasked with literally seeding clouds. I actually had to fly an airplane and dispel a solution of silver iodide into the air to -- quite literally -- "make it rain". I would've stayed here forever if the company hadn't gone under. Unfortunately, as it turned out, hiring programmers to manipulate the weather turned out to be a bad management policy. Oh well... ce la vie.

astral303 11y ago

"We did not realize soon enough that this low overall error rate was giving some customers disproportionately high error rates." Critical lesson for running multitenant SaaS services at scale: you need to monitor not just overall error rates, but individual tenant/customer error rates.