5 ms·

"It was impacting a relatively small number of customers, but we should have posted the green-i to the dashboard sooner than we did on Monday." Amazon. No. P

"It was impacting a relatively small number of customers, but we should have posted the green-i to the dashboard sooner than we did on Monday." Amazon. No. Please listen. You should never post a green-i. Green-i means nothing to anyone. It's a minimization of a problem. You should change the indicator to show that there is a problem. If there is a problem, that is what the indicator is for. Customers don't care about whatever weird internal political pressure causes you to want to show as little information as possible on that dashboard. Customers want to know that there is a problem, and we will totally be able to figure out if it is small or large on our own.

deanCommie 11y ago
What's with this all-or-nothing attitude? What is so wrong about different severity levels? Green-i: Some small percentage of customers are affected, but most of you have nothing to worry about. So if you are a customer having problems and see this notice, you are re-assured that you are not crazy. And if you are NOT seeing problems, you probably won't. Amazon has not always been the best at posting this information quickly enough, hence a commitment to be better at messaging for those customers Yellow: A significant amount of customers is affected. Potentially serious problems, and if you haven't seen them yet, don't be surprised if you will. Start monitoring your service health and preparing for a failover to another region. Red: FUBAR. Happens rarely, but when it does you'll probably know even before your application alarms - you'll notice when half the internet shuts down.
- felixgallo 11y ago
  different severity levels are fine. That's what the indicator is for. Decorating the indicator, normally completely incorrectly, with a tiny extra indicator, is incomprehensible. That's why you've never seen a tiny red light in the upper corner of your green traffic light. Your 'check engine' light is not green with a tiny extra indicator in the corner. The metaphors go on and on. Customers are totally not interested in the fact that some other random customer may not be experiencing problems and that amazon has a wide variety of services, some of which may not be affected in certain geographies. Customers go to that panel with one express goal: to discover if the problems they are seeing with their servers could be related to them. It is a triage check. The question is not, is Amazon super great and are the availability engineers the bestest on average over time? The question is, what the fuck is going on? Amazon has suffered catastrophic unavailability, and the green-i has appeared belatedly, an hour or more into the problem. Because as revealed in this post, it is manually put up by engineers. Manually! Manually!
  - deanCommie 11y ago
    The question of timing of the green-I is valid, but different from your original complaint. In your hypothetical scenario, the customer that is going to the panel to "find out what the fuck is going on" is not going to miss the indicator i, and will read the description of the message.
    - RyanGWU82 11y ago
      It's really easy to miss the indicator "i". I've been doing this for 4 years and it still doesn't catch my eye when scanning the page.
      - deskamess 11y ago
        I agree 100%. That column of uniform green makes it easy to miss. It is a poor UI choice but can be easily fixed.
        rstupek 11y ago
        However it is hard to miss the difference in the column with details and the more link even upon a cursory scan.
    - toomuchtodo 11y ago
      I don't care what that "i" indicator is; if you're showing green while you're impacted, your status board is useless and I'm going to Twitter and HN to see if you're broken (which is what I already have to do for AWS).
      - mgkimsal 11y ago
        When we've seen problems, some of my clients go there and see green and say "Amazon shows they're fine - it must be your code or something you did or didn't do - fix it". Well... it's green but it's not fine. And there's little I can do to demonstrate to someone that it's not code but infrastructure.
- nickpsecurity 11y ago
  Green traditionally means we're good, ready to go, etc. Non-intuitive to use any green indicator for a situation deserving a write-up like this. It's yellow at best.
  - cthalupa 11y ago
    During the actual outage this write up was about, didn't it go red?
    - nickpsecurity 11y ago
      I wasn't watching the indicator. I started with the parent comment's assertion. If it was wrong, my comment doesn't apply to this outage and becomes a rule of thumb.
- takeda 11y ago
  Yep, especially yellow is labeled clearly as "Performance issues". It is basically lying to make things appear better than they really are.
- mentat 11y ago
  > Some small percentage of customers are affected, but most of you have nothing to worry about. That's absolutely not what it means practically for any of the recent outages. It has come to mean "the service is down but not hard enough for us to admit it".
- d23 11y ago
  > By 2:37am PDT, the error rate in customer requests to DynamoDB had risen far beyond any level experienced in the last 3 years, finally stabilizing at approximately 55%. Yeah, a small green-i should cover it.
jamiesonbecker 11y ago
I'm starting to wonder how much more I should trust AWS. Is everything swept under the carpet? e.g., not even attributed or thanks for root escalation vulnerabilities that I surfaced for them: https://news.ycombinator.com/item?id=10261507 https://news.ycombinator.com/item?id=10261507

deanCommie 11y ago

What's with this all-or-nothing attitude? What is so wrong about different severity levels? Green-i: Some small percentage of customers are affected, but most of you have nothing to worry about. So if you are a customer having problems and see this notice, you are re-assured that you are not crazy. And if you are NOT seeing problems, you probably won't. Amazon has not always been the best at posting this information quickly enough, hence a commitment to be better at messaging for those customers Yellow: A significant amount of customers is affected. Potentially serious problems, and if you haven't seen them yet, don't be surprised if you will. Start monitoring your service health and preparing for a failover to another region. Red: FUBAR. Happens rarely, but when it does you'll probably know even before your application alarms - you'll notice when half the internet shuts down.

felixgallo 11y ago
different severity levels are fine. That's what the indicator is for. Decorating the indicator, normally completely incorrectly, with a tiny extra indicator, is incomprehensible. That's why you've never seen a tiny red light in the upper corner of your green traffic light. Your 'check engine' light is not green with a tiny extra indicator in the corner. The metaphors go on and on. Customers are totally not interested in the fact that some other random customer may not be experiencing problems and that amazon has a wide variety of services, some of which may not be affected in certain geographies. Customers go to that panel with one express goal: to discover if the problems they are seeing with their servers could be related to them. It is a triage check. The question is not, is Amazon super great and are the availability engineers the bestest on average over time? The question is, what the fuck is going on? Amazon has suffered catastrophic unavailability, and the green-i has appeared belatedly, an hour or more into the problem. Because as revealed in this post, it is manually put up by engineers. Manually! Manually!
- deanCommie 11y ago
  The question of timing of the green-I is valid, but different from your original complaint. In your hypothetical scenario, the customer that is going to the panel to "find out what the fuck is going on" is not going to miss the indicator i, and will read the description of the message.
  - RyanGWU82 11y ago
    It's really easy to miss the indicator "i". I've been doing this for 4 years and it still doesn't catch my eye when scanning the page.
    - deskamess 11y ago
      I agree 100%. That column of uniform green makes it easy to miss. It is a poor UI choice but can be easily fixed.
      - rstupek 11y ago
        However it is hard to miss the difference in the column with details and the more link even upon a cursory scan.
  - toomuchtodo 11y ago
    I don't care what that "i" indicator is; if you're showing green while you're impacted, your status board is useless and I'm going to Twitter and HN to see if you're broken (which is what I already have to do for AWS).
    - mgkimsal 11y ago
      When we've seen problems, some of my clients go there and see green and say "Amazon shows they're fine - it must be your code or something you did or didn't do - fix it". Well... it's green but it's not fine. And there's little I can do to demonstrate to someone that it's not code but infrastructure.
nickpsecurity 11y ago
Green traditionally means we're good, ready to go, etc. Non-intuitive to use any green indicator for a situation deserving a write-up like this. It's yellow at best.
- cthalupa 11y ago
  During the actual outage this write up was about, didn't it go red?
  - nickpsecurity 11y ago
    I wasn't watching the indicator. I started with the parent comment's assertion. If it was wrong, my comment doesn't apply to this outage and becomes a rule of thumb.
takeda 11y ago
Yep, especially yellow is labeled clearly as "Performance issues". It is basically lying to make things appear better than they really are.
mentat 11y ago
> Some small percentage of customers are affected, but most of you have nothing to worry about. That's absolutely not what it means practically for any of the recent outages. It has come to mean "the service is down but not hard enough for us to admit it".
d23 11y ago
> By 2:37am PDT, the error rate in customer requests to DynamoDB had risen far beyond any level experienced in the last 3 years, finally stabilizing at approximately 55%. Yeah, a small green-i should cover it.

jamiesonbecker 11y ago

I'm starting to wonder how much more I should trust AWS. Is everything swept under the carpet? e.g., not even attributed or thanks for root escalation vulnerabilities that I surfaced for them: https://news.ycombinator.com/item?id=10261507 https://news.ycombinator.com/item?id=10261507