6 ms·

I can't help but feel terrible for the team of people there that ultimately gets blamed for this, I hope they can get some sleep tonight.

matte_black 8y ago
A night of missed sleep didn’t hurt anybody.
- coryfklein 8y ago
  It's not so much the missed sleep, as it is the 48 hours of intense heart-rending stress.
  - matte_black 8y ago
    I’m blown away by the sudden and swift down votes to my original comment. These engineers work at a world class company and are paid vast sums of money to not fuck things up. They live way better off than the majority of the country and their mere presence makes life more expensive and stressful for communities around them. To suggest they cannot go a mere 48 hours or less without sleep on one of their company’s most hyped days is out of touch.
    - toomuchtodo 8y ago
      Amazon is not world class. The pay is not comparatively great. There is no retail job (including Amazon) where undue stress and sleepless nights are warranted. You're not saving lives, you're selling books and cat litter on the internet.
      - cheeze 8y ago
        Is their pay not comparatively great? Usually when I see this statement, folks are comparing seattle to silicon valley 1:1 which isn't a fair comparison. Seattle is expensive but not that expensive. My friends who work at amzn seem to be compensated in line with everyone else I know, but maybe I'm wrong?
        toomuchtodo 8y ago
        The vesting schedule and the sweatshop-esq environment arguably don't make the pay worth it (compared to what you can make elsewhere). Disclaimer: My knowledge is based solely off of public reporting and first hand experiences of SWEs and TAMs no longer at Amazon/AWS.
    - aninhumer 8y ago
      That someone can earnestly suggest that going 48 hours without sleep is somehow an effective way to address an outage is an indictment of a messed up work culture. People need sleep to think straight, and no amount of money or responsibility is going to change that.
- LocalPCGuy 8y ago
  That...just isn't true. Missed sleep costs millions of dollars a day and worse, numerous human lives in sleep related accidents (car, medical and more).
Cofike 8y ago
The bathroom schedule will be tightened up by 15 minutes as punishment.
- baud147258 8y ago
  Well, with this punishment, there won't be any bathroom break left!
wgerard 8y ago
I thought Amazon (or at least the AWS side of things) was supposedly fairly good about blameless post-mortems? I remember people were praising them as such when this outage happened: https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/
- ben509 8y ago
  They are pretty good about it, and the "Correction of Error" for this one will be pretty epic. Generally, the rule at Amazon is that any particular f*-up is forgivable... once. (Especially if you can show that you had preventative measures, documented procedures and redundancy in place.) That said, there will be finger pointing and blame because you're dealing with human beings.
smn1234 8y ago
People likely won’t be blamed. It’ll most probably be a review of process and a correction of errors to prevent future occurrence.

matte_black 8y ago

A night of missed sleep didn’t hurt anybody.

coryfklein 8y ago
It's not so much the missed sleep, as it is the 48 hours of intense heart-rending stress.
- matte_black 8y ago
  I’m blown away by the sudden and swift down votes to my original comment. These engineers work at a world class company and are paid vast sums of money to not fuck things up. They live way better off than the majority of the country and their mere presence makes life more expensive and stressful for communities around them. To suggest they cannot go a mere 48 hours or less without sleep on one of their company’s most hyped days is out of touch.
  - toomuchtodo 8y ago
    Amazon is not world class. The pay is not comparatively great. There is no retail job (including Amazon) where undue stress and sleepless nights are warranted. You're not saving lives, you're selling books and cat litter on the internet.
    - cheeze 8y ago
      Is their pay not comparatively great? Usually when I see this statement, folks are comparing seattle to silicon valley 1:1 which isn't a fair comparison. Seattle is expensive but not that expensive. My friends who work at amzn seem to be compensated in line with everyone else I know, but maybe I'm wrong?
      - toomuchtodo 8y ago
        The vesting schedule and the sweatshop-esq environment arguably don't make the pay worth it (compared to what you can make elsewhere). Disclaimer: My knowledge is based solely off of public reporting and first hand experiences of SWEs and TAMs no longer at Amazon/AWS.
  - aninhumer 8y ago
    That someone can earnestly suggest that going 48 hours without sleep is somehow an effective way to address an outage is an indictment of a messed up work culture. People need sleep to think straight, and no amount of money or responsibility is going to change that.
LocalPCGuy 8y ago
That...just isn't true. Missed sleep costs millions of dollars a day and worse, numerous human lives in sleep related accidents (car, medical and more).

Cofike 8y ago

The bathroom schedule will be tightened up by 15 minutes as punishment.

baud147258 8y ago
Well, with this punishment, there won't be any bathroom break left!

wgerard 8y ago

I thought Amazon (or at least the AWS side of things) was supposedly fairly good about blameless post-mortems? I remember people were praising them as such when this outage happened: https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/

ben509 8y ago
They are pretty good about it, and the "Correction of Error" for this one will be pretty epic. Generally, the rule at Amazon is that any particular f*-up is forgivable... once. (Especially if you can show that you had preventative measures, documented procedures and redundancy in place.) That said, there will be finger pointing and blame because you're dealing with human beings.

smn1234 8y ago

People likely won’t be blamed. It’ll most probably be a review of process and a correction of errors to prevent future occurrence.