4 ms·

So this is the second high profile outage in the last month caused by a simple command line mistake. > Unfortunately, one of the inputs to the command was ente

So this is the second high profile outage in the last month caused by a simple command line mistake. > Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. If I would have guessed anyone could prevent mistakes like this from propagating it would be AWS. It points to just how easy it is to make these errors. I am sure that the SRE who made this mistake is amazing and competent and just had one bad moment. While I hope that AWS would be as understanding as Gitlab, I doubt the outcome is the same.

imsofuture 10y ago
Amazon has the wherewithal to not freaking publicly name their actually human employee, so I'd imagine their culture around outages is probably a lot more healthy.
- Dangeranger 10y ago
  Well to be fair, he named himself within his notes and did not object to the public nature of the disclosure. I agree with your sentiment though that names should not be included within postmortems in the general case.
- draw_down 10y ago
  I couldn't believe that when I saw it in that incident report. Just crazy.
rolandog 10y ago
> While I hope that AWS would be as understanding as Gitlab, I doubt the outcome is the same. I hadn't thought about how 'prone to forgive' an organization is. Should there be policies in place as to how big a mistake you are allowed to commit? Something akin to: ```if(log10(cost_of_mistake) >= 5): worker.fire() ```
andrewguenther 10y ago
Nothing is going to happen to the engineer that did this, other than embarrassment and probably a couple jokes once sufficient time has passed. Amazon has a strong culture around blaming the process, not the person. The failure here wasn't that the engineer ran the command, it was that the engineer was able to run the command. Source: AWS employee
- Dangeranger 10y ago
  That's encouraging to hear. I am glad that Amazon has the right target in it's sights for correction.

imsofuture 10y ago

Amazon has the wherewithal to not freaking publicly name their actually human employee, so I'd imagine their culture around outages is probably a lot more healthy.

Dangeranger 10y ago
Well to be fair, he named himself within his notes and did not object to the public nature of the disclosure. I agree with your sentiment though that names should not be included within postmortems in the general case.
draw_down 10y ago
I couldn't believe that when I saw it in that incident report. Just crazy.

rolandog 10y ago

> While I hope that AWS would be as understanding as Gitlab, I doubt the outcome is the same. I hadn't thought about how 'prone to forgive' an organization is. Should there be policies in place as to how big a mistake you are allowed to commit? Something akin to: ```if(log10(cost_of_mistake) >= 5): worker.fire() ```

andrewguenther 10y ago

Nothing is going to happen to the engineer that did this, other than embarrassment and probably a couple jokes once sufficient time has passed. Amazon has a strong culture around blaming the process, not the person. The failure here wasn't that the engineer ran the command, it was that the engineer was able to run the command. Source: AWS employee

Dangeranger 10y ago
That's encouraging to hear. I am glad that Amazon has the right target in it's sights for correction.