15 ms·

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any su

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his department at Amazon, being at fault for an outage was one of the worst things that could happen to you. He wanted to avoid that mark any way possible. YMMV, of course. Amazon is a big company and I've had other friends work there in different departments who said this wasn't common at all. I will always remember the look of sheer panic he had when we insisted that he update the status page to accurately reflect an outage, though.

broknbottle 5y ago
This gets posted every time there's an AWS outage. It mind as well be a copy pasta at this point.
- Spivak 5y ago
  I mean it's true at every company I've ever worked at too. If you can lawyer incidents into not being an outage you avoid like 15 meetings with the business stakeholders about all the things we "have to do" to prevent things like this in the future that get canceled the moment they realize that how much dev/infra time it will take to implement.
- JoelMcCracken 5y ago
  well, this is the first time I've seen it, so I am glad it was posted this time.
  - jjoonathan 5y ago
    First time I've seen it too. Definitely not my first "AWS us-east-1 is down but the status board is green" thread, either.
  - rconti 5y ago
    Ditto, it's always annoyed me that their status page is useless, but glad someone else mentioned it.
- deleted 5y ago
  [deleted]
- ignoramous 5y ago
  I had that deja vu feeling reading PragmaticPulp's comment, too. And sure enough, PragmaticPulp did post a similar comment on a thread about Amazon India's alleged hire-to-fire policy 6 months back: https://news.ycombinator.com/item?id=27570411 https://news.ycombinator.com/item?id=27570411 You and I, we aren't among the 10000, but there are potentially 10000 others who might be: https://xkcd.com/1053/ https://xkcd.com/1053/
- PragmaticPulp 5y ago
  Sorry. I'm probably to blame because I've posted this a couple times on HN before. It strikes a nerve with me because it caused so much trouble for everyone around him. He had other personal issues, though, so I should probably clarify that I'm not entirely blaming Amazon for his habits. Though his time at Amazon clearly did exacerbate his personal issues.
- Rapzid 5y ago
  It's the "grandma got run over by a reindeer" of AWS outages. Really no outage thread would be complete without this anecdote.
howdydoo 5y ago
Manually updated status pages are an anti-pattern to begin with. At that point, why not just call it a blog?
ashr 5y ago
This is the exact opposite of my experience at AWS. Amazon is all about blameless fact finding when it comes to root cause analysis. Your company just hired a not so great engineer or misunderstood him.
- Insanity 5y ago
  Adding my piece of anecdata to this.. the process is quite blameless. If a postmortem seems like it points blame, this is pointed out and removed.
  - swiftcoder 5y ago
    Blameless, maybe, but not repercussion-less. A bad CoE was liable to upend the team's entire roadmap and put their existing goals at risk. To be fair, management was fairly receptive to "we need to throw out the roadmap and push our launch out to the following reinvent", but it wasn't an easy position for teams to be in.
jacquesm 5y ago
And this is exactly why you can expect these headlines to hit with great regularity. These things are never a problem at the individual level, they are always at the level of culture and organization.
StreamBright 5y ago
Yep I can confirm that. The process when the outage is caused by you is called COE (correction of errors). I was oncall once for two teams because I was switching teams and I got 11 escalations in 2 hours. 10 of these were caused by an overly sensitive monitoring setting. The 11th was a real one. Guess which one I ignored. :)
avalys 5y ago
I can totally picture this. Poor guy.
jonhohle 5y ago
On the retail/marketplace side this wasn't my experience, but we also didn't have any public dashboards. On Prime we occasionally had to refund in bulk, and when it was called for (internally or externally) we would right up a detailed post-mortem. This wasn't fun, but it was never about blaming a person and more about finding flaws in process or monitoring.
kache_ 5y ago
Sometimes, these large companies tack on too much "necessary" incident "remediation" actions with Arbitrary Due Date SLAs that completely wrench any ongoing work. And ongoing, strategically defined ""muh high impact"" projects are what get you promoted, not doing incident remediations. When you get to the level you want, you get to not really give a shit and actually do The Right Thing. However, for all of the engineers clamoring to get out of the intermediate brick laying trenches, opening an incident can have pervasive incentives.
- pts_ 5y ago
  Politicized cloud meh.
- bendbro 5y ago
  In my experience this is the actual reason for fear of the formal error correction process.
1-6 5y ago
Perhaps reward structure should be changed to incentivize the post-mortems. There could be several flaws that run underreported otherwise. We may run into the problem of everything documented and possible deliberate acts but for a service that relies heavily on uptime, that’s a small price to pay for a bulletproof operation.
- A4ET8a8uTh0 5y ago
  Then we would drown in a sea of meetings and 'lessons learned' emails. There is a reason for post-mortems, but there has to be balance.
  - 1-6 5y ago
    I find post-mortems interesting to read through especially when it’s not my fault. Most of them would probably be routine to read through but there are occasional ones that make me cringe or laugh. Post-mortems can sometime be thought of like safety training. There is a big imbalance of time dedicated to learning proper safety handling just for those small incidences.
    - hinkley 5y ago
      Does Disney still play the "Instructional Videos" series starring Goofy where he's supposed to be teaching you how to do something and instead we learn how NOT to do something? Or did I just date myself badly?
mrweasel 5y ago
That’s idiotic, the service is down regardless. If you foster that kind of culture, why have a status page at all? It make AWS engineers look stupid, because it looks like they are not monitoring their services.
- nine_zeros 5y ago
  > It make AWS engineers look stupid, because it looks like they are not monitoring their services. Management.
- mountainofdeath 5y ago
  The status page is as much a political tool as a technical one. Giving your service a non-green state makes your entire management chain responsible. You don't want to be one that upsets some VPs advancement plans.
kortex 5y ago
That sounds like the exact opposite of human-factors engineering. No one likes taking blame. But when things go sideways, people are extra spicy and defensive, which makes them clam up and often withhold useful information, which can extend the outage. No-blame analysis is a much better pattern. Everyone wins. It's about building the system that builds the system. Stuff broke; fix the stuff that broke, then fix the things that let stuff break.
- maximedupre 5y ago
  Or just take responsibility. People will respect you for doing that and you will demonstrate leadership.
  - artificial 5y ago
    Way more fun argument: Outages just, uh… uh… find a way.
  - melony 5y ago
    And the guy who doesn't take responsibility gets promoted. Employees are not responsible for failures of management to set a good culture.
    - tomrod 5y ago
      Not in healthy organizations, they don't.
      - foobiekr 5y ago
        You can work an entire career and maybe enjoy life in one healthy organization in that entire time even if you work in a variety of companies. It just isn't that common, though of course voicing the _ideals_ is very, very common.
      - jacquesm 5y ago
        Once you reach a certain size there are surprisingly few healthy organization, most of them turn into externalization engines with 4 beats per year.
maximedupre 5y ago
Damn, he had serious PTSD lol
jimt1234 5y ago
Every incident review meeting I've ever been in starts out like, _"This meeting isn't to place blame..."_, then, 5 minutes later, it turns into the Blame Game.
taf2 5y ago
What if they just can't access the console to update the status page...
- Slartie 5y ago
  They could still go into the data center, open up the status page servers' physical...ah wait, what if their keyfobs don't work?
mijoharas 5y ago
That's a real shame, one of the leadership principles used to be "be vocally self-critical" which I think was supposed to explicitly counteract this kind of behaviour. I think they got rid of it at some point though.
sharpy 5y ago
Haha... This bring back memories. It really depends on the org. I've had push backs on my postmortems before because of phrasing that could be constituted as laying some of the blame on some person/team when it's supposed to be blameless. And for a long time, it was fairly blameless. You would still be punished with the extra work of writing high quality postmortems, but I have seen people accidentally bring down critical tier-1 services and not be adversely affected in terms of promotion, etc. But somewhere along the way, it became politicized. Things like the wheel of death, public grilling of teams on why they didn't follow one of the thousands of best practices, etc, etc. Some orgs are still pretty good at keeping it blameless at the individual level, but... being a big company, your mileage may vary.
- hinkley 5y ago
  We're in a situation where the balls of mud made people afraid to touch some things in the system. As experiences and processes have improved we've started to crack back into those things and guess what, when you are being groomed to own a process you're going to fuck it up from time to time. Objectively, we're still breaking production less often per year than other teams, but we are breaking it, and that's novel behavior, so we have to keep reminding people why. The moment that affects promotions negatively, or your coworkers throw you under the bus, you should 1) be assertive and 2) proof-read your resume as a precursor to job hunting.
  - sharpy 5y ago
    Or problems just persisting, because the fix is easy, but explaining it to others who do not work on the system are hard. Esp. justifying why it won't cause an issue, and being told that the fixes need to be done via scripts that will only ever be used once, but nevertheless needs to be code reviewed and tested... I wanted to be proactive and fix things before they became an issue, but such things just drained life out of me, to the point I just left.
hinkley 5y ago
I am finding that I have a very bimodal response to "He did it". When I write an RCA or just talk about near misses, I may give you enough details to figure out that Tom was the one who broke it, but I'm not going to say Tom on the record anywhere, with one extremely obvious exception. If I think Tom has a toxic combination of poor judgement, Dunning-Kruger syndrome, and a hint of narcissism (I'm not sure but I may be repeating myself here), such that he won't listen to reason and he actively steers others into bad situations (and especially if he then disappears when shit hits the fan), then I will nail him to a fucking cross every chance I get. Public shaming is only a tool for getting people to discount advice from a bad actor. If it comes down to a vote between my idea and his, then I'm going to make sure everyone knows that his bets keep biting us in the ass. This guy kinda sounds like the Toxic Tom. What is important when I turned out to be the cause of the issue is a bit like some court cases. Would a reasonable person in this situation have come to the same conclusion I did? If so, then I'm just the person who lost the lottery. Either way, fixing it for me might fix it for other people. Sometimes the answer is, "I was trying to juggle three things at once and a ball got dropped." If the process dictated those three things then the process is wrong, or the tooling is wrong. If someone was asking me questions we should think about being more pro-active about deflecting them to someone else or asking them to come back in a half hour. Or maybe I shouldn't be trying to watch training videos while babysitting a deployment to production. If you never say "my bad" then your advice starts to sound like a lecture, and people avoid lectures so then you never get the whole story. Also as an engineer you should know that owning a mistake early on lets you get to what most of us consider the interesting bit of solving the problem instead of talking about feelings for an hour and then using whatever is left of your brain afterward to fix the problem. In fact in some cases you can shut down someone who is about to start a rant (which is funny as hell because they look like their head is about to pop like a balloon when you say, "yep, I broke it, let's move on to how do we fix it?")
- bgribble 5y ago
  To me, the point of "blameless" PM is not to hide the identity of the person who was closest to the failure point. You can't understand what happened unless you know who did what, when. "Blameless" to me means you acknowledge that the ultimate problem isn't that someone made a mistake that caused an outage. The problem is that you had a system in place where someone could make a single mistake and cause an outage. If someone fat-fingers a SQL query and drops your database, the problem isn't that they need typing lessons! If you put a DBA in a position where they have to be typing SQL directly at a production DB to do their job, THAT is the cause of the outage, the actual DBA's error is almost irrelevant because it would have happened eventually to someone.
kator 5y ago
I've worked for Amazon for 4 years, including stints at AWS, and even in my current role my team is involved in LSE's. I've never seen this behavior, the general culture has been find the problem, fix it, and then do root cause analysis to avoid it again. Jeff himself has said many times in All Hands and in public "Amazon is the best place to fail". Mainly because things will break, it's not that they break that's interesting, it's what you've learned and how you can avoid that problem in the future.
- jsperson 5y ago
  I guess the question is why can't you (AWS) fix the problem of the status page not reflecting an outage? Maybe acceptable if the console has a hiccup, but when www.amazon.com isn't working right, there should be some yellow and red dots out there. With the size of your customer base there were man years spent confirming the outage after checking the status.
  - andrewguenther 5y ago
    Because there's a VP approval step for updating the status page and no repercussions for VPs who don't approve updates in a timely manner. Updating the status page is fully automated on both sides of VP approval. If the status page doesn't update, it's because a VP wouldn't do it.
- Eduard 5y ago
  LSE?
  - merciBien 5y ago
    Large Scale Event
soheil 5y ago
This may not actually be that bad of thing. If you think about it if they're fighting tooth and nail to keep the status page still green that tells you they were probably doing that at every step of the way before the failure became eminent. Gotta have respect for that.
300bps 5y ago
being at fault for an outage was one of the worst things that could happen to you Imagine how stressful life would be thinking that you had to be perfect all the time.
- errcorrectcode 5y ago
  That's been most of my life. Welcome to perfectionism.
mbordenet 5y ago
When I worked for AMZN (2012-2015, Prime Video & Outbound Fulfillment), attempting to sweep issues under the rug was a clear path to termination. The Correction-Of-Error (COE) process can work wonders in a healthy, data-driven, growth-mindset culture. I wonder if the ex-Amazonian you're referring to did not leave AMZN by their own accord? Blame deflection is a recipe for repeat outages and unhappy customers.
- PragmaticPulp 5y ago
  > I wonder if the ex-Amazonian you're referring to did not leave AMZN by their own accord? Entirely possible, and something I've always suspected.
mountainofdeath 5y ago
Former AWSser. I can totally believe that happened and continues to happen in some teams. Officially, it's not supposed to be done that way. Some AWS managers and engineers bring their corporate cultural baggage with them when they join AWS and it takes a few years to unlearn it.
- PragmaticPulp 5y ago
  Thanks for the perspective. I was beginning to regret posting this after so many people claiming this wouldn’t happen at AWS. Amazon is a huge company so I have no doubt YMMV depending on your manager.
amzn-throw 5y ago
It's popular to upvote this during outages, because it fits a narrative. The truth (as always) is more complex: * No, this isn't the broad culture. It's not even a blip. These are EXCEPTIONAL circumstances by extremely bad teams that - if and when found out - would be intervened dramatically. * The broad culture is blameless post-mortems. Not whose fault is it. But what was the problem and how to fix it. And one of the internal "Ten commandments of AWS availability" is you own your dependencies. You don't blame others. * Depending on the service one customer's experience is not the broad experience. Someone might be having a really bad day but 99.9% of the region is operating successfully, so there is no reason to update the overall status dashboard. * Every AWS customer has a PERSONAL health dashboard in the console that should indicate their experience. * Yes, VP approval is needed to make any updates on the status dashboard. But that's not as hard as it may seem. AWS executives are extremely operation-obsessed, and when there is an outage of any size are engaged with their service teams immediately.
- miken123 5y ago
  Well, the narrative is sort of what Amazon is asking for, heh? The whole us-east-1 management console is gone, what is Amazon posting for the management console on their website? "Service degradation" It's not a degradation if it's outright down. Use the red status a little bit more often, this is a "disruption", not a "degradation".
  - whoknowswhat11 5y ago
    I've always wondered why services are not counted down more often. Is there some sliver of customers who have access to the management console for example? An increase in error rates - no biggie, any large system is going to have errors. But when 80%+ of customers loads in the region are impacted (cross availability zones for whatever good those do) - that counts as down doesn't it? Error rates in one AZ - degraded. Multi-AZ failures - down?
    - bwestpha 5y ago
      yes there were. I'm from central europe and we were at least able to get some pages of the console in us-east-1 -but i assume this was more caching related. Even though the console loaded and worked for listing some entries - we weren't able to post a support case nor viewing SQS messages etc. So i aggree that degraded is not the proper wording - but it's / was not completly vanished. so.... hard to tell what is an common acceptable wording here.
throwaway82931 5y ago
This fits with everything I've heard about terrible code quality at Amazon and engineers working ridiculous hours to close tickets any way they can. Amazon as a corporate entity seems to be remarkably distrustful of and hostile to its labor force.
dehrmann 5y ago
> explanation about why the outage was someone else's fault In my experience, it's rarely clear who was at fault for any sort of non-trivial outage. The issue tends to be at interfaces and involve multiple owners.
staticassertion 5y ago
I don't think anecdotes like this are even worth sharing, honestly. There's so much context lost here, so much that can be lost in translation. No one should be drawing any conclusions from this post.
dang 5y ago
(This was originally a reply to https://news.ycombinator.com/item?id=29473759 https://news.ycombinator.com/item?id=29473759 but I've pruned it to make the thread less top-heavy.)
soheil 5y ago
> I will always remember the look of sheer panic I don't know if you're exaggerating or not, but even if true why would anyone show that emotion about losing a job in the worst case? You certainly had a lot of relevate-to-todays-top-hn-post stories throughout you career. And I'm less and less surprised to continuously find PragmaticPulp as one of the top commenters if not the top that resonates with a good chunk of HN.
throwthere 5y ago
Maybe it’s more telling that that engineer no longer works at Amazon.
imwillofficial 5y ago
This is weird, on my team it’s taken as a learning opportunity. I caused a pretty big outage and we just did a COE.

broknbottle 5y ago

This gets posted every time there's an AWS outage. It mind as well be a copy pasta at this point.

Spivak 5y ago
I mean it's true at every company I've ever worked at too. If you can lawyer incidents into not being an outage you avoid like 15 meetings with the business stakeholders about all the things we "have to do" to prevent things like this in the future that get canceled the moment they realize that how much dev/infra time it will take to implement.
JoelMcCracken 5y ago
well, this is the first time I've seen it, so I am glad it was posted this time.
- jjoonathan 5y ago
  First time I've seen it too. Definitely not my first "AWS us-east-1 is down but the status board is green" thread, either.
- rconti 5y ago
  Ditto, it's always annoyed me that their status page is useless, but glad someone else mentioned it.
deleted 5y ago
[deleted]
ignoramous 5y ago
I had that deja vu feeling reading PragmaticPulp's comment, too. And sure enough, PragmaticPulp did post a similar comment on a thread about Amazon India's alleged hire-to-fire policy 6 months back: https://news.ycombinator.com/item?id=27570411 https://news.ycombinator.com/item?id=27570411 You and I, we aren't among the 10000, but there are potentially 10000 others who might be: https://xkcd.com/1053/ https://xkcd.com/1053/
PragmaticPulp 5y ago
Sorry. I'm probably to blame because I've posted this a couple times on HN before. It strikes a nerve with me because it caused so much trouble for everyone around him. He had other personal issues, though, so I should probably clarify that I'm not entirely blaming Amazon for his habits. Though his time at Amazon clearly did exacerbate his personal issues.
Rapzid 5y ago
It's the "grandma got run over by a reindeer" of AWS outages. Really no outage thread would be complete without this anecdote.

howdydoo 5y ago

Manually updated status pages are an anti-pattern to begin with. At that point, why not just call it a blog?

ashr 5y ago

This is the exact opposite of my experience at AWS. Amazon is all about blameless fact finding when it comes to root cause analysis. Your company just hired a not so great engineer or misunderstood him.

Insanity 5y ago
Adding my piece of anecdata to this.. the process is quite blameless. If a postmortem seems like it points blame, this is pointed out and removed.
- swiftcoder 5y ago
  Blameless, maybe, but not repercussion-less. A bad CoE was liable to upend the team's entire roadmap and put their existing goals at risk. To be fair, management was fairly receptive to "we need to throw out the roadmap and push our launch out to the following reinvent", but it wasn't an easy position for teams to be in.

jacquesm 5y ago

And this is exactly why you can expect these headlines to hit with great regularity. These things are never a problem at the individual level, they are always at the level of culture and organization.

StreamBright 5y ago

Yep I can confirm that. The process when the outage is caused by you is called COE (correction of errors). I was oncall once for two teams because I was switching teams and I got 11 escalations in 2 hours. 10 of these were caused by an overly sensitive monitoring setting. The 11th was a real one. Guess which one I ignored. :)

avalys 5y ago

I can totally picture this. Poor guy.

jonhohle 5y ago

On the retail/marketplace side this wasn't my experience, but we also didn't have any public dashboards. On Prime we occasionally had to refund in bulk, and when it was called for (internally or externally) we would right up a detailed post-mortem. This wasn't fun, but it was never about blaming a person and more about finding flaws in process or monitoring.

kache_ 5y ago

Sometimes, these large companies tack on too much "necessary" incident "remediation" actions with Arbitrary Due Date SLAs that completely wrench any ongoing work. And ongoing, strategically defined ""muh high impact"" projects are what get you promoted, not doing incident remediations. When you get to the level you want, you get to not really give a shit and actually do The Right Thing. However, for all of the engineers clamoring to get out of the intermediate brick laying trenches, opening an incident can have pervasive incentives.

pts_ 5y ago
Politicized cloud meh.
bendbro 5y ago
In my experience this is the actual reason for fear of the formal error correction process.

1-6 5y ago

Perhaps reward structure should be changed to incentivize the post-mortems. There could be several flaws that run underreported otherwise. We may run into the problem of everything documented and possible deliberate acts but for a service that relies heavily on uptime, that’s a small price to pay for a bulletproof operation.

A4ET8a8uTh0 5y ago
Then we would drown in a sea of meetings and 'lessons learned' emails. There is a reason for post-mortems, but there has to be balance.
- 1-6 5y ago
  I find post-mortems interesting to read through especially when it’s not my fault. Most of them would probably be routine to read through but there are occasional ones that make me cringe or laugh. Post-mortems can sometime be thought of like safety training. There is a big imbalance of time dedicated to learning proper safety handling just for those small incidences.
  - hinkley 5y ago
    Does Disney still play the "Instructional Videos" series starring Goofy where he's supposed to be teaching you how to do something and instead we learn how NOT to do something? Or did I just date myself badly?

mrweasel 5y ago

That’s idiotic, the service is down regardless. If you foster that kind of culture, why have a status page at all? It make AWS engineers look stupid, because it looks like they are not monitoring their services.

nine_zeros 5y ago
> It make AWS engineers look stupid, because it looks like they are not monitoring their services. Management.
mountainofdeath 5y ago
The status page is as much a political tool as a technical one. Giving your service a non-green state makes your entire management chain responsible. You don't want to be one that upsets some VPs advancement plans.

kortex 5y ago

That sounds like the exact opposite of human-factors engineering. No one likes taking blame. But when things go sideways, people are extra spicy and defensive, which makes them clam up and often withhold useful information, which can extend the outage. No-blame analysis is a much better pattern. Everyone wins. It's about building the system that builds the system. Stuff broke; fix the stuff that broke, then fix the things that let stuff break.

maximedupre 5y ago
Or just take responsibility. People will respect you for doing that and you will demonstrate leadership.
- artificial 5y ago
  Way more fun argument: Outages just, uh… uh… find a way.
- melony 5y ago
  And the guy who doesn't take responsibility gets promoted. Employees are not responsible for failures of management to set a good culture.
  - tomrod 5y ago
    Not in healthy organizations, they don't.
    - foobiekr 5y ago
      You can work an entire career and maybe enjoy life in one healthy organization in that entire time even if you work in a variety of companies. It just isn't that common, though of course voicing the _ideals_ is very, very common.
    - jacquesm 5y ago
      Once you reach a certain size there are surprisingly few healthy organization, most of them turn into externalization engines with 4 beats per year.

maximedupre 5y ago

Damn, he had serious PTSD lol

jimt1234 5y ago

Every incident review meeting I've ever been in starts out like, _"This meeting isn't to place blame..."_, then, 5 minutes later, it turns into the Blame Game.

taf2 5y ago

What if they just can't access the console to update the status page...

Slartie 5y ago
They could still go into the data center, open up the status page servers' physical...ah wait, what if their keyfobs don't work?

mijoharas 5y ago

That's a real shame, one of the leadership principles used to be "be vocally self-critical" which I think was supposed to explicitly counteract this kind of behaviour. I think they got rid of it at some point though.

sharpy 5y ago

Haha... This bring back memories. It really depends on the org. I've had push backs on my postmortems before because of phrasing that could be constituted as laying some of the blame on some person/team when it's supposed to be blameless. And for a long time, it was fairly blameless. You would still be punished with the extra work of writing high quality postmortems, but I have seen people accidentally bring down critical tier-1 services and not be adversely affected in terms of promotion, etc. But somewhere along the way, it became politicized. Things like the wheel of death, public grilling of teams on why they didn't follow one of the thousands of best practices, etc, etc. Some orgs are still pretty good at keeping it blameless at the individual level, but... being a big company, your mileage may vary.

hinkley 5y ago
We're in a situation where the balls of mud made people afraid to touch some things in the system. As experiences and processes have improved we've started to crack back into those things and guess what, when you are being groomed to own a process you're going to fuck it up from time to time. Objectively, we're still breaking production less often per year than other teams, but we are breaking it, and that's novel behavior, so we have to keep reminding people why. The moment that affects promotions negatively, or your coworkers throw you under the bus, you should 1) be assertive and 2) proof-read your resume as a precursor to job hunting.
- sharpy 5y ago
  Or problems just persisting, because the fix is easy, but explaining it to others who do not work on the system are hard. Esp. justifying why it won't cause an issue, and being told that the fixes need to be done via scripts that will only ever be used once, but nevertheless needs to be code reviewed and tested... I wanted to be proactive and fix things before they became an issue, but such things just drained life out of me, to the point I just left.

hinkley 5y ago

I am finding that I have a very bimodal response to "He did it". When I write an RCA or just talk about near misses, I may give you enough details to figure out that Tom was the one who broke it, but I'm not going to say Tom on the record anywhere, with one extremely obvious exception. If I think Tom has a toxic combination of poor judgement, Dunning-Kruger syndrome, and a hint of narcissism (I'm not sure but I may be repeating myself here), such that he won't listen to reason and he actively steers others into bad situations (and especially if he then disappears when shit hits the fan), then I will nail him to a fucking cross every chance I get. Public shaming is only a tool for getting people to discount advice from a bad actor. If it comes down to a vote between my idea and his, then I'm going to make sure everyone knows that his bets keep biting us in the ass. This guy kinda sounds like the Toxic Tom. What is important when I turned out to be the cause of the issue is a bit like some court cases. Would a reasonable person in this situation have come to the same conclusion I did? If so, then I'm just the person who lost the lottery. Either way, fixing it for me might fix it for other people. Sometimes the answer is, "I was trying to juggle three things at once and a ball got dropped." If the process dictated those three things then the process is wrong, or the tooling is wrong. If someone was asking me questions we should think about being more pro-active about deflecting them to someone else or asking them to come back in a half hour. Or maybe I shouldn't be trying to watch training videos while babysitting a deployment to production. If you never say "my bad" then your advice starts to sound like a lecture, and people avoid lectures so then you never get the whole story. Also as an engineer you should know that owning a mistake early on lets you get to what most of us consider the interesting bit of solving the problem instead of talking about feelings for an hour and then using whatever is left of your brain afterward to fix the problem. In fact in some cases you can shut down someone who is about to start a rant (which is funny as hell because they look like their head is about to pop like a balloon when you say, "yep, I broke it, let's move on to how do we fix it?")

bgribble 5y ago
To me, the point of "blameless" PM is not to hide the identity of the person who was closest to the failure point. You can't understand what happened unless you know who did what, when. "Blameless" to me means you acknowledge that the ultimate problem isn't that someone made a mistake that caused an outage. The problem is that you had a system in place where someone could make a single mistake and cause an outage. If someone fat-fingers a SQL query and drops your database, the problem isn't that they need typing lessons! If you put a DBA in a position where they have to be typing SQL directly at a production DB to do their job, THAT is the cause of the outage, the actual DBA's error is almost irrelevant because it would have happened eventually to someone.

kator 5y ago

I've worked for Amazon for 4 years, including stints at AWS, and even in my current role my team is involved in LSE's. I've never seen this behavior, the general culture has been find the problem, fix it, and then do root cause analysis to avoid it again. Jeff himself has said many times in All Hands and in public "Amazon is the best place to fail". Mainly because things will break, it's not that they break that's interesting, it's what you've learned and how you can avoid that problem in the future.

jsperson 5y ago
I guess the question is why can't you (AWS) fix the problem of the status page not reflecting an outage? Maybe acceptable if the console has a hiccup, but when www.amazon.com isn't working right, there should be some yellow and red dots out there. With the size of your customer base there were man years spent confirming the outage after checking the status.
- andrewguenther 5y ago
  Because there's a VP approval step for updating the status page and no repercussions for VPs who don't approve updates in a timely manner. Updating the status page is fully automated on both sides of VP approval. If the status page doesn't update, it's because a VP wouldn't do it.
Eduard 5y ago
LSE?
- merciBien 5y ago
  Large Scale Event

soheil 5y ago

This may not actually be that bad of thing. If you think about it if they're fighting tooth and nail to keep the status page still green that tells you they were probably doing that at every step of the way before the failure became eminent. Gotta have respect for that.

300bps 5y ago

being at fault for an outage was one of the worst things that could happen to you Imagine how stressful life would be thinking that you had to be perfect all the time.

errcorrectcode 5y ago
That's been most of my life. Welcome to perfectionism.

mbordenet 5y ago

When I worked for AMZN (2012-2015, Prime Video & Outbound Fulfillment), attempting to sweep issues under the rug was a clear path to termination. The Correction-Of-Error (COE) process can work wonders in a healthy, data-driven, growth-mindset culture. I wonder if the ex-Amazonian you're referring to did not leave AMZN by their own accord? Blame deflection is a recipe for repeat outages and unhappy customers.

PragmaticPulp 5y ago
> I wonder if the ex-Amazonian you're referring to did not leave AMZN by their own accord? Entirely possible, and something I've always suspected.

mountainofdeath 5y ago

Former AWSser. I can totally believe that happened and continues to happen in some teams. Officially, it's not supposed to be done that way. Some AWS managers and engineers bring their corporate cultural baggage with them when they join AWS and it takes a few years to unlearn it.

PragmaticPulp 5y ago
Thanks for the perspective. I was beginning to regret posting this after so many people claiming this wouldn’t happen at AWS. Amazon is a huge company so I have no doubt YMMV depending on your manager.

amzn-throw 5y ago

It's popular to upvote this during outages, because it fits a narrative. The truth (as always) is more complex: * No, this isn't the broad culture. It's not even a blip. These are EXCEPTIONAL circumstances by extremely bad teams that - if and when found out - would be intervened dramatically. * The broad culture is blameless post-mortems. Not whose fault is it. But what was the problem and how to fix it. And one of the internal "Ten commandments of AWS availability" is you own your dependencies. You don't blame others. * Depending on the service one customer's experience is not the broad experience. Someone might be having a really bad day but 99.9% of the region is operating successfully, so there is no reason to update the overall status dashboard. * Every AWS customer has a PERSONAL health dashboard in the console that should indicate their experience. * Yes, VP approval is needed to make any updates on the status dashboard. But that's not as hard as it may seem. AWS executives are extremely operation-obsessed, and when there is an outage of any size are engaged with their service teams immediately.

miken123 5y ago
Well, the narrative is sort of what Amazon is asking for, heh? The whole us-east-1 management console is gone, what is Amazon posting for the management console on their website? "Service degradation" It's not a degradation if it's outright down. Use the red status a little bit more often, this is a "disruption", not a "degradation".
- whoknowswhat11 5y ago
  I've always wondered why services are not counted down more often. Is there some sliver of customers who have access to the management console for example? An increase in error rates - no biggie, any large system is going to have errors. But when 80%+ of customers loads in the region are impacted (cross availability zones for whatever good those do) - that counts as down doesn't it? Error rates in one AZ - degraded. Multi-AZ failures - down?
  - bwestpha 5y ago
    yes there were. I'm from central europe and we were at least able to get some pages of the console in us-east-1 -but i assume this was more caching related. Even though the console loaded and worked for listing some entries - we weren't able to post a support case nor viewing SQS messages etc. So i aggree that degraded is not the proper wording - but it's / was not completly vanished. so.... hard to tell what is an common acceptable wording here.

throwaway82931 5y ago

This fits with everything I've heard about terrible code quality at Amazon and engineers working ridiculous hours to close tickets any way they can. Amazon as a corporate entity seems to be remarkably distrustful of and hostile to its labor force.

dehrmann 5y ago

> explanation about why the outage was someone else's fault In my experience, it's rarely clear who was at fault for any sort of non-trivial outage. The issue tends to be at interfaces and involve multiple owners.

staticassertion 5y ago

I don't think anecdotes like this are even worth sharing, honestly. There's so much context lost here, so much that can be lost in translation. No one should be drawing any conclusions from this post.

dang 5y ago

(This was originally a reply to https://news.ycombinator.com/item?id=29473759 https://news.ycombinator.com/item?id=29473759 but I've pruned it to make the thread less top-heavy.)

soheil 5y ago

> I will always remember the look of sheer panic I don't know if you're exaggerating or not, but even if true why would anyone show that emotion about losing a job in the worst case? You certainly had a lot of relevate-to-todays-top-hn-post stories throughout you career. And I'm less and less surprised to continuously find PragmaticPulp as one of the top commenters if not the top that resonates with a good chunk of HN.

throwthere 5y ago

Maybe it’s more telling that that engineer no longer works at Amazon.

imwillofficial 5y ago

This is weird, on my team it’s taken as a learning opportunity. I caused a pretty big outage and we just did a COE.