8 ms·

> “We try to have a blameless postmortem,” said one engineer. “That email from Thuan [...] was great example of not following that.” Besides, the employee added

> “We try to have a blameless postmortem,” said one engineer. “That email from Thuan [...] was great example of not following that.” Besides, the employee added, “if you’ve been woken up at 3 a.m. for the last five days, and you’re only sleeping three to four hours a day, and you make a mistake, how much at fault are you, really?” > In a follow-up email two days later, Pham addressed criticism of his decision to email the staff about the engineer’s mistake. "It came to my attention that some people in the org think that my note to the company is overly critical and has the feel of throwing people under the bus,” he wrote. > “I don't want to create a culture [where] people are fearful of making mistakes or causing outages because they want to move fast and take smart risks, but I also don't want a culture where we do substandard work and cause outages that are easily avoidable,” Pham wrote. > He continued: “Feeling defensive, or feeling like a victim, is NOT the way to get ourselves better.” I find Pham's response to criticism about the blameless post-mortem to be lacking in self-reflection. The entire point point of a blameless post mortem is to ensure that people don't feel defensive or feel like a victim. To get criticism about failing to follow a blameless post-mortem process and respond by saying: "don't feel like a victim?" tells me that they haven't internalized the criticism in any way.

oh_sigh 9y ago
Blameless post mortems only work when you know that engineers or whoever is on call are operating in good faith. From Pham's first email, it seems like there is some question whether that was the case, because the engineer repeatedly acked the pages without resolving any issue: >The on-call engineer received/acknowledged three alerts about master database being low on disk space, but ignored it. This is not acceptable,” Pham wrote in the email, sent to more than 3,500 employees and obtained by BuzzFeed News. “We are looking to determine whether this is negligence or whether a different on-call engineer could have reasonably missed the alerts amidst a flood of other alerts from the systems at that time.” I think the timing of the statement was bad - it should have been a result of the post mortem if it was negligence, and not so explicitly stated publicly going in, if only because it can tarnish someone's reputation even if it turns out not true.
- wdb 9y ago
  I can''t believe they would only have one engineer on-call. That would just be bad practice by Uber. Every major company I worked with we had many people on-call
- ncallaway 9y ago
  > Blameless post mortems only work when you know that engineers or whoever is on call are operating in good faith. There's no question of someone operating in bad faith in the article, though. Pham has a concern about negligence, but that is not bad faith. An engineer can be operating in good faith and be negligent. I think the emails go beyond bad timing. They clearly indicate ahead of time that the post mortem will be "negligence or a root cause that would have effected a different engineer". That's not a blameless post-mortem. That's the opposite.
- drewbug01 9y ago
  > From Pham's first email, it seems like there is some question whether that was the case, because the engineer repeatedly acked the pages without resolving any issue: As an employee who has been on-call for production systems for most of his career, this smacks of someone who's never been on-call and is indeed looking for a scapegoat. From the article: > “We are looking to determine whether this is negligence or whether a different on-call engineer could have reasonably missed the alerts amidst a flood of other alerts from the systems at that time.” The key phrase is "flood of other alerts" - to me, this sounds as if other shit was hitting the fan and the engineer rolled the dice on letting this sit while fixing something else that was perceived to be more important. Unfortunately, that seems to have been a bad move. However, it's an eminently understandable move (and indeed, one I've made myself). How much disk is left? What's the rate of change? Can you even know that immediately while still trying to deal with the "flood of other alerts?" This sounds like there's a lot of things went wrong here, but I doubt the engineer was one of them.
  - CaptainZapp 9y ago
    The key phrase is "flood of other alerts" I wish I could upvote you more than once for this being so true, Need a case study? In 2005 the entire power grid for the clean, famed and punctual Swiss Federal Railways shut down: http://news.bbc.co.uk/2/hi/europe/4121072.stm http://news.bbc.co.uk/2/hi/europe/4121072.stm Yep, exactly that. Every train on the entire network shut down due to an overload of the grid. Probably the only disadvantage on being on a 100% electrified network. It would have been avoidable, the errors where displayed on the relevant consoles, drowned by 100s of other irrelevant errors and warnings and thus not noticed. Blaming it on the guy who should have made sense of it all and essentially throwing him under the bus in a corporate wide email just about shows what a shitty company Uber really is.
draw_down 9y ago
Oof, that's really bad. At the point you're publicly shaming people to do better, you need to fix your organization. Like fast.

oh_sigh 9y ago

Blameless post mortems only work when you know that engineers or whoever is on call are operating in good faith. From Pham's first email, it seems like there is some question whether that was the case, because the engineer repeatedly acked the pages without resolving any issue: >The on-call engineer received/acknowledged three alerts about master database being low on disk space, but ignored it. This is not acceptable,” Pham wrote in the email, sent to more than 3,500 employees and obtained by BuzzFeed News. “We are looking to determine whether this is negligence or whether a different on-call engineer could have reasonably missed the alerts amidst a flood of other alerts from the systems at that time.” I think the timing of the statement was bad - it should have been a result of the post mortem if it was negligence, and not so explicitly stated publicly going in, if only because it can tarnish someone's reputation even if it turns out not true.

wdb 9y ago
I can''t believe they would only have one engineer on-call. That would just be bad practice by Uber. Every major company I worked with we had many people on-call
ncallaway 9y ago
> Blameless post mortems only work when you know that engineers or whoever is on call are operating in good faith. There's no question of someone operating in bad faith in the article, though. Pham has a concern about negligence, but that is not bad faith. An engineer can be operating in good faith and be negligent. I think the emails go beyond bad timing. They clearly indicate ahead of time that the post mortem will be "negligence or a root cause that would have effected a different engineer". That's not a blameless post-mortem. That's the opposite.
drewbug01 9y ago
> From Pham's first email, it seems like there is some question whether that was the case, because the engineer repeatedly acked the pages without resolving any issue: As an employee who has been on-call for production systems for most of his career, this smacks of someone who's never been on-call and is indeed looking for a scapegoat. From the article: > “We are looking to determine whether this is negligence or whether a different on-call engineer could have reasonably missed the alerts amidst a flood of other alerts from the systems at that time.” The key phrase is "flood of other alerts" - to me, this sounds as if other shit was hitting the fan and the engineer rolled the dice on letting this sit while fixing something else that was perceived to be more important. Unfortunately, that seems to have been a bad move. However, it's an eminently understandable move (and indeed, one I've made myself). How much disk is left? What's the rate of change? Can you even know that immediately while still trying to deal with the "flood of other alerts?" This sounds like there's a lot of things went wrong here, but I doubt the engineer was one of them.
- CaptainZapp 9y ago
  The key phrase is "flood of other alerts" I wish I could upvote you more than once for this being so true, Need a case study? In 2005 the entire power grid for the clean, famed and punctual Swiss Federal Railways shut down: http://news.bbc.co.uk/2/hi/europe/4121072.stm http://news.bbc.co.uk/2/hi/europe/4121072.stm Yep, exactly that. Every train on the entire network shut down due to an overload of the grid. Probably the only disadvantage on being on a 100% electrified network. It would have been avoidable, the errors where displayed on the relevant consoles, drowned by 100s of other irrelevant errors and warnings and thus not noticed. Blaming it on the guy who should have made sense of it all and essentially throwing him under the bus in a corporate wide email just about shows what a shitty company Uber really is.

draw_down 9y ago

Oof, that's really bad. At the point you're publicly shaming people to do better, you need to fix your organization. Like fast.