4 ms·
Blameless post mortems only work when you know that engineers or whoever is on call are operating in good faith. From Pham's first email, it seems like there is
by oh_sigh 9y ago
Blameless post mortems only work when you know that engineers or whoever is on call are operating in good faith. From Pham's first email, it seems like there is some question whether that was the case, because the engineer repeatedly acked the pages without resolving any issue:
>The on-call engineer received/acknowledged three alerts about master database being low on disk space, but ignored it. This is not acceptable,” Pham wrote in the email, sent to more than 3,500 employees and obtained by BuzzFeed News. “We are looking to determine whether this is negligence or whether a different on-call engineer could have reasonably missed the alerts amidst a flood of other alerts from the systems at that time.”
I think the timing of the statement was bad - it should have been a result of the post mortem if it was negligence, and not so explicitly stated publicly going in, if only because it can tarnish someone's reputation even if it turns out not true.
- wdb 9y agoI can''t believe they would only have one engineer on-call. That would just be bad practice by Uber. Every major company I worked with we had many people on-call
- ncallaway 9y ago> Blameless post mortems only work when you know that engineers or whoever is on call are operating in good faith. There's no question of someone operating in bad faith in the article, though. Pham has a concern about negligence, but that is not bad faith. An engineer can be operating in good faith and be negligent. I think the emails go beyond bad timing. They clearly indicate ahead of time that the post mortem will be "negligence or a root cause that would have effected a different engineer". That's not a blameless post-mortem. That's the opposite.
- drewbug01 9y ago> From Pham's first email, it seems like there is some question whether that was the case, because the engineer repeatedly acked the pages without resolving any issue: As an employee who has been on-call for production systems for most of his career, this smacks of someone who's never been on-call and is indeed looking for a scapegoat. From the article: > “We are looking to determine whether this is negligence or whether a different on-call engineer could have reasonably missed the alerts amidst a flood of other alerts from the systems at that time.” The key phrase is "flood of other alerts" - to me, this sounds as if other shit was hitting the fan and the engineer rolled the dice on letting this sit while fixing something else that was perceived to be more important. Unfortunately, that seems to have been a bad move. However, it's an eminently understandable move (and indeed, one I've made myself). How much disk is left? What's the rate of change? Can you even know that immediately while still trying to deal with the "flood of other alerts?" This sounds like there's a lot of things went wrong here, but I doubt the engineer was one of them.
- CaptainZapp 9y agoThe key phrase is "flood of other alerts" I wish I could upvote you more than once for this being so true, Need a case study? In 2005 the entire power grid for the clean, famed and punctual Swiss Federal Railways shut down: http://news.bbc.co.uk/2/hi/europe/4121072.stm http://news.bbc.co.uk/2/hi/europe/4121072.stm Yep, exactly that. Every train on the entire network shut down due to an overload of the grid. Probably the only disadvantage on being on a 100% electrified network. It would have been avoidable, the errors where displayed on the relevant consoles, drowned by 100s of other irrelevant errors and warnings and thus not noticed. Blaming it on the guy who should have made sense of it all and essentially throwing him under the bus in a corporate wide email just about shows what a shitty company Uber really is.