3 ms·

I was a sceptic, when I first saw 'blameless' post mortem used. Our sister group wiped out our production database and put together a lovely GIT issues templat

by heelix 5y ago

I was a sceptic, when I first saw 'blameless' post mortem used. Our sister group wiped out our production database and put together a lovely GIT issues template on how this was mostly not their fault. Great... However... this has become one of the most powerful changes in our extended group's engineering culture. The focus on what happened and how, rather than the who. Each time, it is a chance to examine how things went kittywampus and how we can break that chain in the future. For the most part, I don't even think we include the names of the people involved. The process grew on me, the more we used it. Folks would fess up, rather than hide it and creating a snipe hunt for root cause. We were able to categorize where our pain point were and what we could do to stop it - and it worked. It really made the folks who could analyze what happened and how we could do better with what we have, shine.

ashout33 5y ago
I'd like to hear more about how you can avoid names. How are actions items assigned? Genuinely curious, this sounds cool
- heelix 5y ago
  We say 'developer' not 'Bob'. The actions are typically things that are not for one person, but rather our scrum teams as a whole. For example, had a junior tasked with a 'rush' action for modifying an index. They botched the script and dropped all documents when they attempted to modify an index. Took down everything. The chain that got us there... Developers had drop access to the prod system - all of them. That got fixed. The account that they were using had more permissions then should be used. Reviews were not done, because rush rush and vacations, the developer did some stack overflow that had unintended consequences. Tests matter. A few other bits that were slipping became checklist items for the change control. It was far more important to fix the processes and gaps than say 'Bob' screwed it up. The other bit was when folks were chewed on, they hit the mistakes. It consumed so much time, when someone would log in and 'fix' something with a system account, and we had to guess what broke the system. It takes time. I was so, so very skeptical when we started. It sounded like someone just avoiding responsibility. First few, were. Looking at what broke, why it broke, and how do we avoid it in the future. At the end of the year, I'll also use these writeups to see where our emphasis areas should be. 2021's top - humans suck at keeping x509 certificates up to date when manually added.

by heelix 5y ago

ashout33 5y ago

I'd like to hear more about how you can avoid names. How are actions items assigned? Genuinely curious, this sounds cool

heelix 5y ago
We say 'developer' not 'Bob'. The actions are typically things that are not for one person, but rather our scrum teams as a whole. For example, had a junior tasked with a 'rush' action for modifying an index. They botched the script and dropped all documents when they attempted to modify an index. Took down everything. The chain that got us there... Developers had drop access to the prod system - all of them. That got fixed. The account that they were using had more permissions then should be used. Reviews were not done, because rush rush and vacations, the developer did some stack overflow that had unintended consequences. Tests matter. A few other bits that were slipping became checklist items for the change control. It was far more important to fix the processes and gaps than say 'Bob' screwed it up. The other bit was when folks were chewed on, they hit the mistakes. It consumed so much time, when someone would log in and 'fix' something with a system account, and we had to guess what broke the system. It takes time. I was so, so very skeptical when we started. It sounded like someone just avoiding responsibility. First few, were. Looking at what broke, why it broke, and how do we avoid it in the future. At the end of the year, I'll also use these writeups to see where our emphasis areas should be. 2021's top - humans suck at keeping x509 certificates up to date when manually added.