97 ms·

Please keep working on that piece I think it will be very useful for incident reviewers. Someone said the quiet part loud! : """ Common circumstances missing

Please keep working on that piece I think it will be very useful for incident reviewers. Someone said the quiet part loud! : """ Common circumstances missing from accident reports are: Pressures to cut costs or work quicker, Competing requests for colleagues, Unnecessarily complicated systems, Broken tools, Biological needs (e.g. sleep or hunger), Cumbersome enforced processes, Fear of being consequences of doing something out of the ordinary, and Shame of feeling in over one’s head. """

VirusNewbie 1y ago
>Pressures to cut costs or work quicker, >Unnecessarily complicated systems, >Broken tools, >cumbersome enforced processes, I have seen all of these specifically called out in Post Mortems at Google, so that's a plus in my book.
hinkley 1y ago
Saying "I was wrong" can be a very powerful debate tool, especially if you can follow it up with concrete actions to fix the problem. There were tools I wrote at my last job because basically an incident could IMO be tracked back to, "I was on step 5 and George interrupted me to ask a question of some other high priority effort, and when I got back I forgot to finish step 5". The more familiar you are with a runbook, the more your brain will confuse memories from an hour or a day ago with memories of the same routine at a different time. It's literally "did I turn off the stove". You remember turning off the stove. Many, many times. But did you turn off both burners you turned on for lunch? You're certain about one. But what about the other? That's a blur. But we're software developers. The more mundane a task, the more likely that we can replace it with a program to do the same thing. And as you get more familiar with the task you keep adding more to the automatic parts until one day it's just a couple buttons that can be reliably pushed during peak traffic on your services. Just make sure there isn't an active outage going on when you push the buttons.
- hshdhdhehd 1y ago
  Tip. Every runbook should have this first step: 1. Create a document from template (xyz), put it in this share location (abc) and fill it in as you perform each step. I am blessed with a bad memory so I do this anyway! But not everyone has that "advantage".
  - hinkley 1y ago
    I think the tools should have this builtin.
  - accrual 1y ago
    > 1. Create a document from template (xyz), put it in this share location (abc) and fill it in as you perform each step. I do this! I have a common checklist that I copy/paste into the project notes. It's all plain text and I just X items off as I go. Indeed, a missing X has caused problems before.

VirusNewbie 1y ago

>Pressures to cut costs or work quicker, >Unnecessarily complicated systems, >Broken tools, >cumbersome enforced processes, I have seen all of these specifically called out in Post Mortems at Google, so that's a plus in my book.

hinkley 1y ago

Saying "I was wrong" can be a very powerful debate tool, especially if you can follow it up with concrete actions to fix the problem. There were tools I wrote at my last job because basically an incident could IMO be tracked back to, "I was on step 5 and George interrupted me to ask a question of some other high priority effort, and when I got back I forgot to finish step 5". The more familiar you are with a runbook, the more your brain will confuse memories from an hour or a day ago with memories of the same routine at a different time. It's literally "did I turn off the stove". You remember turning off the stove. Many, many times. But did you turn off both burners you turned on for lunch? You're certain about one. But what about the other? That's a blur. But we're software developers. The more mundane a task, the more likely that we can replace it with a program to do the same thing. And as you get more familiar with the task you keep adding more to the automatic parts until one day it's just a couple buttons that can be reliably pushed during peak traffic on your services. Just make sure there isn't an active outage going on when you push the buttons.

hshdhdhehd 1y ago
Tip. Every runbook should have this first step: 1. Create a document from template (xyz), put it in this share location (abc) and fill it in as you perform each step. I am blessed with a bad memory so I do this anyway! But not everyone has that "advantage".
- hinkley 1y ago
  I think the tools should have this builtin.
- accrual 1y ago
  > 1. Create a document from template (xyz), put it in this share location (abc) and fill it in as you perform each step. I do this! I have a common checklist that I copy/paste into the project notes. It's all plain text and I just X items off as I go. Indeed, a missing X has caused problems before.