3 ms·
The goal with automation is that the number of unidentified corner cases reduces over time. A manual runbook is a game of, "I did step 12, I think I did step 1
by hinkley 3y ago
The goal with automation is that the number of unidentified corner cases reduces over time.
A manual runbook is a game of, "I did step 12, I think I did step 13, so the next step is 14." that plays out every single time you do server work. The thing with the human brain is that when you interrupt a task you've done a million times in the middle, most people can't reliably discern between this iteration and false memories from the last time they did it.
So unless there are interlocks that prevent skipping a step, it's a gamble every single time. And the effort involved in creating interlocks is a large fraction of the cost of automating.
- WJW 3y ago1. Print the checklist/runbook out on paper with actual empty boxes next to the steps. 2. Laminate the printed checklist and put it in a big folder. 3. Every time you run the checklist, use a sharpie to mark the checkbox after you've done the step. 4. When you are done with the entire process, use whiteboard cleaner to wipe out the checks again and put the checklist back in the big folder with all the other checklists. This is how every safety critical profession (aviation, shipping, medical, power generation, etc) has worked for decades and unless people are willingly being obtuse it is extremely hard to do it wrong. You just need people to turn off their ego and follow the process instead of trying to show off by doing it from memory. This last part might be more difficult in software settings.
- MiyamotoAkira 3y agoRegarding checklist people like Hollnagel, Wears, Braithwaite, Dekker, ... have done a bit on investigation (Hollnagel mostly on Healthcare, Dekker started on air industry but spread from there). Read the "Safety-I vs Safety-II" paper or the "When a checklist is not enough: How to improve them and what else is needed" paper
- WJW 3y agoGreat point and sometimes checklists are indeed not enough. My previous post was triggered more by the "I think I did step 13," part of the post I was responding to. That is not a flaw with checklists but a flaw in the safety culture of the operator team. It should never happen that you lose track of where you are in the process because human memory is unreliable, rather you should fix that through better processes and outsource the memorizing to paper.