4 ms·

I work in devops support for an under-development software project. At request of the developers, I started keeping up a page in our wiki where, whenever someth

I work in devops support for an under-development software project. At request of the developers, I started keeping up a page in our wiki where, whenever something broke that affected the developers, I just logged what happened, why, and what I'm doing to prevent it happening again. The developers really like this so they feel more in the loop about system issues, but it turns out that I really like it because it's just really useful to have that log. I don't want to keep the wiki page super busy, so I now keep a text file with basically a much more verbose version of the same thing. What happened, what I tried, what I eventually found was the root cause, how I fixed it, and what I'm changing to prevent a repeat. This has two big advantages. First, the obvious one is that when things act up I can sometimes find details on a previous incident that I don't remember very well. The second and I think even greater advantage, though, is that it forces me to actually think seriously about the root cause and preventative measures every single time, instead of just fixing the immediate problem and then going on to the next thing in a rush.

yjftsjthsd-h 9y ago
Mine is a text for called "problems" that lists things that I've has to fix, each with a symptom list (including error codes and log output so it's easy to search), root cause where known, steps to fix, and date+time of every known occurrence. Partially helpful because some errors reoccurr.
karlding 9y ago
That's basically what postmortems are! For those unfamiliar with the term, postmortems are a written account of an incident, its impact, the actions taken to resolve resolve it, root causes, and identifying any follow-up actions to prevent the incident from recurring. I think postmortems should be a valuable learning opportunity for the entire company, and are by no means a punishment. In an organization that fosters "blameless culture," this allows the organization to grow and to identify and fix a weakness. Google's SRE book [0] also talks a bit about this in the chapter about Postmortems. [0] https://landing.google.com/sre/book https://landing.google.com/sre/book

yjftsjthsd-h 9y ago

Mine is a text for called "problems" that lists things that I've has to fix, each with a symptom list (including error codes and log output so it's easy to search), root cause where known, steps to fix, and date+time of every known occurrence. Partially helpful because some errors reoccurr.

karlding 9y ago

That's basically what postmortems are! For those unfamiliar with the term, postmortems are a written account of an incident, its impact, the actions taken to resolve resolve it, root causes, and identifying any follow-up actions to prevent the incident from recurring. I think postmortems should be a valuable learning opportunity for the entire company, and are by no means a punishment. In an organization that fosters "blameless culture," this allows the organization to grow and to identify and fix a weakness. Google's SRE book [0] also talks a bit about this in the chapter about Postmortems. [0] https://landing.google.com/sre/book https://landing.google.com/sre/book