> An on-call training program is essential to building and maintaining a successful incident response practice and culture. I think that employee training in g

> An on-call training program is essential to building and maintaining a successful incident response practice and culture. I think that employee training in general is essential to the overall efficiency and quality of your business. Companies always skimp on it, and you can see how it hurts productivity and the bottom line. But it's not necessarily going to impact your culture or day to day practices. The thing that's going to maintain a successful incident response practice is follow-through. Training just tells them what they're supposed to do. But if they're not doing it, training doesn't matter. What's the most common problem with on-call? It's not waking people up in the middle of the night, or a lack of runbooks, or lack of postmortems. It's the follow-through. When you have an incident, and you "deal with it", and even have a postmortem, what's the last thing on the page? "Action Items" or "Next Steps". And are those action items actually getting done? Is a regular review of your practices getting done? Is the same alert popping up again and again? That is the biggest problem with on-call, and training doesn't solve that. To run a good incident response process you need people who are getting paid to do two things: 1) due diligence, 2) continuous improvement. 1) Give someone the primary responsibility of ensuring everyone is following the process. Do not let action items stagnate. Do not let weeks go by without writing the postmortem. Do not have alerts without resolutions, or runbooks for that matter. Somebody needs to get paid to do this, because it's boring as hell, not fun, but needs to be done. 2) Make sure the process is getting easier, and better. You don't want to have to check up on everything all the time, of course, and it doesn't scale well. So you want to improve your process so that it's more likely that the right things get done the first time. You can train your folks what to do. And you can "get by" with an incident response process that nobody actually follows and is never improved. But as incidents mount, it will drain your people and org more and more. By all means, if you're currently facing a deluge of incidents, do training! But the due diligence and continuous improvement is what's actually gonna push incident numbers down.