4 ms·

I don't have any particular reading material, and my example of the on-call load at AWS that I'm referring to is probably very basic to most people. On my team

I don't have any particular reading material, and my example of the on-call load at AWS that I'm referring to is probably very basic to most people. On my team at AWS, leadership has given specific instruction that we do not believe in on-call runbooks or automation to triage issues, for example. Leadership's reasoning for this is that they think runbooks prevent engineers from applying personal judgement, and every single issue should be handled manually by an engineer on an ad-hoc basis. This leads to a significant amount of on-call time and cognitive load spent doing stuff like verifying the most basic of issues. Even if you have seen the same issue come up for the 1000th time, and even though the previous 999 times it came up the answer was always the same, leadership still insists that the on-call engineer go through a full ad-hoc process of investigating the issue "just to be sure" that this time isn't different. It's a similar situation with documenting our integration guidance for other teams. Our leadership insists that any documented guidance be vague, and that whenever another team wishes to integrate with our software they must schedule meetings with us to discuss even the most basic of design questions. I'm talking very simple stuff like "should you use the HTTPS endpoint to communicate with our service?" where the answer is "yes" 99.99% of the time, and could easily be included in some documentation. But leadership insists that we spend multiple hours per week in meetings to discuss this just in case that 0.01% design comes up.

YZF 5y ago
There is something to be said for this approach. If the root cause is to be fixed someone needs to look at it in depth rather than running some play book procedure to recover. If you have too many problems though you're beyond the point where that helps. Let's say your software has worked flawlessly for a year, no issues, now an issue pops up, the engineers should definitely spend a lot of time understanding it, understanding why it popped up, fixing it properly and fixing the underlying process/org causes that made it pop up. It should not be "follow some playbook to recover". If issues pop up every week this is unsustainable, you're well beyond the point where stuff can actually be fixed. Automation has its own dangers, it is additional software to maintain, it has its own bugs etc. The right amount of automation makes life better for sure.
- awsthro00945 5y ago
  >If the root cause is to be fixed someone needs to look at it in depth The root cause has already been looked at in depth 999 times when the same issue has come up. It's already been RCAed and the fix has been put in the backlog to be implemented sometime next year. In the meantime while we wait for the fix, we will continue to do a full, ad-hoc RCA every time the exact same issue appears, with the exact same results every time, because managers genuinely think it is a valuable way to spend our time. I understand your point, but the relative utopia of a team you're describing is not really the situation I'm talking about. We have on-call periods where the exact same issue will appear 10-20 times per week, and each and every time it is treated as a completely novel issue with an ad-hoc response, even though we already know beforehand what the root cause is and what the fix is. It's an incredible waste of time and contributes significantly to on-call engineers being overloaded, and yet we continue to do it and then are baffled when all of our engineers leave the team due to being overworked. There's also nothing excluding runbooks and root cause analyses from existing together, either. In fact, most good runbooks specifically include steps to determine when an RCA is necessary and how to conduct one. There really is no excuse to not use runbooks as much as possible. If over-reliance on runbooks is having a negative impact due to engineers not applying personal judgement, then that is certainly an issue to be addressed, but the answer is almost never to completely abolish runbooks and documentation.
  - throwaway210620 5y ago
    > the exact same issue will appear 10-20 times per week, and each and every time it is treated as a completely novel issue with an ad-hoc response Yeah, this sounds like a very bad situation where management won't let you do something that reduces ops pain because it isn't the most desirable solution, but they won't let you prioritize the right solution either. The next thing that happens is that on-call folks develop ad-hoc quasi-runbooks and share them amongst a subset of people (or just keep them to themselves to make their own life easier) and those quasi-runbooks become critical to ops, but not documented or shared by everyone. It's pure dysfunction.
  - YZF 5y ago
    This does sound pretty dysfunctional. You'd think that for something that's causing 999 on-calls getting the root cause fixed would be a priority. What I described obviously falls apart when the team has no ability to actually fix issues. Perhaps the original intent was to get those issues fixed but that somehow got lost as the org grows larger.
wikibob 5y ago
It is absolutely fascinating how wrong this approach is. Every single issue that comes up in on-call should be evaluated under the lens of “does fixing this absolutely require human judgement, or can it be automated, ideally by fixing the code in the main system. If it does require human judgement, are there ways to redesign so that is no longer true?”

YZF 5y ago

There is something to be said for this approach. If the root cause is to be fixed someone needs to look at it in depth rather than running some play book procedure to recover. If you have too many problems though you're beyond the point where that helps. Let's say your software has worked flawlessly for a year, no issues, now an issue pops up, the engineers should definitely spend a lot of time understanding it, understanding why it popped up, fixing it properly and fixing the underlying process/org causes that made it pop up. It should not be "follow some playbook to recover". If issues pop up every week this is unsustainable, you're well beyond the point where stuff can actually be fixed. Automation has its own dangers, it is additional software to maintain, it has its own bugs etc. The right amount of automation makes life better for sure.

awsthro00945 5y ago
>If the root cause is to be fixed someone needs to look at it in depth The root cause has already been looked at in depth 999 times when the same issue has come up. It's already been RCAed and the fix has been put in the backlog to be implemented sometime next year. In the meantime while we wait for the fix, we will continue to do a full, ad-hoc RCA every time the exact same issue appears, with the exact same results every time, because managers genuinely think it is a valuable way to spend our time. I understand your point, but the relative utopia of a team you're describing is not really the situation I'm talking about. We have on-call periods where the exact same issue will appear 10-20 times per week, and each and every time it is treated as a completely novel issue with an ad-hoc response, even though we already know beforehand what the root cause is and what the fix is. It's an incredible waste of time and contributes significantly to on-call engineers being overloaded, and yet we continue to do it and then are baffled when all of our engineers leave the team due to being overworked. There's also nothing excluding runbooks and root cause analyses from existing together, either. In fact, most good runbooks specifically include steps to determine when an RCA is necessary and how to conduct one. There really is no excuse to not use runbooks as much as possible. If over-reliance on runbooks is having a negative impact due to engineers not applying personal judgement, then that is certainly an issue to be addressed, but the answer is almost never to completely abolish runbooks and documentation.
- throwaway210620 5y ago
  > the exact same issue will appear 10-20 times per week, and each and every time it is treated as a completely novel issue with an ad-hoc response Yeah, this sounds like a very bad situation where management won't let you do something that reduces ops pain because it isn't the most desirable solution, but they won't let you prioritize the right solution either. The next thing that happens is that on-call folks develop ad-hoc quasi-runbooks and share them amongst a subset of people (or just keep them to themselves to make their own life easier) and those quasi-runbooks become critical to ops, but not documented or shared by everyone. It's pure dysfunction.
- YZF 5y ago
  This does sound pretty dysfunctional. You'd think that for something that's causing 999 on-calls getting the root cause fixed would be a priority. What I described obviously falls apart when the team has no ability to actually fix issues. Perhaps the original intent was to get those issues fixed but that somehow got lost as the org grows larger.

wikibob 5y ago

It is absolutely fascinating how wrong this approach is. Every single issue that comes up in on-call should be evaluated under the lens of “does fixing this absolutely require human judgement, or can it be automated, ideally by fixing the code in the main system. If it does require human judgement, are there ways to redesign so that is no longer true?”