20 ms·
Can you share your knowledge or reading materials for how to reduce on-call load? I’ve worked at a number of big companies but all the problems driving the onc
by vlovich123 5y ago
Can you share your knowledge or reading materials for how to reduce on-call load?
I’ve worked at a number of big companies but all the problems driving the oncall load seemed, at best, domain specific if not application specific with highly variable fix times and unpredictable occurrence (eg started becoming more of a problem due to unrelated change X). As a result each team has to decide the cost of fixing the pain vs focusing on other things.
If there’s actually best-practices here that help that we’re not already doing, I’d be extremely eager to learn about them. I’m not an Amazon engineer but I’ve been bitten by oncall stuff.
- awsthro00945 5y agoI don't have any particular reading material, and my example of the on-call load at AWS that I'm referring to is probably very basic to most people. On my team at AWS, leadership has given specific instruction that we do not believe in on-call runbooks or automation to triage issues, for example. Leadership's reasoning for this is that they think runbooks prevent engineers from applying personal judgement, and every single issue should be handled manually by an engineer on an ad-hoc basis. This leads to a significant amount of on-call time and cognitive load spent doing stuff like verifying the most basic of issues. Even if you have seen the same issue come up for the 1000th time, and even though the previous 999 times it came up the answer was always the same, leadership still insists that the on-call engineer go through a full ad-hoc process of investigating the issue "just to be sure" that this time isn't different. It's a similar situation with documenting our integration guidance for other teams. Our leadership insists that any documented guidance be vague, and that whenever another team wishes to integrate with our software they must schedule meetings with us to discuss even the most basic of design questions. I'm talking very simple stuff like "should you use the HTTPS endpoint to communicate with our service?" where the answer is "yes" 99.99% of the time, and could easily be included in some documentation. But leadership insists that we spend multiple hours per week in meetings to discuss this just in case that 0.01% design comes up.
- YZF 5y agoThere is something to be said for this approach. If the root cause is to be fixed someone needs to look at it in depth rather than running some play book procedure to recover. If you have too many problems though you're beyond the point where that helps. Let's say your software has worked flawlessly for a year, no issues, now an issue pops up, the engineers should definitely spend a lot of time understanding it, understanding why it popped up, fixing it properly and fixing the underlying process/org causes that made it pop up. It should not be "follow some playbook to recover". If issues pop up every week this is unsustainable, you're well beyond the point where stuff can actually be fixed. Automation has its own dangers, it is additional software to maintain, it has its own bugs etc. The right amount of automation makes life better for sure.
- awsthro00945 5y ago>If the root cause is to be fixed someone needs to look at it in depth The root cause has already been looked at in depth 999 times when the same issue has come up. It's already been RCAed and the fix has been put in the backlog to be implemented sometime next year. In the meantime while we wait for the fix, we will continue to do a full, ad-hoc RCA every time the exact same issue appears, with the exact same results every time, because managers genuinely think it is a valuable way to spend our time. I understand your point, but the relative utopia of a team you're describing is not really the situation I'm talking about. We have on-call periods where the exact same issue will appear 10-20 times per week, and each and every time it is treated as a completely novel issue with an ad-hoc response, even though we already know beforehand what the root cause is and what the fix is. It's an incredible waste of time and contributes significantly to on-call engineers being overloaded, and yet we continue to do it and then are baffled when all of our engineers leave the team due to being overworked. There's also nothing excluding runbooks and root cause analyses from existing together, either. In fact, most good runbooks specifically include steps to determine when an RCA is necessary and how to conduct one. There really is no excuse to not use runbooks as much as possible. If over-reliance on runbooks is having a negative impact due to engineers not applying personal judgement, then that is certainly an issue to be addressed, but the answer is almost never to completely abolish runbooks and documentation.
- wikibob 5y agoIt is absolutely fascinating how wrong this approach is. Every single issue that comes up in on-call should be evaluated under the lens of “does fixing this absolutely require human judgement, or can it be automated, ideally by fixing the code in the main system. If it does require human judgement, are there ways to redesign so that is no longer true?”
- YZF 5y agoI'm not the parent but lemme chime in on this topic. It's pretty simple, if you don't build crappy software you won't get a heavy on-call load. You're really asking how to build great software. Build strong teams, with experienced people, follow good practices, reward quality and stability and not features or lines of code, reduce complexity, etc. etc. I've worked on software used by millions of people with a very low problem rate and then I worked on software used by hundreds of people where nothing ever works. Often in the latter the team, through lack of experience or ability, assumes that this is just the way all software is. There's plenty of examples of widely used software systems that are generally quite reliable and well built, and there's plenty of examples of stuff that's garbage, held together by duct tape, works by chance.
- rossjudson 5y agoThis is oversimplification. One reality is more like "You are handed a system with a heavy on-call load. Make it better." The other is "The system you built was great with 1e9 load, but now we're heading for 1e12 load. Make it better."
- walshemj 5y agoSometimes throw it away and start again is the right answer
- throwaway210620 5y agoIn my experience: 1) Make a goal: we should get paged for at most N incidents per week. That goal should be closer to 0 than 10 IMO. 2) Track stats on this goal, both in aggregate and broken down by cases you think you can address separately. Example: tickets from alarms vs tickets filed by people. Alarms having to do with external dependencies vs alarms caused by your own bugs. Don't just make this a "it seems like we've had fewer pages lately" thing. Real numbers. 3) Review these stats on a graph every week. Someone should have an explanation of why they have spiked, why they haven't dropped, the breakdown of problem type, etc. There should be congratulations when they drop and a request for plans when they don't. 4) have management that can communicate upwards to leadership that ops improvement is a priority for your team and that you ultimately won't be able to continue other feature development if you are always mired in ops pain and people are either busy with mundanity or driven to leave the team. 5) dedicate time in each sprint to working on the most recent identified target from plans made in #3. This isn't particularly complicated, and typing it out almost sounds like I'm giving you worthless common sense advice, but I think the key here is that multiple levels of your organization need to commit to making this important enough to spend time reviewing it, agree it is a priority, and put actual dedicated dev work into it. edit: formatted so indented list is readable on mobile. I have no idea how to do this without a code block. HN, please make this easier :)
- wikibob 5y agoThere is a lot of SRE content about fixing on-call. My short summary is, fixing on-call is a HUMAN problem, not a technical engineering problem. Here’s an excellent place to start: https://monitoring.love/articles/how-to-improve-on-call/ https://monitoring.love/articles/how-to-improve-on-call/ There is a wonderful list of talks and resources there, and those will lead you to yet more concepts and ideas to research.
- yourapostasy 5y ago> If there’s actually best-practices here that help that we’re not already doing... Unless you have a leadership that recognizes the time sink on-call generates, all the best practices in the world are not going to help. This goes back to leadership culture. If they aren't in the trenches taking the same rotations with the on-call staff and at least simply staying up with them when the calls come in, then they will need metrics they believe to understand the deleterious effects of excessive on-call. This doesn't even begin to touch upon excessive on-call environments strongly signalling ignored technical and/or product debt that drags down new feature implementation and innovation. Leadership that truly comprehends this will allocate significant (20-25%) of time to addressing it, and push back on feature demands. If you get to that point, then 80% of your struggle is over. Once you have leadership backing bringing the necessary time and resources to bear, reducing on-call is mostly lots of bookkeeping and documentation to identify trends and commonalities to prioritize and fix. A lot of the specific implementation details depend upon what you have available to leverage; ideally put someone in the PM role of tracking details for weekly stand-ups who is a relentless detailed note-taker and follow-upper.