4 ms·

>>> A good schedule has 6–8 engineers. Stop right there. The minimum is 10-12 engineers to have a sane on-call schedule and they have to be distributed across

>>> A good schedule has 6–8 engineers. Stop right there. The minimum is 10-12 engineers to have a sane on-call schedule and they have to be distributed across the world (timezone). With 6 people you're going to be oncall almost every week. That's practically only 4 people on rota because 1-2 people are not participating in rota (last joiners are not yet trained and other reasons). Then when there are issues actually happening, they are getting escalated to the person on rota and then escalated to you/team (it's rare than one man can fix/debug much alone), so you're forced to work even when off rota.

dilyevsky 6y ago
Oncalls are usually week long so you’re oncall once every 1.5 months with six people
- user5994461 6y ago
  That's the point, in theory it's every 6 weeks, in practice it's every 2-3 weeks. Because of people not participating in oncall equally and because of getting escalations while you were not officially on call.
  - madhadron 6y ago
    > people not participating in oncall equally Why would this be? If you're on the rotation, you have the same number of shifts as anyone else. If you can't take your shift you have to swap with someone and take their shift instead. > getting escalations while you were not officially on call This usually means that the training of those on the rotation is inadequate. It's possible for oncalls to be awful, but they don't have to be. The important part is to make those who have the power to change the sources of pain to suffer it.
  - dilyevsky 6y ago
    If people are not participating then it’s not 6 person rotation. Every oncall tool out there is capable of setting even schedule that is reviewed and agreed in advance in every case that I’ve seen. Regarding escalations if your oncall primary frequently requires escalating to other folks you haven’t done a good job of training them or writing runbooks for your component.
  - sethammons 6y ago
    I’m on a team where 6 people are on rotation. We are nearly always 6 weeks until your next rotation. On some cases, you may swap, and then maybe it was 5wks or 7wks since your last rotation. For some smaller teams, they may have fewer people. We try to staff up those teams or otherwise work to reduce the on call burden.
bobbiechen 6y ago
I would interpret the 6-8 number as the number of people actually on the rotation, so excluding those who are not yet trained. >they are getting escalated to the person on rota and then escalated to you/team (it's rare than one man can fix/debug much alone) This is probably true at first, but over time a team can build up knowledge with better runbooks. When I first joined my team's on-call, my tech lead was very clear that if I wasn't sure what to do, I should just page her or the subject matter expert on the team without hesitating. For each time this happened, we went over the response in a postmortem and added instructions about how to diagnose and fix to the runbook, so the next person to get paged for a similar reason could follow those instead.
- user5994461 6y ago
  I agree with what you say on principles, the challenge is doing it in the real world with real people and real organizations. You're practically saying that the tech lead and manager are on call 24/7 because they can be called anytime. Are they okay with that? can they actually do something about the pages/runbooks? (in many organizations it's not that simple). What's the average tenure in tech? Something like a year (you can surely imagine that it gets shorter with bad oncall). You're constantly having new joiners (and leavers). It's not as simple as there being 6 people at the ready at all times. Practically the team starts from zero and have to ramp up to 6 members, how easy is it to recruit 6 dev/ops/sre? It's not that easy and it takes a lot of time and they leave. Outside of a few large organization, teams/department might never reach the size where on call is bearable.

dilyevsky 6y ago

Oncalls are usually week long so you’re oncall once every 1.5 months with six people

user5994461 6y ago
That's the point, in theory it's every 6 weeks, in practice it's every 2-3 weeks. Because of people not participating in oncall equally and because of getting escalations while you were not officially on call.
- madhadron 6y ago
  > people not participating in oncall equally Why would this be? If you're on the rotation, you have the same number of shifts as anyone else. If you can't take your shift you have to swap with someone and take their shift instead. > getting escalations while you were not officially on call This usually means that the training of those on the rotation is inadequate. It's possible for oncalls to be awful, but they don't have to be. The important part is to make those who have the power to change the sources of pain to suffer it.
- dilyevsky 6y ago
  If people are not participating then it’s not 6 person rotation. Every oncall tool out there is capable of setting even schedule that is reviewed and agreed in advance in every case that I’ve seen. Regarding escalations if your oncall primary frequently requires escalating to other folks you haven’t done a good job of training them or writing runbooks for your component.
- sethammons 6y ago
  I’m on a team where 6 people are on rotation. We are nearly always 6 weeks until your next rotation. On some cases, you may swap, and then maybe it was 5wks or 7wks since your last rotation. For some smaller teams, they may have fewer people. We try to staff up those teams or otherwise work to reduce the on call burden.

bobbiechen 6y ago

I would interpret the 6-8 number as the number of people actually on the rotation, so excluding those who are not yet trained. >they are getting escalated to the person on rota and then escalated to you/team (it's rare than one man can fix/debug much alone) This is probably true at first, but over time a team can build up knowledge with better runbooks. When I first joined my team's on-call, my tech lead was very clear that if I wasn't sure what to do, I should just page her or the subject matter expert on the team without hesitating. For each time this happened, we went over the response in a postmortem and added instructions about how to diagnose and fix to the runbook, so the next person to get paged for a similar reason could follow those instead.

user5994461 6y ago
I agree with what you say on principles, the challenge is doing it in the real world with real people and real organizations. You're practically saying that the tech lead and manager are on call 24/7 because they can be called anytime. Are they okay with that? can they actually do something about the pages/runbooks? (in many organizations it's not that simple). What's the average tenure in tech? Something like a year (you can surely imagine that it gets shorter with bad oncall). You're constantly having new joiners (and leavers). It's not as simple as there being 6 people at the ready at all times. Practically the team starts from zero and have to ramp up to 6 members, how easy is it to recruit 6 dev/ops/sre? It's not that easy and it takes a lot of time and they leave. Outside of a few large organization, teams/department might never reach the size where on call is bearable.