15 ms·

One thing I've been wondering about is how smaller organizations handle on call. Looking at the roles laid out [1] and assuming an on-call schedule of 4 weeks

by luhn 8y ago

One thing I've been wondering about is how smaller organizations handle on call. Looking at the roles laid out [1] and assuming an on-call schedule of 4 weeks off, 1 week secondary, and 1 week primary, that's a team of at least 25 people. For an organization of just a handful of engineers, how do they make on call work? A single on-call rotation would stretch the team to its limits, and it's likely that certain domains would only have a single Subject Matter Expert. [1] https://response.pagerduty.com/before/different_roles/ https://response.pagerduty.com/before/different_roles/

zufallsheld 8y ago
In my old team we managed our on call rotation for years with five persons. One primary, no secondary, 7 days in a row. Of course we had to arrange ourselves because of vacation and sick leave but we always managed. We also had an organizational rule that forbade being on call for more than ten days in a row or on two consecutive weekends.
- simonsaidit 8y ago
  I was on call for years for a global lottery software and services provider in a 4 man rotation 1 week each. Took people 3-5 years to be ready to take on this responsibility. The last few years i was alone as people left with stress and i was 24/7 365. The pay was good but the day my new boss told me he didet get paid so i could escalate at nite and nobody could find a solution for months i started to look for new oppertunities. They hadent tried to prepare new people for years but managed to get a guy who was not ready to take the call but it also resulted in major incident and data loss. By the time i got out i had constant tinitus so i could barely hear on one ear from the stress so Im pretty happy Im done with it now but Sometimes miss the action ;)
  - fencepost 8y ago
    the day my new boss told me he didet get paid so i could escalate at nite Great, I want to go hourly, or if staying salaried I want to be paid as if hourly. That means I'm paid 33% for all time that I keep myself available while on-call, and paid 2x (or more?) for time spent actually responding because it's after-hours overtime. If I'm going to be 24/7/365 with no backup and get crap about "I'm not paid for you to escalate to me at night," you're going to pay through the nose.
    - simonsaidit 8y ago
      Thats What i told him and it quickly escalated to HR ;) and it was 30% for after hours i dident get extra for responding but was it was often at nite it would sleep in the next day.
ams6110 8y ago
Do everything you can to eliminate the need for on-call support. It sucks, your engineers will quickly grow to hate it and will start looking for new jobs.
- protomyth 8y ago
  One of the big problems is the people making the priority software decisions sometimes forget that a software problem that often requires manual intervention at 2AM will do a lot of damage to a team if left to sit.
- luhn 8y ago
  Certainly everything should be done to reduce the number of incidents, but you can't eliminate the need for on call.
  - deleted 8y ago
    [deleted]
- tetha 8y ago
  I consider it part of my job description to get angry at anything that requires OOH escalation. Well, without drinks and pizza I guess.
dsfyu404ed 8y ago
Give people enough money and few enough incidents that they feel that being on call is a good deal. I'm on a team that's under 20. Nobody complains about being on call because odds are you won't get called and you get a few hundred bucks regardless. That said we don't have to deal with week day on call because half of us are in India and when the US is sleeping they are on call (i.e. their normal workday)
- sbov 8y ago
  Yeah, our last on call incident was like 2 weeks ago. Before that, months ago. A long time ago we had every little thing page us. Plz no. So many false hits. Things that can wait don't page us. It sends an email, and if we see it and have nothing else better to do, we might fix it. Otherwise it waits till work hours. Paging means something needs to be fixed now. I have a friend who gets paged even for "low" (not really low) disk space. Here's a tip: turn up the alerts during work hours so you can resolve that junk before people on call have to deal with it.
  - bigiain 8y ago
    The linked page on Alerting Principle there is good: https://response.pagerduty.com/oncall/alerting_principles/ https://response.pagerduty.com/oncall/alerting_principles/ "We manage how we get alerted based on a simple principle, an alert is something which requires a human to perform an action. Anything else is a notification, which is something that we cannot control, and for which we cannot make any action to affect it. Notifications are useful, but they shouldn't be waking people up under any circumstance." They also distinguish between "needs fixing right now", and "needs to be fixed asap, but can wait until office hours"...
  - outworlder 8y ago
    > I have a friend who gets paged even for "low" (not really low) disk space. I am your friend, I guess. Fighting this battle for a couple of years now. Change is hard. I keep saying that I couldn't care less if CPU has reached 100% in some random server, if customer metrics are unaffected. It is not an emergency. Same for disk. I don't care if it is at 80% full. Page me if it is going to run out before the weekend ends. Or in a month if it's during normal business hours. Were it up to me, I'd tear down all these metrics and replace them with the four golden signals.
outworlder 8y ago
If they are doing their jobs properly: on-call should rarely wake someone up in the middle of the night. If it does, it's for an once in a blue moon issue that wasn't predicted. This is fixed and never happens again. Then in a few months something else happens, and the cycle repeats. If they are handling it poorly: they will overwork the poor sap who gets assigned to oncall (in disfunctional companies, this is likely to be the junior-most members, who don't yet have a say on the matter). Things will be barely patched and people will quit.
donavanm 8y ago
A sustainable weekly oncall rota can work with about 6-7 people, as you noted. To answer your question you collapse roles when appropriate. If you really only have 10 people then maybe it’s simply “primary” and “secondary/escalation” covering all the roles as needed. Get a little larger, to say 3 small teams and you might have the developrs in primary & secondary with all of the technical people managers/team leads/senior ICs in a dedicated “escalation management” rota that handles the “commander”, “scribe”, and “liason” roles. Dedicated support for those specialized roles only really makes sense when you get big enough that coordination becomes its own problem.

by luhn 8y ago

zufallsheld 8y ago

In my old team we managed our on call rotation for years with five persons. One primary, no secondary, 7 days in a row. Of course we had to arrange ourselves because of vacation and sick leave but we always managed. We also had an organizational rule that forbade being on call for more than ten days in a row or on two consecutive weekends.

simonsaidit 8y ago
I was on call for years for a global lottery software and services provider in a 4 man rotation 1 week each. Took people 3-5 years to be ready to take on this responsibility. The last few years i was alone as people left with stress and i was 24/7 365. The pay was good but the day my new boss told me he didet get paid so i could escalate at nite and nobody could find a solution for months i started to look for new oppertunities. They hadent tried to prepare new people for years but managed to get a guy who was not ready to take the call but it also resulted in major incident and data loss. By the time i got out i had constant tinitus so i could barely hear on one ear from the stress so Im pretty happy Im done with it now but Sometimes miss the action ;)
- fencepost 8y ago
  the day my new boss told me he didet get paid so i could escalate at nite Great, I want to go hourly, or if staying salaried I want to be paid as if hourly. That means I'm paid 33% for all time that I keep myself available while on-call, and paid 2x (or more?) for time spent actually responding because it's after-hours overtime. If I'm going to be 24/7/365 with no backup and get crap about "I'm not paid for you to escalate to me at night," you're going to pay through the nose.
  - simonsaidit 8y ago
    Thats What i told him and it quickly escalated to HR ;) and it was 30% for after hours i dident get extra for responding but was it was often at nite it would sleep in the next day.

ams6110 8y ago

Do everything you can to eliminate the need for on-call support. It sucks, your engineers will quickly grow to hate it and will start looking for new jobs.

protomyth 8y ago
One of the big problems is the people making the priority software decisions sometimes forget that a software problem that often requires manual intervention at 2AM will do a lot of damage to a team if left to sit.
luhn 8y ago
Certainly everything should be done to reduce the number of incidents, but you can't eliminate the need for on call.
- deleted 8y ago
  [deleted]
tetha 8y ago
I consider it part of my job description to get angry at anything that requires OOH escalation. Well, without drinks and pizza I guess.

dsfyu404ed 8y ago

Give people enough money and few enough incidents that they feel that being on call is a good deal. I'm on a team that's under 20. Nobody complains about being on call because odds are you won't get called and you get a few hundred bucks regardless. That said we don't have to deal with week day on call because half of us are in India and when the US is sleeping they are on call (i.e. their normal workday)

sbov 8y ago
Yeah, our last on call incident was like 2 weeks ago. Before that, months ago. A long time ago we had every little thing page us. Plz no. So many false hits. Things that can wait don't page us. It sends an email, and if we see it and have nothing else better to do, we might fix it. Otherwise it waits till work hours. Paging means something needs to be fixed now. I have a friend who gets paged even for "low" (not really low) disk space. Here's a tip: turn up the alerts during work hours so you can resolve that junk before people on call have to deal with it.
- bigiain 8y ago
  The linked page on Alerting Principle there is good: https://response.pagerduty.com/oncall/alerting_principles/ https://response.pagerduty.com/oncall/alerting_principles/ "We manage how we get alerted based on a simple principle, an alert is something which requires a human to perform an action. Anything else is a notification, which is something that we cannot control, and for which we cannot make any action to affect it. Notifications are useful, but they shouldn't be waking people up under any circumstance." They also distinguish between "needs fixing right now", and "needs to be fixed asap, but can wait until office hours"...
- outworlder 8y ago
  > I have a friend who gets paged even for "low" (not really low) disk space. I am your friend, I guess. Fighting this battle for a couple of years now. Change is hard. I keep saying that I couldn't care less if CPU has reached 100% in some random server, if customer metrics are unaffected. It is not an emergency. Same for disk. I don't care if it is at 80% full. Page me if it is going to run out before the weekend ends. Or in a month if it's during normal business hours. Were it up to me, I'd tear down all these metrics and replace them with the four golden signals.

outworlder 8y ago

If they are doing their jobs properly: on-call should rarely wake someone up in the middle of the night. If it does, it's for an once in a blue moon issue that wasn't predicted. This is fixed and never happens again. Then in a few months something else happens, and the cycle repeats. If they are handling it poorly: they will overwork the poor sap who gets assigned to oncall (in disfunctional companies, this is likely to be the junior-most members, who don't yet have a say on the matter). Things will be barely patched and people will quit.

donavanm 8y ago

A sustainable weekly oncall rota can work with about 6-7 people, as you noted. To answer your question you collapse roles when appropriate. If you really only have 10 people then maybe it’s simply “primary” and “secondary/escalation” covering all the roles as needed. Get a little larger, to say 3 small teams and you might have the developrs in primary & secondary with all of the technical people managers/team leads/senior ICs in a dedicated “escalation management” rota that handles the “commander”, “scribe”, and “liason” roles. Dedicated support for those specialized roles only really makes sense when you get big enough that coordination becomes its own problem.