4 ms·

I did pager duty (on-call) for 8 years in my last job as part of professional services team. I actually enjoyed it as I thrive in stress situations. We had a go

by akg_67 12y ago

I did pager duty (on-call) for 8 years in my last job as part of professional services team. I actually enjoyed it as I thrive in stress situations. We had a good team and boss that also helped. Some of the suggestions based on my experience: 1. Make sure there are enough team members are in on-call rotation so that you get your 1 week on-call every 6 to 8 weeks or more. If on-call is too frequent, it will be disruptive to your normal life and you and your family will resent the job. 2. If your on-call only requires remote phone/access support, make sure company picks the tab for your phone and mobile internet. If, like mine, on-call requires onsite visit, company is properly compensating for mileage and auto-expense. Also get company to pay for on-call either in cash or with time-off. You can also work these out informally within your team and boss. My company paid for my cell service, home internet, and provided auto allowance. 3. You should have a place in your house where you can quickly go, talk, and work in the middle of the night without disturbing rest of the family. 4. Make sure your team and boss are okay with you coming to work late or skipping days coming to office when you are on-call and receive calls in the middle of night. My worse on-calls used to be woken up between 2:00 - 4:00 AM when I was typically in deep sleep. 5. Avoid scheduling anything important during the on-call week. And, let everyone know that you may have drop everything else if you receive a call. 6. During the on-call week relax, don't take too much stress, don't do too much of regular work, don't force yourself to have a normal day-and-night, go with the flow. 7. Avoid going to places like movie theater where you can't take phone call and quickly get out of. 8. Don't get anxious during on-call week. I had co-workers who used to have panic attack during the on-call week.

jmsduran 12y ago
It seems, especially for major corporations, that on-call/pager duty is quickly becoming the norm for software development teams. I do agree that pager duty is a symptom of a fundamental flaw within the system/architecture. I think it would be in a company's best interest to devote time in improving the reliability and stability of their infrastructure, instead of relying on the band-aid approach that pager duty seems to be. Regarding #8 though, when you are pressured to resolve a complex issue within a short time window, it can absolutely induce a sense of panic for those who do not handle stress well. In my opinion, I believe the remedy for this would be to have two individuals designated as on-call at a time, assuming the team is large enough.
- devicenull 12y ago
  > It seems, especially for major corporations, that on-call/pager duty is quickly becoming the norm for software development teams. I do agree that pager duty is a symptom of a fundamental flaw within the system/architecture. I think it would be in a company's best interest to devote time in improving the reliability and stability of their infrastructure, instead of relying on the band-aid approach that pager duty seems to be. I can't see there ever being a time where there is no on-call requirement. You always need someone standing by in case of some terrible disaster that cannot be handled automatically. Better to have this a formal responsibility that never gets used, then to not have it and end up with an extended downtime because you can't contact anyone. That being said, if you're getting paged continuously during on-call, then there's a bigger problem that needs to be resolved.
  - _delirium 12y ago
    > You always need someone standing by in case of some terrible disaster that cannot be handled automatically. If it's a really terrible disaster, a once-a-decade kind of thing where everything goes haywire and you need as many staff as possible to get online ASAP, then yes. But aren't we talking more about the kinds of "disasters" that happen once a month or so, and can be handled by a few staff (not waking up the whole team). To me that sounds more like just staffing for normal operations. At large engineering companies this is typically handled via literally having someone standing by, i.e. formally on duty, rather than having off-duty employees be on pager duty. There'll be at least a bare-bones staff on the after-hours shift (probably not in all offices, but in some kind of 24/7 operations center), enough of a staff that reasonably foreseeable things can be handled. Of course there are some pros and cons to that from an employee perspective. On the one hand the night shift isn't that pleasant, but on the other hand your responsibilities are at least formally limited to 40 hours/wk; if you're on night shift one week, you don't come in during the day, or carry a pager during the day.
    - pavel_lishin 12y ago
      That seems pretty wasteful if emergencies are rare. We have three people on-call on my team, and we typically have an issue at most once a month - and so far, in 95% of cases, the issue can be resolved by killing an errant ec2 instance and waiting for its replacement to spin up in 5 minutes. It would be much more disruptive and annoying if I had to work the graveyard shift even once every two months or so; aside from shifting my sleep schedule once every two months, it would be a week where I would probably be fairly unproductive.
TheSwordsman 12y ago
To echo akg, I've also been in experiences where the boss and team make the experience great. It's great if you like working on small tasks that are truly dynamic. Sometimes, getting paged allows me to get my head clear of your normal responsibilities. I kind of use the pages as a nice refresher for my regular work. However, personal preference. To comment on one of akg's points, the length of the schedule rotation can actually be a problem. Longer than 6-8, and you are then very detatched from the operational issues your team is having. Make sure you link up with the previous primary to get the run down on anything (and maybe even the person who was primary before her). You definitely want to get more familiar with the team you will be rotating with, but: 1: Do not fear escalating to secondary / asking for help when you need it. While getting paged sucks, every team I've been on this has been one of the biggest guidelines. We would rather someone ask for help sooner, than do something they are unsure of / delay fixing something. 2 (More like an extension of akg's #6): This is something to clear with your boss, but when I'm on call I also try to put the engineering effort in to fix alarms that are less than ideal. If I get paged at 3AM for some bullshit reason, I will spend the time the next day to make it alert when there is something actionable to be done. If you are at a startup it's probably unlikely you'll be reimbursed for your cell phone (receiving calls). However, they should have provisions for mobile internet. Using your personal cell phone for work-related activities could have interesting legal implications. If the company is getting investigated, your personal phone could (in theory) be included as work-related things have gone through it. I've been on pager duty for a few years now. I've no regrets. However, I'm sure there's one day in my life where I'll be over it. Best advice I can give while on call: keep calm and have fun.

by akg_67 12y ago

jmsduran 12y ago

It seems, especially for major corporations, that on-call/pager duty is quickly becoming the norm for software development teams. I do agree that pager duty is a symptom of a fundamental flaw within the system/architecture. I think it would be in a company's best interest to devote time in improving the reliability and stability of their infrastructure, instead of relying on the band-aid approach that pager duty seems to be. Regarding #8 though, when you are pressured to resolve a complex issue within a short time window, it can absolutely induce a sense of panic for those who do not handle stress well. In my opinion, I believe the remedy for this would be to have two individuals designated as on-call at a time, assuming the team is large enough.

devicenull 12y ago
> It seems, especially for major corporations, that on-call/pager duty is quickly becoming the norm for software development teams. I do agree that pager duty is a symptom of a fundamental flaw within the system/architecture. I think it would be in a company's best interest to devote time in improving the reliability and stability of their infrastructure, instead of relying on the band-aid approach that pager duty seems to be. I can't see there ever being a time where there is no on-call requirement. You always need someone standing by in case of some terrible disaster that cannot be handled automatically. Better to have this a formal responsibility that never gets used, then to not have it and end up with an extended downtime because you can't contact anyone. That being said, if you're getting paged continuously during on-call, then there's a bigger problem that needs to be resolved.
- _delirium 12y ago
  > You always need someone standing by in case of some terrible disaster that cannot be handled automatically. If it's a really terrible disaster, a once-a-decade kind of thing where everything goes haywire and you need as many staff as possible to get online ASAP, then yes. But aren't we talking more about the kinds of "disasters" that happen once a month or so, and can be handled by a few staff (not waking up the whole team). To me that sounds more like just staffing for normal operations. At large engineering companies this is typically handled via literally having someone standing by, i.e. formally on duty, rather than having off-duty employees be on pager duty. There'll be at least a bare-bones staff on the after-hours shift (probably not in all offices, but in some kind of 24/7 operations center), enough of a staff that reasonably foreseeable things can be handled. Of course there are some pros and cons to that from an employee perspective. On the one hand the night shift isn't that pleasant, but on the other hand your responsibilities are at least formally limited to 40 hours/wk; if you're on night shift one week, you don't come in during the day, or carry a pager during the day.
  - pavel_lishin 12y ago
    That seems pretty wasteful if emergencies are rare. We have three people on-call on my team, and we typically have an issue at most once a month - and so far, in 95% of cases, the issue can be resolved by killing an errant ec2 instance and waiting for its replacement to spin up in 5 minutes. It would be much more disruptive and annoying if I had to work the graveyard shift even once every two months or so; aside from shifting my sleep schedule once every two months, it would be a week where I would probably be fairly unproductive.

TheSwordsman 12y ago

To echo akg, I've also been in experiences where the boss and team make the experience great. It's great if you like working on small tasks that are truly dynamic. Sometimes, getting paged allows me to get my head clear of your normal responsibilities. I kind of use the pages as a nice refresher for my regular work. However, personal preference. To comment on one of akg's points, the length of the schedule rotation can actually be a problem. Longer than 6-8, and you are then very detatched from the operational issues your team is having. Make sure you link up with the previous primary to get the run down on anything (and maybe even the person who was primary before her). You definitely want to get more familiar with the team you will be rotating with, but: 1: Do not fear escalating to secondary / asking for help when you need it. While getting paged sucks, every team I've been on this has been one of the biggest guidelines. We would rather someone ask for help sooner, than do something they are unsure of / delay fixing something. 2 (More like an extension of akg's #6): This is something to clear with your boss, but when I'm on call I also try to put the engineering effort in to fix alarms that are less than ideal. If I get paged at 3AM for some bullshit reason, I will spend the time the next day to make it alert when there is something actionable to be done. If you are at a startup it's probably unlikely you'll be reimbursed for your cell phone (receiving calls). However, they should have provisions for mobile internet. Using your personal cell phone for work-related activities could have interesting legal implications. If the company is getting investigated, your personal phone could (in theory) be included as work-related things have gone through it. I've been on pager duty for a few years now. I've no regrets. However, I'm sure there's one day in my life where I'll be over it. Best advice I can give while on call: keep calm and have fun.