6 ms·

I carried a pager for a short while, pager, remember those? On-call weeks were hell, even when nothing happened. It's the expectation of getting the call, and

I carried a pager for a short while, pager, remember those? On-call weeks were hell, even when nothing happened. It's the expectation of getting the call, and the expectation that a system outside your primary domain of expertise will be down. I wasn't paid enough to weather the stress, and I don't think there's a hard number to suit every one. If I ever had input to how an on-call rotation should be handled, I'd make it a pair of on-call persons. They'd switch off as they please, but there will always be 2 persons coordinating the response. One person could come in, with the other person on the phone, or both. It's the loneliness. Somewhere in this vast organization connected with pipes that no one single person has put together, some element would break. Mercifully the impact ripples and you kinda know about the subsystems to track down the leak and patch it up and alert the domain owners for the next working day. You're the person they're going to call, there's no cavalry, it's you looking at incomprehensible gibberish while the rest of the world is asleep. Even the desk who has called you doesn't get involved, they're done. For me, the idea that it's just me until it's fixed (or degrades further), was the hardest thing to cope with. I'd pair up my ideal on-call team. It'll cost more, but I'll pay it. Incidentally pagers are still being carried in SCIFs, where there are no mobiles and sometimes no outside line.

bob1029 6y ago
I was also on pager duty with physical pagers many years ago. The perspective I had was that it was a rite of passage for a systems engineer. I noticed that only the more junior engineers were on front-line duty. I believe the thinking was that by forcing junior engineers to go through pager duty, they would be much more diligent when they propose, create or approve changes down the road. Whether or not this is a perverse incentive which ultimately ruins productivity & team morale is a separate conversation... We had a primary & backup who got paged at the same time (due to the financial implications of timing), so there wasn't a total sense of isolation. But, there was definitely that sense of helplessness when it turned out to be a problem in a totally different system (and country). Worrying about something going down definitely kept you on a tight leash with regard to WiFi and laptop accessibility. We did get a little wild with the wording on the policy. Some bars downtown had excellent WiFi, so when some of us were on call during Friday/Saturday we would just make sure we had a laptop in the car and pager on body. Definitely not the best situation to be in, but for some teams the amount of time you were on pager duty could be measured in years.
sershe 6y ago
The usual way to handle the "different system" issue is to have on-calls in each team, so you could loop them in for their stuff or even transfer the ticket and go back to sleep; and on-call-focused documentation within teams so you can have some notion of every component that they are not intimately familiar with. I hate on-call; it doesn't stress me out but I'm generally very protective of my personal time, especially weekends. However, I actually found my greatest on-call highs were from figuring out issues in other people's stuff that was 100% unfamiliar to me, sometime at 4am. In the most memorable case of Rogue Night-Time CPU Spike I didn't even have source code access :)
- steveBK123 6y ago
  I have found in my career that being too good at on-call/L3 escalations only makes them come to you more. Teams that have slopey shoulders and phones that seem to always go to voicemail.. don’t get reprimanded, and your team gets to keep playing hero.
  - devchix 6y ago
    This is an observable truth. Be good, but don't show that you're too good. You'll get called all the time, even for stuff you're not good at. You'll be the janitor for all kinds of idiotic mess, and the perverse incentive will make them call and rely on you more, and you get less of your work and what's important to you done. Here's a joke that'll provoke cynical laughter: What do you get for winning a pie eating contest? More pies. The world breaks everyone.
    - sershe 6y ago
      I suppose that could be true in small organizations... not a problem in a more regimented on-call setup. My phone is on alarms-only at night when I'm not on-call, so they cannot physically call me :) I suppose people could still ask during the day, but generally in a good on-call culture it's frowned upon (unless you own the component or it's an all-hands-on-deck emergency where extra hands are actually helpful).

bob1029 6y ago

I was also on pager duty with physical pagers many years ago. The perspective I had was that it was a rite of passage for a systems engineer. I noticed that only the more junior engineers were on front-line duty. I believe the thinking was that by forcing junior engineers to go through pager duty, they would be much more diligent when they propose, create or approve changes down the road. Whether or not this is a perverse incentive which ultimately ruins productivity & team morale is a separate conversation... We had a primary & backup who got paged at the same time (due to the financial implications of timing), so there wasn't a total sense of isolation. But, there was definitely that sense of helplessness when it turned out to be a problem in a totally different system (and country). Worrying about something going down definitely kept you on a tight leash with regard to WiFi and laptop accessibility. We did get a little wild with the wording on the policy. Some bars downtown had excellent WiFi, so when some of us were on call during Friday/Saturday we would just make sure we had a laptop in the car and pager on body. Definitely not the best situation to be in, but for some teams the amount of time you were on pager duty could be measured in years.

sershe 6y ago

The usual way to handle the "different system" issue is to have on-calls in each team, so you could loop them in for their stuff or even transfer the ticket and go back to sleep; and on-call-focused documentation within teams so you can have some notion of every component that they are not intimately familiar with. I hate on-call; it doesn't stress me out but I'm generally very protective of my personal time, especially weekends. However, I actually found my greatest on-call highs were from figuring out issues in other people's stuff that was 100% unfamiliar to me, sometime at 4am. In the most memorable case of Rogue Night-Time CPU Spike I didn't even have source code access :)

steveBK123 6y ago
I have found in my career that being too good at on-call/L3 escalations only makes them come to you more. Teams that have slopey shoulders and phones that seem to always go to voicemail.. don’t get reprimanded, and your team gets to keep playing hero.
- devchix 6y ago
  This is an observable truth. Be good, but don't show that you're too good. You'll get called all the time, even for stuff you're not good at. You'll be the janitor for all kinds of idiotic mess, and the perverse incentive will make them call and rely on you more, and you get less of your work and what's important to you done. Here's a joke that'll provoke cynical laughter: What do you get for winning a pie eating contest? More pies. The world breaks everyone.
  - sershe 6y ago
    I suppose that could be true in small organizations... not a problem in a more regimented on-call setup. My phone is on alarms-only at night when I'm not on-call, so they cannot physically call me :) I suppose people could still ask during the day, but generally in a good on-call culture it's frowned upon (unless you own the component or it's an all-hands-on-deck emergency where extra hands are actually helpful).