4 ms·

I get it, trying to make on-call not sound so bad. Fair enough. But then she goes into the "example week" including a weekend page. It seems like there's not m

I get it, trying to make on-call not sound so bad. Fair enough. But then she goes into the "example week" including a weekend page. It seems like there's not much going on, not many pages, not much out of regular hours. And then this prime example of why you do not want to have engineers on-call: At 5 a.m., I got paged. I jumped out of bed and ran over to my computer. The page self-resolved at 5:01 a.m. She puts a smiley but this can really drive one mad. Second time this happens (and let's face it, it happens, even if you tweak and adjust those settings over and over) I'm out of there. I get it, all the talk about "you will build better services because otherwise you will be paged". In the end I think it's just about saving money for dedicated 24/7 network operations if that is what your business requires. If it doesn't have customers 24/7, then don't set up any pagers and support staff, if it does, pay for it! I've been on call in a SaaS environment before. In my entire time (counted in years) doing that I have had about the same amount of pages in total (!) as are contained in her example week. I carried my laptop and the phone to tether dutifully but I have never been paged by the 24/7 support staff for something like the 5:01 self-resolving alert.

deleted 5y ago
[deleted]
bauerd 5y ago
Hire in complimentary timezones so no one has to be on call at night
- deleted 5y ago
  [deleted]
- Arainach 5y ago
  Unless you're hiring 3-4x the engineers, then this doesn't scale. I would much rather occasionally be paged at night than have to be on call all or even most of the time. Even if that "on call" was only 9-5, getting distracted and losing half a day to tracking down the latest flaky test or owner for a failing dependency really adds up and makes it significantly more difficult to do my day job. When I'm oncall, I'm oncall 24/7 for a week (30 minute SLO) and the expectation is that the amount of non-oncall tasks I'm able to get done that week is significantly impacted. I'm fortunate to work on a team that acknowledges that reality; I know many people, even at large software companies who should know better, whose management is far less reasonable.
  - odonnellryan 5y ago
    It's true but most people do not have the luxury of being able to focus on a single task each day until done. If that is a requirement for you I'm not gonna knock it but I feel it is rare to get.
- hawaiianbrah 5y ago
  We’ve been trying to staff a 3x8 follow the sun model for over 6 months and have been unable to fill the roles in EU and especially APAC
  - EdwardDiego 5y ago
    Plenty of good devs open to competitive offers down here in APAC. I joined a US company as part of a follow the sun model. Although ironically I seem to support people who are up at midnight in Europe more than in my area of the world. Happy to have a chat, see if I can offer any useful feedback :)
  - Aeolun 5y ago
    Hmm, living in APAC, it seems like companies in the US are either never hiring for remote positions overseas, or offer laughably low salaries.
    - hawaiianbrah 5y ago
      Yeah, I wish I had more power here.
  - onion2k 5y ago
    We’ve been trying to staff a 3x8 follow the sun model for over 6 months and have been unable to fill the roles in EU and especially APAC. Pay more. It literally solves all recruitment problems.
    - hawaiianbrah 5y ago
      Gee, I wish I could control that.
  - CalRobert 5y ago
    Salaries in the EU are not what they were and have improved dramatically. I've found most companies to be wildly off the mark when making offers. Perhaps you need to recalibrate.
XorNot 5y ago
Reminds me of my first job: on call didn't change much at night because the system was so twitchy (as in, SMS every 15 minutes that then autoresolved) that I just put my phone on silent and ignored it over night. Of course we also had a night shift guy who would actually call you if he got stuck, so hell if I knew why the SMS system existed. (That role lasted about a year at most - night shift for tech is stupid).
Psychlist 5y ago
My current on call setup that page would count as one hour worked of my weekly 40. Ideally I'd be paid extra as well (I get about an hours pay a week as an on call allowance). What makes it work is that I'm the senior dev on the project that I'm on call for - if it hits the fan it really is my problem. And more accurately, in the last two years I've been called out twice. So that extra hour a week of pay... it's just free money.
- lmarcos 5y ago
  > it's just free money. No. The downside of being on-call is not the number of times you get paged at 3am. The real downside is that if you are on call 24/7 and you have to react to an alert within the next 15min that means: - you cannot go out for dinner - you cannot go out for a run - you cannot go and pick up your kids (if the ride takes more than 15 min) - etc. The only thing you can do is stay at home with your laptop next to you, or wherever you go, take your laptop with you. That's being on call, and it's NOT free money.
  - grogers 5y ago
    If actually getting paged is rare them just bring your laptop to dinner, that's what I always used to do. Sure, it might be that time you are unlucky and have to work through dinner which sucks a lot. But it shouldn't happen much. Running is an issue, unless you don't mind a sweaty back to carry your laptop, or just run loops nearish to your house with your phone on you. What I usually did when picking up kids (close by, about 30 min round trip, YMMV) was to ack the page from my phone and log in when I got back home. Yes it sucks to be tethered to a laptop. But if after-hours pages are rare, and you aren't on call more than ~25% of the time, for me personally it's acceptable
  - Psychlist 5y ago
    That's not how my on call setup works. I need my phone and my laptop, and I need to be able to respond within half an hour or so. The changes I have to make are in code that then goes through CI and a load test before hitting production, so there's no "you have 10 seconds" stuff. We don't have anyone on call in the sense you mean, and I've never seen it in the wild. If they want someone right there all the time they've paid them to be in the office where they have proper access to critical systems. When I worked in a medical environment that was what we did, and those people got paid very well for doing overnight "tech support" ... they were bored enough that answering random IT questions seemed worth while for them.
btgeekboy 5y ago
If your alert is going off at 5:00 am and self-resolving at 5:01, and it happens more than once, the prudent course would be to fix the underlying problem or modify the alarming configuration. Nobody benefits from alarms that don’t require action.
kcb 5y ago
Someone once had the bright idea of forwarding several international toll-free numbers directly to a developers phone on a rotational basis. All to save the internal costs of the in-house operations team. We only ever received spam calls and wrong numbers at odd hours. It was a nightmare and total productivity killer.

deleted 5y ago

[deleted]

bauerd 5y ago

Hire in complimentary timezones so no one has to be on call at night

deleted 5y ago
[deleted]
Arainach 5y ago
Unless you're hiring 3-4x the engineers, then this doesn't scale. I would much rather occasionally be paged at night than have to be on call all or even most of the time. Even if that "on call" was only 9-5, getting distracted and losing half a day to tracking down the latest flaky test or owner for a failing dependency really adds up and makes it significantly more difficult to do my day job. When I'm oncall, I'm oncall 24/7 for a week (30 minute SLO) and the expectation is that the amount of non-oncall tasks I'm able to get done that week is significantly impacted. I'm fortunate to work on a team that acknowledges that reality; I know many people, even at large software companies who should know better, whose management is far less reasonable.
- odonnellryan 5y ago
  It's true but most people do not have the luxury of being able to focus on a single task each day until done. If that is a requirement for you I'm not gonna knock it but I feel it is rare to get.
hawaiianbrah 5y ago
We’ve been trying to staff a 3x8 follow the sun model for over 6 months and have been unable to fill the roles in EU and especially APAC
- EdwardDiego 5y ago
  Plenty of good devs open to competitive offers down here in APAC. I joined a US company as part of a follow the sun model. Although ironically I seem to support people who are up at midnight in Europe more than in my area of the world. Happy to have a chat, see if I can offer any useful feedback :)
- Aeolun 5y ago
  Hmm, living in APAC, it seems like companies in the US are either never hiring for remote positions overseas, or offer laughably low salaries.
  - hawaiianbrah 5y ago
    Yeah, I wish I had more power here.
- onion2k 5y ago
  We’ve been trying to staff a 3x8 follow the sun model for over 6 months and have been unable to fill the roles in EU and especially APAC. Pay more. It literally solves all recruitment problems.
  - hawaiianbrah 5y ago
    Gee, I wish I could control that.
- CalRobert 5y ago
  Salaries in the EU are not what they were and have improved dramatically. I've found most companies to be wildly off the mark when making offers. Perhaps you need to recalibrate.

XorNot 5y ago

Reminds me of my first job: on call didn't change much at night because the system was so twitchy (as in, SMS every 15 minutes that then autoresolved) that I just put my phone on silent and ignored it over night. Of course we also had a night shift guy who would actually call you if he got stuck, so hell if I knew why the SMS system existed. (That role lasted about a year at most - night shift for tech is stupid).

Psychlist 5y ago

My current on call setup that page would count as one hour worked of my weekly 40. Ideally I'd be paid extra as well (I get about an hours pay a week as an on call allowance). What makes it work is that I'm the senior dev on the project that I'm on call for - if it hits the fan it really is my problem. And more accurately, in the last two years I've been called out twice. So that extra hour a week of pay... it's just free money.

lmarcos 5y ago
> it's just free money. No. The downside of being on-call is not the number of times you get paged at 3am. The real downside is that if you are on call 24/7 and you have to react to an alert within the next 15min that means: - you cannot go out for dinner - you cannot go out for a run - you cannot go and pick up your kids (if the ride takes more than 15 min) - etc. The only thing you can do is stay at home with your laptop next to you, or wherever you go, take your laptop with you. That's being on call, and it's NOT free money.
- grogers 5y ago
  If actually getting paged is rare them just bring your laptop to dinner, that's what I always used to do. Sure, it might be that time you are unlucky and have to work through dinner which sucks a lot. But it shouldn't happen much. Running is an issue, unless you don't mind a sweaty back to carry your laptop, or just run loops nearish to your house with your phone on you. What I usually did when picking up kids (close by, about 30 min round trip, YMMV) was to ack the page from my phone and log in when I got back home. Yes it sucks to be tethered to a laptop. But if after-hours pages are rare, and you aren't on call more than ~25% of the time, for me personally it's acceptable
- Psychlist 5y ago
  That's not how my on call setup works. I need my phone and my laptop, and I need to be able to respond within half an hour or so. The changes I have to make are in code that then goes through CI and a load test before hitting production, so there's no "you have 10 seconds" stuff. We don't have anyone on call in the sense you mean, and I've never seen it in the wild. If they want someone right there all the time they've paid them to be in the office where they have proper access to critical systems. When I worked in a medical environment that was what we did, and those people got paid very well for doing overnight "tech support" ... they were bored enough that answering random IT questions seemed worth while for them.

btgeekboy 5y ago

If your alert is going off at 5:00 am and self-resolving at 5:01, and it happens more than once, the prudent course would be to fix the underlying problem or modify the alarming configuration. Nobody benefits from alarms that don’t require action.

kcb 5y ago

Someone once had the bright idea of forwarding several international toll-free numbers directly to a developers phone on a rotational basis. All to save the internal costs of the in-house operations team. We only ever received spam calls and wrong numbers at odd hours. It was a nightmare and total productivity killer.