8 ms·

How can I get my engineers to accept being on call?

45 points by anotherhue 3y ago

Nellyz 3y ago
[dead]
jki275 3y ago
Fuck you, hire an SRE team. Current employer is trying to have me on call for free 24/7. Not happening.
johng 3y ago
Reward them for issues fixed in off hours while on call. Make a game with rewards out of it.
- eureka-belief 3y ago
  Like badges and leaderboards and candy prizes?
  - johng 3y ago
    With money.
anotherhue 3y ago
Submitter but not OP. Just to be clear the only answer to this is a daily stipend for being on call PLUS a payout per incident (min. 1hr billing). And of course, it's opt-in. Anything else is devaluing yourself to the poor decisions of others. "I will work harder!".
- lkbm 3y ago
  This only works if the incident rates are completely divorced from the work of the engineers. Otherwise it creates a perverse incentive to increase the number of incidents.
  - anotherhue 3y ago
    HR and performance reviews should be able to handle that edge-case.
  - jrockway 3y ago
    Yeah, I think just paying volunteers to be on call is fine. Adjusted for inflation, I'd expect this number to be around $2000/week. This is incredibly cheap compared to hiring 3 teams for follow-the-sun support, and gives the early-career folks on your team the opportunity to make a little bit of extra money; somewhat rare in the world of "engineering ladders" and whatnot where it is kind of a slog before you are rolling in cash.
- chrisandchris 3y ago
  No, the only solution in this case is full pay for each minute one is on-call. A (professional) firefighter is mostly on-call on his shift and is paid the same amount whether he's waiting or at an incident. Sure, he does stuff while waiting (like maintenance). But nobody said your It stuff couldn't do maintenance too. If you really need on-call with 15min response time, design your business around it (and your finances!). The whole blog could just stop at: > The CTO mandated [...]
  - anotherhue 3y ago
    > pay for each minute one is on-call Isn't that just overtime? I'd much prefer to bring a laptop with me a few weekends a year than be expected to fill a full workday with my family around the house on a weekend.
    - chrisandchris 3y ago
      See it otherwise: It's not overtime, as a firefighter does not overtime 8h a day. It's your job and your job is to be on-call while writing code or vice-versa. I think the only viable method for such a fast response time is that it's your work. You will have off on Friday but work on Saturday (or something like that). If it's more like 1h response time sure, take your laptop with you to familys dinner.
      - johnnyanmac 3y ago
        I don't really like the firefighter analogy to be honest. A Firefighter is on call but to compensate for their physical well-being they tend to only work 3-4 days a week (even if those 3 days are 24 hours spent in the station, potentially on the weekend). Moreover, fires are inevitable and frequent, despite being sporadic. I think there is certainly merit to wondering what format a business needs based on client demands, but a mentally focused white collar job doesn't need to go as extreme as a firefighter who is literally burning out their body in the line of duty.
        3y ago
prirun 3y ago
Pay more.
not_your_vase 3y ago
Oh, I remember this. It made my eyes twitch a bit when I read that the manager wanted to put the dev on PIP, after changing the employment terms for fun, even though the dev asked about being on-call very specifically during interviews, according to the manager...
mytailorisrich 3y ago
In many jurisdictions suddenly "mandating" that an employee be on call every week/month would be a proposed change of employment contract. I.e. the employee would be fully within their right to refuse until the employer makes an acceptable offer, if at all. Also, this "manager" sounds totally clueless...
- avidiax 3y ago
  In many jurisdictions, requiring uncompensated oncall would, by itself, possibly be "constructive dismissal". PIPing without an actual performance problem is also constructive dismissal, and doing it because the employee is refusing a change is possibly retaliation. Realistically, in the U.S. at least, there's very little risk that the employer will get sued, but it's still shitty and will eventually bite the employer if they get into a habit of this.
  - knallfrosch 3y ago
    The people you have will quit and the people you hire want to be compensated just as the others. So firing really isn't an option, unless you think you're overpaying your current staff and can get the same quality cheaper.
woeirua 3y ago
Pay more or let them go. That’s it. If the job suddenly requires on call, and someone refuses to comply then show them the door and hire someone else with the explicit expectation that there will be on-call.
- eureka-belief 3y ago
  “Show them the door” seems a bit callous. If the hiring manager didn’t think to talk about on call responsibilities, that’s their own mistake. Maybe that doesn’t change that they need someone on call and maybe they still need to find a new employee, but a job doesn’t own a person 24/7 unless that signed up for it. If a manager pulled that bait and switch shit on me I’d definitely never be recommending to my smart friends to work there.
- klyrs 3y ago
  Prepare yourself for a constructive dismissal settlement.
koollman 3y ago
opt-in. Pay them for it. And have a culture of fixing stuff and spending enough before shit happens, so that on-call is rare and problems don't repeat too often
- sp332 3y ago
  My #1 answer is make sure they don't get called very often.
croo 3y ago
I am amazed by how beautiful and clear the first answer is. How nice, uncondemning it is, how it really gives the possibility of the OP is truly clueless that he's in the wrong.
- TheCowboy 3y ago
  It really is underrated how effective calm direct communication can be when it's not coupled with attempts to injure with insults.
troupe 3y ago
The whole, "How can I get people to do X when they don't want to do it for $Y?" is just a funny question. It is like asking, "How do I buy a sandwich when no one wants to sell me one for $1?" Of course if everyone except a single employee IS willing to do X for $Y then the company many choose to replace them.
- EA-3167 3y ago
  Seriously, it's crazy that everyone doesn't already know the answer is, "Pay them enough to make it worth their time."
  - deleted 3y ago
    [deleted]
  - troupe 3y ago
    They might even be able to do other things that would make it worth it. Like giving additional vacation for being on call, etc.
  - saltminer 3y ago
    Or alternatively, just don't have on-call? It's incredible how many teams have on-call for their internal service which is only used during business hours across 1-4 timezones. If your system has predictable usage and isn't global, there's very little reason to have on-call unless you have a client willing to pay $$$ or the loss of business would be catastrophic, in which case you can absolutely afford to compensate your devs for it.
    - EA-3167 3y ago
      These are all very good points and well said, I agree!
- johnnyanmac 3y ago
  >"How do I buy a sandwich when no one wants to sell me one for $1? Easy, you learn how to make your own sandwich. buy bulk and you amortize the cost to less than $1/sandwich Of course, technical expertise isn't as easy as putting condiments between bread. Guess you gotta pay your labor if you're not willing to do it yourself.
hosteur 3y ago
Easy. Pay them a decent salary for every hour they are on call. Simple as that.
- klyrs 3y ago
  You can't pay me a decent* salary for on-call. This is true of many people. One developer I worked with took medication at night which rendered him completely useless, and skipping the medication resulted in several very bad days. For me, it's simply a work/life balance thing, and I can find work that doesn't need it. The "easy" answer is to put it in the job ad, and hire people who agree to it. Because forcing that requirement on an existing employee just isn't going to happen. * by all means, pay me an indecent wage, enough that I can retire after a year!
  - kelnos 3y ago
    Yup, agreed. At the very least, make it opt-in, and keep raising the on-call pay until you have enough existing employees who will decide the trade off is worth it. At some point it maybe ends up being more economical to follow your suggestion to put out a job ad specifically for on-call people.
  - mariodiana 3y ago
    Years ago when I worked second-level support, I was the on-call guy for Thanksgiving. Our customers were specialty retailers, so I thought nothing of it. I figured they'd be closed. Turns out, we had an "Urban Clothing Store" as a customer, and on that day — what?! — someone couldn't ring up an 8-ball jacket? I was with my family, enjoying a cocktail when I got the call. I have seldom been more pissed off. In the end, from what I could remember, after wasting almost an hour, we tabled the whole incident. I'm with you. On-call is never worth it. If a company needs 24/7 coverage, they should hire round the clock.
- jki275 3y ago
  As another reply said, you can't afford me. I did on call 24/7/365 for 23 years. I'll never do it again. My phone goes silent in the evening and doesn't come online again until I get up to come to work. That's never going to change. I'm not 20 anymore, I don't function on no sleep.
captainbland 3y ago
Recruit people into being on call by positively incentivising it so that it's their own choice, don't force it on them.
Apreche 3y ago
One other solution nobody has suggested. Hire lots and lots of staff, and work in shifts. Have enough people on each shift that even when people use PTO or fall ill, someone is always working. Nobody ever has to be on call. Instead everyone works normal hours, but someone is always available. You’ll still have to pay extra if you want people working on holidays, though.
- kelnos 3y ago
  A company I worked at started (inconsistently) adopting a "follow the sun" on-call approach once they were large enough to have enough engineering staff on more than one continent. There was certainly still some time that needed coverage that was outside anyone's work hours, but at least people weren't having entire days (including their sleep) ruined by on-call.
- yjftsjthsd-h 3y ago
  What about weekends?
- johnnyanmac 3y ago
  It's probably the worst solution for an employer. if they don't want to give 50% salary to keep current staff on call, they won't want to spend 2x to have staff on hand in staggering hours.
kelnos 3y ago
I've done a fair bit of on-call during my career, but I wouldn't do it anymore (at least not for any dollar amount a company would likely pay). I value my time outside work -- and my sleep -- too much. The standard in SV seems to be that on-call is required and uncompensated (some companies will dishonestly try to suggest that your salary is higher than it would be otherwise). This is gross, and we shouldn't stand for it. There are exceptions of course, but even companies I've heard of that compensate for on-call seem to fall far short of what should be expected and required. On-call should be compensated at a base per-hour rate (outside of 9am-5pm), with an elevated rate for time spent handling an incident if the pager does go off, with a minimum payout of 1 hour of time. It should be opt-in. If not enough people want to do it to get you the coverage you want, raise the rates until your employees feel that the time is worth the money. It's pretty messed up that OP here is throwing shade at an employee who is balking at this, wanting to put them on a PIP, plus a thinly-veiled suggestion that they're lying about having asked about on-call requirements during their interview process. Bottom line is that they're drastically changing the terms of employment without offering extra compensation, and while in many jurisdictions that's legal, it's nonetheless unethical and disgusting.
- knallfrosch 3y ago
  On-call X% (0<X<100), and if you're called: The full 100% and then all the applicable bonuses for night shift and weekend etc.
  - morelisp 3y ago
    It needs to be be 150% minimum, otherwise they're just paying the same for >40h.
- cj 3y ago
  I agree if on call is being triggered regularly. I strongly disagree if events that would trigger on call happen rarely (e.g. once per year). Many companies have stable infrastructure but just need the extremely occasional coverage for things that rarely, if ever, happen. I’ve seen people overreact when on call has been introduced, probably from negative experiences at prior jobs. As long as the on call events happen once in a blue moon, I don’t see why anyone would have a major problem with it.
jwie 3y ago
Put yourself into the oncall rotation for starters. For a US exempt employee it would be fairly straightforward to push oncall expectations into a role. But in other countries with robust labor protections it might be more challenging. Ultimately the solution is money. But doing it yourself goes a long way to convincing your team it’s necessary. I personally do this as a manager and I find it helps a lot. If you get paged a few times at 3am it really helps align your priorities with your team.
hevans66 3y ago
If you are begging your engineers to be on call, you have built the wrong engineering culture. Your core engineering team should feel ownership of the app, it should be their baby. What happens in this case is EVERYONE shows up when there is a site issue, and your best engineers burnt out. Instituting an on-call schedule is what a good engineering manager does to force engineers to NOT be on-call 24/7, and instead institute an equitable distribution of responsibility. I think all small eng teams should have on-call rotations, and it's the reason behind why I built https://heiioncall.com/ https://heiioncall.com/.
- johnnyanmac 3y ago
  > it should be their baby. Made that mistake. Glad it only took one layoff to show that that's never true in Big Business. Never again. I am a worker providing my talent to work on a product I don't own. I will take responsibilities but never "ownership". "ownership" implies that I have a stake in the product, and the product goes with me no matter what. Which I have yet to experience (maybe one day).
temp10723 3y ago
Pay them for the value they provide.
tamimio 3y ago
A lot of managers or CxO think they own the employees and when suddenly they change mindset, they expect it for free, there’s nothing more pity than a C level who’s more of business guy than technical and understands everything comes with a price, and yet wanted it free of charge, pay him for the extra services you require, each contract is a unique case, just because other engineers do or agreed to do so, doesn’t mean he should if the contract says otherwise.
causality0 3y ago
Would be an easy problem to solve with a carrot instead of a stick. Just create a bonus for being the first person to respond to an out of hours group text, another bonus for being the person to respond to the most out of hours problems, and a year-end bonus for the whole team if no out of hours problem goes more than thirty minutes without a response from one of them. They'll organize the on call system themselves with no whining and you'll only spend money on actual problems since you're not paying anyone hourly to be on call. Everybody wins.
ftth_finland 3y ago
I love how the manager’s immediate reaction is to go for constructive dismissal.
- johnnyanmac 3y ago
  Good. If this goes through I hope the employee walks. They clearly want workers who can do those hours and they need to renegotiate that. be it with existing employees or new ones (who are likely going to cost more, plus time to train. Buf if a company wants to cut off its nose to spite the face, go for it).
deleted 3y ago
[deleted]
munchler 3y ago
Another unspoken red flag is that this app is having so many bugs in production that it requires constant attention from its developers. How are all these bugs getting past QA? Is there a legit development process in place? Who is signing off on such buggy software?
- Silhouette 3y ago
  This is what always gets me about these on-call discussions. It's like there is some kind of Stockholm syndrome thing going on where employees hired to write software are forced into working under completely different conditions to what software development should require and no-one ever stops to ask how badly you have screwed up the whole company culture for this issue to be relevant at all. I recommend investing in real QA and dumping the move-fast-and-break-stuff attitude and probably 90% of the madness that gets bundled under the Agile and DevOps banners as an excuse for not planning or designing anything properly any more. Your Operations team who are running a 24/7 schedule (and hiring for it) anyway should be able to deal with genuine emergencies without requiring non-specialist developers to be on call routinely. The world has worked that way forever and still does outside the DevOps bubble - including numerous industries that operate 24/7 and rely on software to do so.
  - morelisp 3y ago
    Assuming everyone can just "have an operations team running 24/7" out the gate is also a bubble. Anyway, lots of jobs with variable demand (doctors, vets, supers, utility workers, etc.) work on a fixed / % of salary daily rate + time-and-a-half when activated. What I agree is bullshit is TFA though, if it's not in your contract, don't do it.
    - Silhouette 3y ago
      But if you don't have 24/7 Operations coverage then you probably aren't really able to run fully reliably all the time anyway. And that is perfectly OK and normal for smaller and newer businesses. They just need to stop kidding themselves that they can operate at that level by exploiting their dev team.
lb1lf 3y ago
Where I work, the on-call deal is that it is opt-in, you're on one week and then have six weeks or so during which you will not be put back on call no matter what. Base compensation which you receive regardless of whether there are any incidents during the week or not, plus hourly compensation (50-100% overtime + another 40% for the inconvenience) Also, those who have opted in are quite flexible when it comes to covering for each other for a few hours - so you can still go to the movies or host a birthday party for a kid without fear of the phone suddenly buzzing. Result? People are doing their best to get in on the roster, as the compensation feels worth it. Try that.
- sayrer 3y ago
  This is a pretty fair comment, but also not scratching the surface in some ways. Firstly, any decent boss will give you days off if some gnarly on-call stuff happens. Anywhere from "you can come in at 2pm and leave at 5pm" to "take Monday and Tuesday off, no one will track it". This is because they know it is a retention issue. Secondly, there is a cultural divide sometimes. Generally, iOS programmers aren't on call, because they can't ship anything in less than a day or two. So, you get this phenomenon where they say "I'm an expert in Apple user interfaces" (a fine thing to know about!), but everyone resents them, because they are not on call.
deleted 3y ago
[deleted]
andrewaylett 3y ago
Incentives are important -- unlike other commentators, I agree with my management that paying extra for time spent actually working when on-call is not a great idea: it sets up a perverse incentive. We have a base hourly rate for on-call (with a half-hour response time) and time off in lieu for actual call-outs. Teams differ, but my team has separate rotas for in-hours and out-of-hours. This means that if you're up all night, you get time to sleep without needing someone to take on-call from you during the day. An average week of out-of-hours involves precicely zero pages. This is also important: when we get paged for something, we take steps to avoid being paged for the same thing again. Not just because we value our sleep, but because management values our services being up and running so we're encouraged to fix stuff that's broken. We did have a team saying that their services were noisier than other teams', and asking if they could be paid more as a consequence. The answer was to fix their service reliability, not to incentivise people to wait until out-of-hours to declare an incident so they trigger the bonus payment. So long as I've got mobile coverage, I'm free to get on with my life. I carry my laptop if I'm more than 30 minutes from home, but I don't expect to need to use it.
- anotherhue 3y ago
  > So long as I've got mobile coverage, I'm free to get on with my life. I don't doubt this, but surely you can agree many people lead lives which they would not be free to live in this arrangement? Putting children to bed. Driving somewhere. Cooking. Going to a bar...
  - andrewaylett 3y ago
    I do all of those things. And yes, I know it's frustrating not to be able to enjoy a drink but that's one of the things we're being paid to compensate for. The others are all possible while on-call, especially when I'm justified in not expecting to be called out. I've also dropped off the on-call rota, and encouraged others to do the same, to care for very young children or family who need care that would be compromised by responding to a page. Or vice-versa, of course: if I've already been up half the night looking after a newborn then I'm not going to be much help. My contract doesn't oblige me to be on-call, I wouldn't do it if the compensation didn't make it worthwhile.
throwaway765123 3y ago
Throwaway so I can talk honestly. I used to work on call in non-software, as a blood analyzer Field Service Engineer for medtech companies, like Siemens, Roche, Beckman-Coulter, Bio-Rad, etc. I spent 5 years doing this. In that industry and job it's standard to work on call rotations that last for 7 days through the weekend and repeat every 3-5 weeks depending on team size. Because the job is hands-on and at customer sites, the minimum response time is usually a bit more than 15 minutes. If anyone is considering accepting an on-call rotation, don't. It takes over your life. At first you may think it's fun to swoop in and be the hero fixit person, or maybe you at least tolerate it because of high base pay or something. Eventually you grow to hate the sound that accompanies the callout text or email. By the end, it triggered a visceral physical negative reaction I later learned was a minor anxiety attack. The thing managers seeking to add on-call don't understand (those who haven't done it themselves), is that when you're on call you can't do anything. Not only can you not drink (minor annoyance), you can't even leave the house for normal stuff like a movie (medium), or attend events like a kids graduation or recital (major). And before you say "just swap rotations around on your team," I worked with about 10-15 guys and you'd be surprised how hard it is to swap. Everyone makes plans for things all the time without realizing it would interfere with on-call, like going to the gym. At the start of the year when the on-call rotations would be released to us, there was a mad rush to submit vacation requests to get it locked in before illness or other things required swapping on-call around, much less having to take field requests from colleagues that you may (will) have to rely on when you in turn need a swap, so you can't just ignore every request. This manager and CTO are hilariously unprepared and uninformed, bordering on pure dreaming. A 15 minute max response time is criminally unrealistic, you can't even go to the grocery store. And once a week seems low on paper until you realize that depending on team size, you can easily wind up being forced to give up a quarter of your weekends for the year (as I had to). I haven't worked on-call for software, so maybe it's different, but I doubt it's really too different. After 5 years of all that lost time, the amount I'd have to be paid to go back on-call is so astronomically high no company would ever pay it. Actually, now that I have a kid, there's probably no amount I could be paid if it means missing time with them.
- Silhouette 3y ago
  It seems like OP and their CTO would like to have reliable 24/7 operations but do not have the skill to do it by designing systems that won't fail catastrophically out of hours or the budget to do it by hiring dedicated staff to be available at all times to fix problems. So instead they are trying to abuse their existing staff. The correct response to this IMHO is for the existing staff to collectively tell management where to go. In life you can't always get what you want if you don't have the money or make the effort it requires.
pyjarrett 3y ago
What's the rationale for setting up the on-call schedule? Is the system that unstable? > on call at least once a week, including the work day and evening, once a month, including the weekend of that week. > I should also note there is a required 15 min response time to pages. The person MUST be able to be online and fixing the problem within 15 mins of the page. I've worked a position where I had extra duties on the weekend, so I effectively worked 12+ days in a row many times. I've also been on an on-call schedule like this with a short response time. Both of these severely impact your personal life. You have to realize that the two most valuable things you can give engineers is their own time back and money. This is how I would probably try to handle it: To start, it would be opt-in and not mandatory, as this type of responsibility is difficult for those with small children or other family members, such as elderly, needing care. The 15 minute response time also means you should be being paid an hourly rate (maybe even 1.5x or 2x) while being on-call, whether there's an incident or not. This is because it severely limits what you can do -- you're basically chained to your laptop and an internet connection. Anyone responding to an on-call event after hours should also be given a half or full day off the next work day. Weekend on-call should at a minimum be given a day off the following week, preferable two.
- kelnos 3y ago
  > Is the system that unstable? I'm generally anti-on-call, but c'mon, this is an unreasonable standard. No system is 100% perfect, never needing manual intervention to get things going again when something happens. Any business that has a service that is supposed to be running 24/7 will likely need some form of on-call.
  - Silhouette 3y ago
    If you have a business that needs to operate 24/7 then you need your Operations team to hire accordingly and then on-call should normally be unnecessary. Operations should be able to fix an emergency that is unsafe, insecure or illegal. Anything else can wait until the next working day even if management might prefer it not to because that costs money. If a business can't or won't make those arrangements (and pay the bill for them) then maybe its management team need to recognise that fully reliable 24/7 operation is not something they can guarantee and they should plan accordingly.
CapitalistCartr 3y ago
Money won't answer this problem. Hire someone(s) to cover this is cheaper.
glyphy177 3y ago
[dead]
theshrike79 3y ago
Over here the standard usually is 50% hourly pay for being on-call. On-call requires you to be 15-30 minutes away from whatever you can use to fix issues, usually work laptop + internet connection. Oh and you need to be sober (...enough to answer the phone while sounding sober =) As soon as something happens, you jump to regular overtime pay so if it's on a weekend+Sunday, you're making around 200-300% pay. If it's just after others logged off, you're getting normal pay, maybe 50% extra depending on the company. I know a lot of people who have paid off loans just by hogging all the on-call hours they can. It's not especially hard if the playbooks are well made and nothing goes horribly wrong.
karmakaze 3y ago
The times I've been on-call as a sr backend dev, my compensation was one of - $700 for any week that I was on call (whether or not there were any incidents, and more often than not there were none) - paid my phone bill, roughly $100/mo (including data + amortized phone cost) - free to take off the next day any time I'm paged off-hours. There have been other times I was unofficially on-call (at small startups/companies) with no compensation, but I don't think there was enough scale/frequency for it to be a sticking point.
joshka 3y ago
This feels like a possible troll. Also by the same person: I work from home as a software engineer and my job is happy with my performance. But I'm putting in little effort. Am I a bad person or employee? [1] --- I used to be a Senior Software Engineer at another company but recently I took a lower level 2 position at another company remotely. I am very good at what I do. I can usually take a well groomed agile story in any tech stack and complete it way ahead of schedule. The company always praises me for this. But the truth is they cannot keep me busy. I know I can help out else where but my output is already so very high, especially compared to everyone else. So instead I just sit around all day playing video games while I wait for my next work assignment. This can sometimes be days or even a week. I don't go looking for it. I have so much free time on hands that I am seriously considering getting a 2nd full time software engineering job remotely and doing this all over again. No one is the wiser. Does this make a bad person or employee? [1]: https://philosophy.stackexchange.com/questions/91282/i-work-from-home-as-a-software-engineer-and-my-job-is-happy-with-my-performance https://philosophy.stackexchange.com/questions/91282/i-work-...
- MollyRealized 3y ago
  Well-noted.
simne 3y ago
From my own experience, I understand, being on call could be important for engineers, but CEO MUST understand, not all people could accept this. So, in real life, need agile approach. Some people just accept to be on call all time, you could consider them as your core team with appropriate benefits and value them, and respect them, really. Second group of people could accept receive calls during their shift. This is also not bad. Value this. And exists people, who are good workers, make great value, but could be on call very limited time or nearly no time. You should decide, with them individually, how you will work. It is really possible, some very valuable people will not be on call at all.
dogmatism 3y ago
*laughs in Medicine
EPWN3D 3y ago
> Should we just PIP the engineer No, fire them without cause like a grown up and take the hit to your unemployment insurance. Don't put them through some three-ring circus act masquerading as employee development. It's undignified and disrespectful. You and the employee have a disagreement over the job description -- this not performance related. Don't try and spin it as that. It amazes me that these B-school pricks managed to enshrine at-will employment across the country, and they still won't just fucking fire people like adults. No, instead we got one of the worst MBA-isms visited on corporate America, the PIP. Fucking cowards, the lot of them.
Nellyz 3y ago
[dead]
Nellyz 3y ago
[dead]
gina00 3y ago
[dead]

45 points by anotherhue 3y ago

Nellyz 3y ago

[dead]

jki275 3y ago

Fuck you, hire an SRE team. Current employer is trying to have me on call for free 24/7. Not happening.

johng 3y ago

Reward them for issues fixed in off hours while on call. Make a game with rewards out of it.

eureka-belief 3y ago
Like badges and leaderboards and candy prizes?
- johng 3y ago
  With money.

anotherhue 3y ago

Submitter but not OP. Just to be clear the only answer to this is a daily stipend for being on call PLUS a payout per incident (min. 1hr billing). And of course, it's opt-in. Anything else is devaluing yourself to the poor decisions of others. "I will work harder!".

lkbm 3y ago
This only works if the incident rates are completely divorced from the work of the engineers. Otherwise it creates a perverse incentive to increase the number of incidents.
- anotherhue 3y ago
  HR and performance reviews should be able to handle that edge-case.
- jrockway 3y ago
  Yeah, I think just paying volunteers to be on call is fine. Adjusted for inflation, I'd expect this number to be around $2000/week. This is incredibly cheap compared to hiring 3 teams for follow-the-sun support, and gives the early-career folks on your team the opportunity to make a little bit of extra money; somewhat rare in the world of "engineering ladders" and whatnot where it is kind of a slog before you are rolling in cash.
chrisandchris 3y ago
No, the only solution in this case is full pay for each minute one is on-call. A (professional) firefighter is mostly on-call on his shift and is paid the same amount whether he's waiting or at an incident. Sure, he does stuff while waiting (like maintenance). But nobody said your It stuff couldn't do maintenance too. If you really need on-call with 15min response time, design your business around it (and your finances!). The whole blog could just stop at: > The CTO mandated [...]
- anotherhue 3y ago
  > pay for each minute one is on-call Isn't that just overtime? I'd much prefer to bring a laptop with me a few weekends a year than be expected to fill a full workday with my family around the house on a weekend.
  - chrisandchris 3y ago
    See it otherwise: It's not overtime, as a firefighter does not overtime 8h a day. It's your job and your job is to be on-call while writing code or vice-versa. I think the only viable method for such a fast response time is that it's your work. You will have off on Friday but work on Saturday (or something like that). If it's more like 1h response time sure, take your laptop with you to familys dinner.
    - johnnyanmac 3y ago
      I don't really like the firefighter analogy to be honest. A Firefighter is on call but to compensate for their physical well-being they tend to only work 3-4 days a week (even if those 3 days are 24 hours spent in the station, potentially on the weekend). Moreover, fires are inevitable and frequent, despite being sporadic. I think there is certainly merit to wondering what format a business needs based on client demands, but a mentally focused white collar job doesn't need to go as extreme as a firefighter who is literally burning out their body in the line of duty.
      - 3y ago

prirun 3y ago

Pay more.

not_your_vase 3y ago

Oh, I remember this. It made my eyes twitch a bit when I read that the manager wanted to put the dev on PIP, after changing the employment terms for fun, even though the dev asked about being on-call very specifically during interviews, according to the manager...

mytailorisrich 3y ago

In many jurisdictions suddenly "mandating" that an employee be on call every week/month would be a proposed change of employment contract. I.e. the employee would be fully within their right to refuse until the employer makes an acceptable offer, if at all. Also, this "manager" sounds totally clueless...

avidiax 3y ago
In many jurisdictions, requiring uncompensated oncall would, by itself, possibly be "constructive dismissal". PIPing without an actual performance problem is also constructive dismissal, and doing it because the employee is refusing a change is possibly retaliation. Realistically, in the U.S. at least, there's very little risk that the employer will get sued, but it's still shitty and will eventually bite the employer if they get into a habit of this.
- knallfrosch 3y ago
  The people you have will quit and the people you hire want to be compensated just as the others. So firing really isn't an option, unless you think you're overpaying your current staff and can get the same quality cheaper.

woeirua 3y ago

Pay more or let them go. That’s it. If the job suddenly requires on call, and someone refuses to comply then show them the door and hire someone else with the explicit expectation that there will be on-call.

eureka-belief 3y ago
“Show them the door” seems a bit callous. If the hiring manager didn’t think to talk about on call responsibilities, that’s their own mistake. Maybe that doesn’t change that they need someone on call and maybe they still need to find a new employee, but a job doesn’t own a person 24/7 unless that signed up for it. If a manager pulled that bait and switch shit on me I’d definitely never be recommending to my smart friends to work there.
klyrs 3y ago
Prepare yourself for a constructive dismissal settlement.

koollman 3y ago

opt-in. Pay them for it. And have a culture of fixing stuff and spending enough before shit happens, so that on-call is rare and problems don't repeat too often

sp332 3y ago
My #1 answer is make sure they don't get called very often.

croo 3y ago

I am amazed by how beautiful and clear the first answer is. How nice, uncondemning it is, how it really gives the possibility of the OP is truly clueless that he's in the wrong.

TheCowboy 3y ago
It really is underrated how effective calm direct communication can be when it's not coupled with attempts to injure with insults.

troupe 3y ago

The whole, "How can I get people to do X when they don't want to do it for $Y?" is just a funny question. It is like asking, "How do I buy a sandwich when no one wants to sell me one for $1?" Of course if everyone except a single employee IS willing to do X for $Y then the company many choose to replace them.

EA-3167 3y ago
Seriously, it's crazy that everyone doesn't already know the answer is, "Pay them enough to make it worth their time."
- deleted 3y ago
  [deleted]
- troupe 3y ago
  They might even be able to do other things that would make it worth it. Like giving additional vacation for being on call, etc.
- saltminer 3y ago
  Or alternatively, just don't have on-call? It's incredible how many teams have on-call for their internal service which is only used during business hours across 1-4 timezones. If your system has predictable usage and isn't global, there's very little reason to have on-call unless you have a client willing to pay $$$ or the loss of business would be catastrophic, in which case you can absolutely afford to compensate your devs for it.
  - EA-3167 3y ago
    These are all very good points and well said, I agree!
johnnyanmac 3y ago
>"How do I buy a sandwich when no one wants to sell me one for $1? Easy, you learn how to make your own sandwich. buy bulk and you amortize the cost to less than $1/sandwich Of course, technical expertise isn't as easy as putting condiments between bread. Guess you gotta pay your labor if you're not willing to do it yourself.

hosteur 3y ago

Easy. Pay them a decent salary for every hour they are on call. Simple as that.

klyrs 3y ago
You can't pay me a decent* salary for on-call. This is true of many people. One developer I worked with took medication at night which rendered him completely useless, and skipping the medication resulted in several very bad days. For me, it's simply a work/life balance thing, and I can find work that doesn't need it. The "easy" answer is to put it in the job ad, and hire people who agree to it. Because forcing that requirement on an existing employee just isn't going to happen. * by all means, pay me an indecent wage, enough that I can retire after a year!
- kelnos 3y ago
  Yup, agreed. At the very least, make it opt-in, and keep raising the on-call pay until you have enough existing employees who will decide the trade off is worth it. At some point it maybe ends up being more economical to follow your suggestion to put out a job ad specifically for on-call people.
- mariodiana 3y ago
  Years ago when I worked second-level support, I was the on-call guy for Thanksgiving. Our customers were specialty retailers, so I thought nothing of it. I figured they'd be closed. Turns out, we had an "Urban Clothing Store" as a customer, and on that day — what?! — someone couldn't ring up an 8-ball jacket? I was with my family, enjoying a cocktail when I got the call. I have seldom been more pissed off. In the end, from what I could remember, after wasting almost an hour, we tabled the whole incident. I'm with you. On-call is never worth it. If a company needs 24/7 coverage, they should hire round the clock.
jki275 3y ago
As another reply said, you can't afford me. I did on call 24/7/365 for 23 years. I'll never do it again. My phone goes silent in the evening and doesn't come online again until I get up to come to work. That's never going to change. I'm not 20 anymore, I don't function on no sleep.

captainbland 3y ago

Recruit people into being on call by positively incentivising it so that it's their own choice, don't force it on them.

Apreche 3y ago

One other solution nobody has suggested. Hire lots and lots of staff, and work in shifts. Have enough people on each shift that even when people use PTO or fall ill, someone is always working. Nobody ever has to be on call. Instead everyone works normal hours, but someone is always available. You’ll still have to pay extra if you want people working on holidays, though.

kelnos 3y ago
A company I worked at started (inconsistently) adopting a "follow the sun" on-call approach once they were large enough to have enough engineering staff on more than one continent. There was certainly still some time that needed coverage that was outside anyone's work hours, but at least people weren't having entire days (including their sleep) ruined by on-call.
yjftsjthsd-h 3y ago
What about weekends?
johnnyanmac 3y ago
It's probably the worst solution for an employer. if they don't want to give 50% salary to keep current staff on call, they won't want to spend 2x to have staff on hand in staggering hours.

kelnos 3y ago

I've done a fair bit of on-call during my career, but I wouldn't do it anymore (at least not for any dollar amount a company would likely pay). I value my time outside work -- and my sleep -- too much. The standard in SV seems to be that on-call is required and uncompensated (some companies will dishonestly try to suggest that your salary is higher than it would be otherwise). This is gross, and we shouldn't stand for it. There are exceptions of course, but even companies I've heard of that compensate for on-call seem to fall far short of what should be expected and required. On-call should be compensated at a base per-hour rate (outside of 9am-5pm), with an elevated rate for time spent handling an incident if the pager does go off, with a minimum payout of 1 hour of time. It should be opt-in. If not enough people want to do it to get you the coverage you want, raise the rates until your employees feel that the time is worth the money. It's pretty messed up that OP here is throwing shade at an employee who is balking at this, wanting to put them on a PIP, plus a thinly-veiled suggestion that they're lying about having asked about on-call requirements during their interview process. Bottom line is that they're drastically changing the terms of employment without offering extra compensation, and while in many jurisdictions that's legal, it's nonetheless unethical and disgusting.

knallfrosch 3y ago
On-call X% (0<X<100), and if you're called: The full 100% and then all the applicable bonuses for night shift and weekend etc.
- morelisp 3y ago
  It needs to be be 150% minimum, otherwise they're just paying the same for >40h.
cj 3y ago
I agree if on call is being triggered regularly. I strongly disagree if events that would trigger on call happen rarely (e.g. once per year). Many companies have stable infrastructure but just need the extremely occasional coverage for things that rarely, if ever, happen. I’ve seen people overreact when on call has been introduced, probably from negative experiences at prior jobs. As long as the on call events happen once in a blue moon, I don’t see why anyone would have a major problem with it.

jwie 3y ago

Put yourself into the oncall rotation for starters. For a US exempt employee it would be fairly straightforward to push oncall expectations into a role. But in other countries with robust labor protections it might be more challenging. Ultimately the solution is money. But doing it yourself goes a long way to convincing your team it’s necessary. I personally do this as a manager and I find it helps a lot. If you get paged a few times at 3am it really helps align your priorities with your team.

hevans66 3y ago

If you are begging your engineers to be on call, you have built the wrong engineering culture. Your core engineering team should feel ownership of the app, it should be their baby. What happens in this case is EVERYONE shows up when there is a site issue, and your best engineers burnt out. Instituting an on-call schedule is what a good engineering manager does to force engineers to NOT be on-call 24/7, and instead institute an equitable distribution of responsibility. I think all small eng teams should have on-call rotations, and it's the reason behind why I built https://heiioncall.com/ https://heiioncall.com/.

johnnyanmac 3y ago
> it should be their baby. Made that mistake. Glad it only took one layoff to show that that's never true in Big Business. Never again. I am a worker providing my talent to work on a product I don't own. I will take responsibilities but never "ownership". "ownership" implies that I have a stake in the product, and the product goes with me no matter what. Which I have yet to experience (maybe one day).

temp10723 3y ago

Pay them for the value they provide.

tamimio 3y ago

A lot of managers or CxO think they own the employees and when suddenly they change mindset, they expect it for free, there’s nothing more pity than a C level who’s more of business guy than technical and understands everything comes with a price, and yet wanted it free of charge, pay him for the extra services you require, each contract is a unique case, just because other engineers do or agreed to do so, doesn’t mean he should if the contract says otherwise.

causality0 3y ago

Would be an easy problem to solve with a carrot instead of a stick. Just create a bonus for being the first person to respond to an out of hours group text, another bonus for being the person to respond to the most out of hours problems, and a year-end bonus for the whole team if no out of hours problem goes more than thirty minutes without a response from one of them. They'll organize the on call system themselves with no whining and you'll only spend money on actual problems since you're not paying anyone hourly to be on call. Everybody wins.

ftth_finland 3y ago

I love how the manager’s immediate reaction is to go for constructive dismissal.

johnnyanmac 3y ago
Good. If this goes through I hope the employee walks. They clearly want workers who can do those hours and they need to renegotiate that. be it with existing employees or new ones (who are likely going to cost more, plus time to train. Buf if a company wants to cut off its nose to spite the face, go for it).

deleted 3y ago

[deleted]

munchler 3y ago

Another unspoken red flag is that this app is having so many bugs in production that it requires constant attention from its developers. How are all these bugs getting past QA? Is there a legit development process in place? Who is signing off on such buggy software?

Silhouette 3y ago
This is what always gets me about these on-call discussions. It's like there is some kind of Stockholm syndrome thing going on where employees hired to write software are forced into working under completely different conditions to what software development should require and no-one ever stops to ask how badly you have screwed up the whole company culture for this issue to be relevant at all. I recommend investing in real QA and dumping the move-fast-and-break-stuff attitude and probably 90% of the madness that gets bundled under the Agile and DevOps banners as an excuse for not planning or designing anything properly any more. Your Operations team who are running a 24/7 schedule (and hiring for it) anyway should be able to deal with genuine emergencies without requiring non-specialist developers to be on call routinely. The world has worked that way forever and still does outside the DevOps bubble - including numerous industries that operate 24/7 and rely on software to do so.
- morelisp 3y ago
  Assuming everyone can just "have an operations team running 24/7" out the gate is also a bubble. Anyway, lots of jobs with variable demand (doctors, vets, supers, utility workers, etc.) work on a fixed / % of salary daily rate + time-and-a-half when activated. What I agree is bullshit is TFA though, if it's not in your contract, don't do it.
  - Silhouette 3y ago
    But if you don't have 24/7 Operations coverage then you probably aren't really able to run fully reliably all the time anyway. And that is perfectly OK and normal for smaller and newer businesses. They just need to stop kidding themselves that they can operate at that level by exploiting their dev team.

lb1lf 3y ago

Where I work, the on-call deal is that it is opt-in, you're on one week and then have six weeks or so during which you will not be put back on call no matter what. Base compensation which you receive regardless of whether there are any incidents during the week or not, plus hourly compensation (50-100% overtime + another 40% for the inconvenience) Also, those who have opted in are quite flexible when it comes to covering for each other for a few hours - so you can still go to the movies or host a birthday party for a kid without fear of the phone suddenly buzzing. Result? People are doing their best to get in on the roster, as the compensation feels worth it. Try that.

sayrer 3y ago
This is a pretty fair comment, but also not scratching the surface in some ways. Firstly, any decent boss will give you days off if some gnarly on-call stuff happens. Anywhere from "you can come in at 2pm and leave at 5pm" to "take Monday and Tuesday off, no one will track it". This is because they know it is a retention issue. Secondly, there is a cultural divide sometimes. Generally, iOS programmers aren't on call, because they can't ship anything in less than a day or two. So, you get this phenomenon where they say "I'm an expert in Apple user interfaces" (a fine thing to know about!), but everyone resents them, because they are not on call.

deleted 3y ago

[deleted]

andrewaylett 3y ago

Incentives are important -- unlike other commentators, I agree with my management that paying extra for time spent actually working when on-call is not a great idea: it sets up a perverse incentive. We have a base hourly rate for on-call (with a half-hour response time) and time off in lieu for actual call-outs. Teams differ, but my team has separate rotas for in-hours and out-of-hours. This means that if you're up all night, you get time to sleep without needing someone to take on-call from you during the day. An average week of out-of-hours involves precicely zero pages. This is also important: when we get paged for something, we take steps to avoid being paged for the same thing again. Not just because we value our sleep, but because management values our services being up and running so we're encouraged to fix stuff that's broken. We did have a team saying that their services were noisier than other teams', and asking if they could be paid more as a consequence. The answer was to fix their service reliability, not to incentivise people to wait until out-of-hours to declare an incident so they trigger the bonus payment. So long as I've got mobile coverage, I'm free to get on with my life. I carry my laptop if I'm more than 30 minutes from home, but I don't expect to need to use it.

anotherhue 3y ago
> So long as I've got mobile coverage, I'm free to get on with my life. I don't doubt this, but surely you can agree many people lead lives which they would not be free to live in this arrangement? Putting children to bed. Driving somewhere. Cooking. Going to a bar...
- andrewaylett 3y ago
  I do all of those things. And yes, I know it's frustrating not to be able to enjoy a drink but that's one of the things we're being paid to compensate for. The others are all possible while on-call, especially when I'm justified in not expecting to be called out. I've also dropped off the on-call rota, and encouraged others to do the same, to care for very young children or family who need care that would be compromised by responding to a page. Or vice-versa, of course: if I've already been up half the night looking after a newborn then I'm not going to be much help. My contract doesn't oblige me to be on-call, I wouldn't do it if the compensation didn't make it worthwhile.

throwaway765123 3y ago

Throwaway so I can talk honestly. I used to work on call in non-software, as a blood analyzer Field Service Engineer for medtech companies, like Siemens, Roche, Beckman-Coulter, Bio-Rad, etc. I spent 5 years doing this. In that industry and job it's standard to work on call rotations that last for 7 days through the weekend and repeat every 3-5 weeks depending on team size. Because the job is hands-on and at customer sites, the minimum response time is usually a bit more than 15 minutes. If anyone is considering accepting an on-call rotation, don't. It takes over your life. At first you may think it's fun to swoop in and be the hero fixit person, or maybe you at least tolerate it because of high base pay or something. Eventually you grow to hate the sound that accompanies the callout text or email. By the end, it triggered a visceral physical negative reaction I later learned was a minor anxiety attack. The thing managers seeking to add on-call don't understand (those who haven't done it themselves), is that when you're on call you can't do anything. Not only can you not drink (minor annoyance), you can't even leave the house for normal stuff like a movie (medium), or attend events like a kids graduation or recital (major). And before you say "just swap rotations around on your team," I worked with about 10-15 guys and you'd be surprised how hard it is to swap. Everyone makes plans for things all the time without realizing it would interfere with on-call, like going to the gym. At the start of the year when the on-call rotations would be released to us, there was a mad rush to submit vacation requests to get it locked in before illness or other things required swapping on-call around, much less having to take field requests from colleagues that you may (will) have to rely on when you in turn need a swap, so you can't just ignore every request. This manager and CTO are hilariously unprepared and uninformed, bordering on pure dreaming. A 15 minute max response time is criminally unrealistic, you can't even go to the grocery store. And once a week seems low on paper until you realize that depending on team size, you can easily wind up being forced to give up a quarter of your weekends for the year (as I had to). I haven't worked on-call for software, so maybe it's different, but I doubt it's really too different. After 5 years of all that lost time, the amount I'd have to be paid to go back on-call is so astronomically high no company would ever pay it. Actually, now that I have a kid, there's probably no amount I could be paid if it means missing time with them.

Silhouette 3y ago
It seems like OP and their CTO would like to have reliable 24/7 operations but do not have the skill to do it by designing systems that won't fail catastrophically out of hours or the budget to do it by hiring dedicated staff to be available at all times to fix problems. So instead they are trying to abuse their existing staff. The correct response to this IMHO is for the existing staff to collectively tell management where to go. In life you can't always get what you want if you don't have the money or make the effort it requires.

pyjarrett 3y ago

What's the rationale for setting up the on-call schedule? Is the system that unstable? > on call at least once a week, including the work day and evening, once a month, including the weekend of that week. > I should also note there is a required 15 min response time to pages. The person MUST be able to be online and fixing the problem within 15 mins of the page. I've worked a position where I had extra duties on the weekend, so I effectively worked 12+ days in a row many times. I've also been on an on-call schedule like this with a short response time. Both of these severely impact your personal life. You have to realize that the two most valuable things you can give engineers is their own time back and money. This is how I would probably try to handle it: To start, it would be opt-in and not mandatory, as this type of responsibility is difficult for those with small children or other family members, such as elderly, needing care. The 15 minute response time also means you should be being paid an hourly rate (maybe even 1.5x or 2x) while being on-call, whether there's an incident or not. This is because it severely limits what you can do -- you're basically chained to your laptop and an internet connection. Anyone responding to an on-call event after hours should also be given a half or full day off the next work day. Weekend on-call should at a minimum be given a day off the following week, preferable two.

kelnos 3y ago
> Is the system that unstable? I'm generally anti-on-call, but c'mon, this is an unreasonable standard. No system is 100% perfect, never needing manual intervention to get things going again when something happens. Any business that has a service that is supposed to be running 24/7 will likely need some form of on-call.
- Silhouette 3y ago
  If you have a business that needs to operate 24/7 then you need your Operations team to hire accordingly and then on-call should normally be unnecessary. Operations should be able to fix an emergency that is unsafe, insecure or illegal. Anything else can wait until the next working day even if management might prefer it not to because that costs money. If a business can't or won't make those arrangements (and pay the bill for them) then maybe its management team need to recognise that fully reliable 24/7 operation is not something they can guarantee and they should plan accordingly.

CapitalistCartr 3y ago

Money won't answer this problem. Hire someone(s) to cover this is cheaper.

glyphy177 3y ago

[dead]

theshrike79 3y ago

Over here the standard usually is 50% hourly pay for being on-call. On-call requires you to be 15-30 minutes away from whatever you can use to fix issues, usually work laptop + internet connection. Oh and you need to be sober (...enough to answer the phone while sounding sober =) As soon as something happens, you jump to regular overtime pay so if it's on a weekend+Sunday, you're making around 200-300% pay. If it's just after others logged off, you're getting normal pay, maybe 50% extra depending on the company. I know a lot of people who have paid off loans just by hogging all the on-call hours they can. It's not especially hard if the playbooks are well made and nothing goes horribly wrong.

karmakaze 3y ago

The times I've been on-call as a sr backend dev, my compensation was one of - $700 for any week that I was on call (whether or not there were any incidents, and more often than not there were none) - paid my phone bill, roughly $100/mo (including data + amortized phone cost) - free to take off the next day any time I'm paged off-hours. There have been other times I was unofficially on-call (at small startups/companies) with no compensation, but I don't think there was enough scale/frequency for it to be a sticking point.

joshka 3y ago

This feels like a possible troll. Also by the same person: I work from home as a software engineer and my job is happy with my performance. But I'm putting in little effort. Am I a bad person or employee? [1] --- I used to be a Senior Software Engineer at another company but recently I took a lower level 2 position at another company remotely. I am very good at what I do. I can usually take a well groomed agile story in any tech stack and complete it way ahead of schedule. The company always praises me for this. But the truth is they cannot keep me busy. I know I can help out else where but my output is already so very high, especially compared to everyone else. So instead I just sit around all day playing video games while I wait for my next work assignment. This can sometimes be days or even a week. I don't go looking for it. I have so much free time on hands that I am seriously considering getting a 2nd full time software engineering job remotely and doing this all over again. No one is the wiser. Does this make a bad person or employee? [1]: https://philosophy.stackexchange.com/questions/91282/i-work-from-home-as-a-software-engineer-and-my-job-is-happy-with-my-performance https://philosophy.stackexchange.com/questions/91282/i-work-...

MollyRealized 3y ago
Well-noted.

simne 3y ago

From my own experience, I understand, being on call could be important for engineers, but CEO MUST understand, not all people could accept this. So, in real life, need agile approach. Some people just accept to be on call all time, you could consider them as your core team with appropriate benefits and value them, and respect them, really. Second group of people could accept receive calls during their shift. This is also not bad. Value this. And exists people, who are good workers, make great value, but could be on call very limited time or nearly no time. You should decide, with them individually, how you will work. It is really possible, some very valuable people will not be on call at all.

dogmatism 3y ago

*laughs in Medicine

EPWN3D 3y ago

> Should we just PIP the engineer No, fire them without cause like a grown up and take the hit to your unemployment insurance. Don't put them through some three-ring circus act masquerading as employee development. It's undignified and disrespectful. You and the employee have a disagreement over the job description -- this not performance related. Don't try and spin it as that. It amazes me that these B-school pricks managed to enshrine at-will employment across the country, and they still won't just fucking fire people like adults. No, instead we got one of the worst MBA-isms visited on corporate America, the PIP. Fucking cowards, the lot of them.

Nellyz 3y ago

[dead]

Nellyz 3y ago

[dead]

gina00 3y ago

[dead]