7 ms·
I just want to jump in as a minority voice here. In case anybody is reading the other comments and feeling... alienated. I refuse to accept on-call duties, ful
by sed_zeppelin 2y ago
I just want to jump in as a minority voice here. In case anybody is reading the other comments and feeling... alienated.
I refuse to accept on-call duties, full stop. If a job posting expects it, I don't apply. If a hiring manager says they have it, I do not accept the offer. If management starts talking about maybe implementing it, I protest. If it becomes enacted, I resign.
There is absolutely no situation in which I will ever participate in another on-call shift. I've been there, I've done it, now that chapter of my life is closed. Find some younger kid, pay them better than you paid me for the miserable intrusion on their life. I'm done.
Just wanted to be the voice who says what, hopefully, some of the more seasoned and battle-scarred readers here are thinking.
- convolvatron 2y agoon call is like hiring civil and structural engineers to build you a bridge over a canyon, and then when they show up to do a site inspection you just push them in. eventually maybe you'll be able to cross.
- ddingus 2y agoI love your comment on this. Perfection. Once, while traveling in an RV for some work related marketing thing, the discussion turned to the lack of fuel economy... The RV might perform better if the engine powered the RV by blowing fuel right out the tail pipe. Horrible efficiency, terrible for the planet, and, and all the negatives packed right into a quick expression. Your comment is on point. Solid and I just felt like sharing my appreciation for the morbid fun it contains. Nice work. Worth a healthy chuckle. Thanks.
- stackskipton 2y agoAs SRE, strongly disagree. On Call is like hiring civil and structural engineers then holding them responsible when their poor bridge collapses under the weight of all the traffic. Sometimes, yes, Devs get called out for stuff outside their control like infrastructure failing. However, at my job, we just had two devs that quit over on call and guess what, their service was one of worst offenders in "Opps, we pushed bug to production."
- convolvatron 2y agofirstly, on call means supporting the entire service. not something in general I built. secondly, many if not most of the issues that arise are part of some infrastructure automation or third party service or database. expecting me to be fluent in all of those to be useful in the hot seat is a pretty substantial investment and qualifies me to be an SRE on top of my other duties thirdly, one major reason why my code might fail in production is that it wasn't sufficiently tested, probably because the service as a whole is basically untestable, and even if it were, building test and test infrastructure is likely not at all valued. in many places just filling in that hole would take a year. onto to the fourth, the story is supposed to be that by operating the service, I'll be incentivized to fix automation and come up with solutions to make it more robust. I actually know how to do this, and every week I'm on call is time that I _dont_ spend doing this. furthermore, getting permission to do so is often like pulling teeth. sounds complicated. sure that would be nice, look at that when you have time in the indefinite future. so what this often looks like from a development perspective is that I'm being paid to be a developer, I was judged based on my ability to be a developer, but at the end of the day I'm not building the service. I _am_ the service.
- stackskipton 2y agoIf you are on call for infrastructure, then I could understand not wanting to be on call. If I'm there, I'm on call for infrastructure as SRE. I get all political reasons that your code may not work. However, refusing to be on call doesn't fix any of those reasons, it's just ignoring work. Flip side as SRE, I ask if Devs are on call. If they are not, I don't take the job because there is zero incentive for them to fix anything vs churn out 5 features, chuck it over the fence and be like "Ops problem now"
- deleted 2y ago[deleted]
- CoffeeOnWrite 2y ago> but at the end of the day I'm not building the service. I _am_ the service. I agree. For me though, it gives me pride to own my services and be fully accountable to the business, especially as part of a team with whom I build comradery, and of course our value to the business justifies our good compensation. It only works because we are empowered to make decisions that keep our on calls sustainable.
- acchow 2y agoIn software, building bug-free software is almost never the goal. There is a constant juggling act of tradeoffs between time, requirements, and tech debt.
- tengbretson 2y agoThis attitude will keep you off the pager rotation, but get used to building meaningless projects or having your expertise relative to the average developer seen as a liability to the org rather than an asset.
- gedy 2y agoI disagree, not the OP but there is a time and place in a career for firefighting, similar to military. I always take responsibility for my own work, even after hours fixes, etc. But active on-call orgs usually are just reaping tech debt that others sowed. Sorry not going to rally for that.
- corytheboyd 2y agoNobody is rooting for on-call, but yeah I put up with it because I am a young stupid idiot thirty-something who needs to make a lot of money now so that the rest of my life can be Nice Enough. I’d love to be able to cherry-pick jobs like this too, but I am not there yet. Not trying to dunk on you, I’m honestly glad you get to do this, it must make your life considerably better.
- 0xbadcafebee 2y agoI am 40 yrs old. I get paid a shit-ton of money (just around $200K) to do this stupid tech work job. I work 40 hours a week, I get benefits, flex time, plus I work remote. If I'm getting paged for a legitimate issue that is related to something I built or maintain, then, yes, I am going to respond on-call. Because it's a fucking privilege to get paid this much money to sit on my ass and type into a screen. If I'm getting paged repeatedly, or for an issue that isn't my responsibility, then I will get pissed off, and yell and scream until I'm no longer on-call (or they fix the issue, whichever comes first). But I am grateful to be able to have this life. I can spend an hour or two after hours to fix my shit that broke.
- majormajor 2y agoAn on-call rotation without sufficient influence over the roadmap and planning to be able to fix persistent problems so they don't repeatedly cause the same issues over and over and over is toxic. And it's gonna kill the team's overall productivity so it's not good for management either. Congrats, you're playing SWE salaries for an ops team that would traditionally cost you less otherwise. In a more healthy situation an on-call rotation is the price of being able to move quickly, get stuff out the door, and have compensation that reflects that the company isn't paying a whole team of extra people to stare at dashboards 24/7 just for the rare situations that things break after-hours. Gigs with low-overhead + customers that don't expect 24/7 operations are kinda the real sweet-spot dev compensation + role-wise, but ... pretty rare.
- 0xbadcafebee 2y agoWell, I have two thoughts about that: 1) gigs without 24/7 operations are rare, because there is no good reason for a tech product not to be 24/7. it's not costing extra electricity to keep the lights on overnight, nor more staff. there are a bunch of these gigs (my last gig had no customers for 2+ years) but you shouldn't expect them, because part of the reason we're paid so much money is we're expected to deliver "continuous value". most devs would agree with this, because they all want to be able to deploy continuously, whenever they want. (which is a terrible idea, but it is the status quo.) furthermore, if you're doing your job right (and so is Ops), supporting a 24/7 product should not result in on-call pages, because nothing should be breaking outside regular business hours. if it is breaking outside regular hours, somebody sucks at their job. and Ops' job is pretty simple, so... 2) you do have lots of control over the roadmap, planning, etc. but nobody is going to walk up to you and say "hey we were just thinking of maybe doing this in the roadmap, is that okay with you?" you have to get involved, early, and consistently. you have to show you're not going to rock the boat, but that you will have good suggestions, and can show they will turn into better outcomes. you have to play a little politics, a little product ownership, and also an engineering role, in order to influence what the business decides to do. as you get more senior this gets easier because people will defer to you more, but even an extremely likeable junior can influence the roadmap. on the off-chance that you're just trapped in engineering hell, with hostile management, a terrible product, and a completely apathetic and terrified staff, quit immediately. this isn't normal and you shouldn't think "oh, I'm trapped here." people don't stay in abusive relationships because there's no other choice, they stay because they've justified their own abuse.
- deathanatos 2y agoYou expect to not be responsible for what happens to the software you put into production? (… and I'd like to avoid distracting arguments that amount to "my company does on-call badly" — yeah, those problems do exist and we should strive to fix them. But if I'm to not categorize the argument here as the baby with the bathwater, then we need something to replace on-call with. Prod goes down on a Saturday afternoon; are you going to tell management "tough cookies" until Monday?)
- rufus_foreman 2y ago>> You expect to not be responsible for what happens to the software you put into production? I'm responsible for the software I put into production from 9 AM to 5 PM for about 200 days a year. At 3 AM, I am responsible for taking care of myself by getting a good night's sleep. If you need 24 hour coverage, taking into account vacations and weekends, you need 5 or 6 people.
- hn_go_brrrrr 2y ago"you need 5-6 people" is moving the goalposts. The root comment said nothing about minimum team size.
- mikedelfino 2y agoIf the company has enough people in the team, someone just works the night shifts or on scheduled weekends. No one needs to be on-call because there would be someone taking care of it already.
- nosefurhairdo 2y agoIs the argument here that every software team should have engineers whose normal working hours have 24/7/365 coverage?
- SuperNinKenDo 2y ago
- deleted 2y ago[deleted]
- dheera 2y agoI 100% fully agree with you. I have survived 2 cardiac arrests (almost died) during high-stress times. I've been stable for a few years now, but only after I enacted VERY HARD boundaries around work/life and never cut down on sleep for any reason (among other health-first changes I made). I have a significant increase in cardiac arrythmias any time I don't sleep enough. I consider myself at this point as having a disability that prevents me from overworking, and I absolutely need my employers to respect that and accommodate that. I can work normal hours, and that's my offer. If you want to pay me less, that's okay, but I'm not doing on-call unless it's business hours only. If customers give a shit about uptime at 2am then it's management's responsibility to find people in other time zones to deal with it, or pay extra for people who are willing to sacrifice and risk their health for a customer (I won't take that deal though).
- nosefurhairdo 2y agoI've been the on call engineer on my team for 75+% of the last year (most of my team is contractors, new hire not onboarded to on call rotation yet, etc.). It's not an issue because we don't break prod. I also feel I'm well compensated. When there have been issues at inconvenient hours, my manager has encouraged me to take it easy after resolving the incident. We've also prioritized improving our integration tests and addressing other issues noted during root cause analysis (RCA), which I suspect is why we haven't had any incidents in recent memory. If on call duties are this frustrating, I'd argue it's team/organizational dysfunction that is the real problem, and bad on call shifts is just one of the symptoms. Ultimately, somebody needs to be available to fix a production incident. One person suffering from on call duties is better than thousands of paying customers suffering from broken software.
- alemanek 2y agoOn call even if you aren’t actually called is still a burden. No drinking or other impairing substances. Need to be available and ready to help on weekends. So unable to disconnect and go on a hike or some other activity without internet and your laptop. 75% on call even if I was never called would be profoundly unhealthy for me. So I wouldn’t dismiss the toll of just being available 24/7. EDIT: I forgot to mention I am on an on call rotation but it is 1week on and 7 weeks off. So, not too horrible.
- nosefurhairdo 2y agoThis is a good point. I am fortunate to have a good manager who recognizes the unfair burden. I have missed a pagerduty notification before, which my manager dealt with. The incident did not appear to affect my subsequent performance review, as evidenced by top of band compensation. I would expect stricter accountability with a more reasonable on-call schedule.
- darkwater 2y agoCongrats, you have a very good work ethic but a very poor personal health ethic. I was like you, and probably still am deep below, and I will fall again in the same trap. But please, try to think about flipping your point of view here and instead of being your manager generous and your company good for not taking into account the time you failed to answer on duty, think about how you are being exploited covering 75% of on duty alone, and the money the company didn't loose just because of you. And how much of that money you got.
- renewiltord 2y agoI think the useful thing here is to mention the trade-off. Practically all my SWE friends making $1m+ are instant responders whether denoted so or not.
- throwaway2037 2y agoI know HN doesn't like jokey replies, but you wrote: "all my SWE friends making $1m+", where "friends" is plural. You have multiple friends who earn more than 1M USD total comp per year? Man, HN is getting crazier by the day.
- darthrupert 2y agoI had on-call jobs at one point in my career. It roughly doubled my salary, and I put most of that into assets that rose about 20,000% in 10 years. So that was nice. The job can be excruciatibg though. Don't do it without proper compensation.
- srhtftw 2y agoAt last, a voice of reason amid the vulgar crowd. I've held several management positions where I've carried a pager. At one employer I helped keep trading databases operational in 7 time-zones on 3 continents. At another I helped fix a backup issue on Christmas eve. Helping those customers was a core part of my responsibility. I fully understood that and took great pride in it. But as a developer I too will never accept another on-call rotation. Companies which assign on-call duties to developers make the mistake that development, management and operations are different kinds of work which require different environments and skill-sets. Other engineering tasks include testing, documentation, training, and maintenance. At small startups the founders and early employees may do some or all of these but that becomes impractical at larger established businesses. Engineers should learn and do all these things in the course of their career but not all at the same time unless quality isn't a concern. My experience at a unicorn a few years ago convinced me companies which assign developers on-call rotation either don't understand or don't care about the quality or sustainability of their business. In that company senior management was replaced by folks from Google and Facebook shortly after I joined. I was moved into a team where I had no role in the design, develop or deployment of its services. I had no say in the hiring or firing of the so-called engineers who rushed failing services into place past a wholly ineffective QA department. I should have seen the writing on the wall when I began to be pressured by managers and recruiters to rubber-stamp candidates who couldn't pass our coding tests but had spent lots of time on-call. The company's priorities slowly became clearer to me as they grew evermore desperate to live up to their promises. Ultimately I suffered an ischemic attack from the stress of this environment and left the company to focus on my health. Oh and the company? It let go of most of its engineers a year later and was eventually acquired by competitor for a few hundred million after having raised over a billion dollars.
- 11mariom 2y ago100% agree. But many companies does not understand when I tell them "nice offer, but you should consider someone cheaper for on-call duties, and I can do the rest". When I was younger I was (kinda) happy to take on-calls, as it was a significant boost in my income. With more experience and better base salary… I simply does not need it and… well - I want to LIVE my life, not WORK (yes, being on call is work, not resting time).