9 ms·

Take this on-call rotation and shove it

191 points by mirawelner 2y ago

dylan604 2y ago
" insure against every possible thing that could ever go wrong, they would have to build a second studio on a separate part of the city’s electric grid, with redundant copies of all the equipment and broadcast content, along with a full crew of understudies ready to take over at a moment’s notice." WTH?? I guess this person has never heard of backup generators? Every broadcast TV station has them.
- deleted 2y ago
  [deleted]
- beepboopboop 2y ago
  That just covers electricity. They seem to be implying coverage of a multi-failure scenario.
- cfraenkel 2y ago
  To begin with, airplanes do fall out of they sky, sometimes right on your backup generator. (speaking from military experience where yes, there is an entire backup studio waiting to take over, just in case. Or rather, the 'studio' is geographically dispersed, with 100% redundancy, which is another way of saying the same thing.) But more importantly, *this* is what you noticed from that article?
  - dylan604 2y ago
    No, but this was the point where the "let's make something up" got to be too much. I'm not talking about low budget UHF channels, but TV stations I've been in and around all have multiple studios. If the switcher in Studio-A goes down, the signals can be routed to the control room for Studio-B. Also, Alex the know-it-all is such a forced thing that is just ridiculous and eye roll inducing. Anybody that is a jack of that many trades is a master of none of them. The entire forced analogy just got to be too much and I lost interest before a point was ever made.
- deleted 2y ago
  [deleted]
- Macha 2y ago
  Also, I don't think broadcast TV news is quite as reliable as the post makes it out to be. Like this exchange happens all the time: "and now we're going to our on site reporter, Onda Premises" <45s silence> "Oh, it appears we've lost them for now, we'll cycle back later. In our next story...."
  - dylan604 2y ago
    Going to a remote is not the same thing as backup/redundancy within the studio. The broadcast was never interrupted. The latency between remotes can be mindnumbing, and with inexperienced reporter/anchor stepping over each other unable to sit through the delay it's pain inducing for the viewer. But that's an unrelated tangent
boznz 2y ago
What they don't tell you about working for yourself is the fact you can be effectively on-call 24x7 every day. I am currently supporting four wineries that are processing thousands of tonnes of receivals 24x7. It happens for two months of the year and I am expected to be available from 06:00 to 22:00 during that time, there is no phoning in sick or having a lazy day, I work alone and only have one reputation. I don't want to be that contractor forever known for destroying a clients business. You can only do this for so long though, when two or three problems come in simultaneously it can cause issues as you drop something halfway through when something more important comes in. I once executed an SQL update query without a where clause under this kind of pressure, and ended up working until the next morning to recover, only to start again at 6AM. I have even had land-line calls at 2AM to bypass my mobile restrictions. The rewards are great, but don't let anyone tell you it is always easy. My current system is 16 years old now and I know all the ins and outs so it has been pretty easy to keep on top of things the last several years, however I am glad the replacement system is nearly written and it will be somebody else problem in 2026.
- netruk44 2y ago
  There's a big difference, though. In your case, you're the only employee of your business. And if you're not there the business will literally go under. And you also get directly rewarded for being there. I would guess that being 'on call' in this manner is possibly less draining on a person's soul (depending on how well they tolerate the risk of owning a business). Contrast that with being 'on call' for your megacorporation, who isn't giving you anything extra for your on call time because they 'already pay you enough'. And where the only negative consequence for the company if you fail to immediately respond within 15 minutes is that some executive in the company is kept waiting longer than 15 minutes, or some ads aren't being shown for 15 minutes. But if you aren't there, your boss is going to get a phone call and that's definitely not going to look good on you. And there's no bonus for fixing the problem, that was already your job in the first place. Sucks that you had to do it outside of scheduled hours, oh well. I'm with the author of this article. Take your on-call rotation and shove it (if you're a large corporation). I'm fortunate enough to be able to take a firm stance on this point, and do so happily.
flerchin 2y ago
Jeez I guess what we do is industry standard best practice, and it sucks.
- darioush 2y ago
  deferring to best practice instead of best judgement is a major plague of the software industry these days. best practices usually come from giant companies with tens of thousands of engineers like google (who doesn't seem to be keeping up with competition btw) and amazon (which is notorious for burning out people). what science or evidence drives the best practices?
  - cratermoon 2y ago
    > what science or evidence drives the best practices? management "science"
yodsanklai 2y ago
Excellent article. I can relate to a lot of it. The sad part is that we can't even control the quality of the systems we're oncall for. We're pushed by management for new features, not for robustness of the tools. Also some systems have no clear ownership, so nobody has an incentive to fix them. It'll be next oncall's business. Oncall is really the worst part of my job. I can stand long hours but this is something else.
- mortar 2y ago
  I feel for you, I’ve also suffered through this a lot over the years, and am finally at the stage of career and wisdom to start pushing back on the quality that I can’t control and ensuring that others are equally as accountable for their mess. For one particular occasion , once we took blame out of the equation (at least within the engineering team) and started doing Post Incident Reports, the incentives finally became clear for the business as we were able to compile a list of recurrent issues during every issue, calculate a financial loss and present it for inspection each and every time they either began a witch hunt for downtime or refused to allocate time to backlog. Small wins.
- cratermoon 2y ago
  One of the sidebars mentions that: "The production system in question is almost certainly a schizophrenic box of compromises brought about through poor decision-making, unaddressed technical debt, design-by-committee, and impossible timelines and budgets. This is not a system that any single rational human being on the team would’ve chosen to build if permitted to do so alone. Trying to assert ownership over an environment like that is just begging to get your shit rocked."
  - sheepscreek 2y ago
    That’s basically every company and every system ever. Things are always in a state of flux, constantly being worked on. People come and go, priorities change and technologies evolve.
    - cratermoon 2y ago
Kwpolska 2y ago
That seems like a very long-winded way to say you hate on-call, which is a completely normal thing to do. That said, is on-call effectively mandatory or very popular in the US startup world? Because here, in the European established company world, I can’t really recall seeing a job posting with on-call listed.
- spongebobstoes 2y ago
  on-call is ubiquitous in the US tech industry. I've never had a job without it.
- Carrok 2y ago
  I've been on-call (sometimes on a rotation, sometimes always) for the past 10+ years. So, yes. It's common.
- qwerpy 2y ago
  Curious how this is the case. Do European companies not provide 24/7 services? Or staff a "follow the sun" model so no one has to answer pages outside of working hours? Do Europeans write better code so they don't need on-call? I've worked at a few FAANGs in the US, and every single one after 2010 had an on-call rotation.
  - Macha 2y ago
    I think it's mostly down to three factors: 1. Uncompensated oncall is legally tricky in many EU companies, so a lot of midsize companies look at the cost of paying for oncall, or sometimes just the time of administering paying for oncall, and decide they can do without. High frequency oncall is also often restricted (e.g. more than 1 week in 6 is not legal here) 2. A lot of the smaller companies are europeans selling to europeans, and are much more used to a business culture of availability during office hours. Especially there's a bigger share of like b2b back office stuff in europe compared to like, restaurant POS systems. 3. Larger companies do seem more into follow the sun. A lot of the big tech in europe are subsidiaries of US companies, so if they're in Europe it means they've already opened one remote location, and therefore are more likely to have another (California, Europe, India is a super common arrangement)
slt2021 2y ago
being oncall forces the quality of software to improve. if you want fewer incidents: ensure better QA, monitoring, smaller rollouts usually developers start becoming more conservative after they do few oncall shifts and suddenly prioritize important reliability improvements, instead of shiny new features nobody will use
- geoffpado 2y ago
  Being on-call forces the *desire* for the quality of software to improve. Shitty management can and will override that. We don't have time for QA or to waste an engineer adding monitoring, we gotta ship ship ship.
- darioush 2y ago
  this doesn't always work. many things can go wrong in distributed systems and you cannot test for all of them. also you have no control of your dependencies like when AWS networking degrades or a 3rd party API provider changes their APIs without letting you know.
  - toast0 2y ago
    I think if you have a reasonable environment, where on-call feeds back to development (which is what OP is suggesting, more or less), you will absolutely get woken up for networking problems, because there's not really an alternative. Maybe some thresholding to allow for minor problems without alerting, but you know. If it's a big enough problem, someone has to fix it, and it doesn't matter if it's your problem or your dependency's problem, it breaks your service so it's your problem. If it happens a lot, you look for another network to run on. For 3rd party APIs, if they're not critical, you start to develop kill switches. So yeah, someone has to wake up and handle it, but all they have to do is set the kill switch and go to sleep. Personally, I did dev and on-call for SMS/Voice verification codes. Most of the time, that's in the nasty corner of it's super critical to the application (users can't use the product if they can't get a verification code) and it depends on 3rd parties that have three nines on a good year. In my case, I got tired of dealing with the disruption and developed automated routing that could manage most providers taking an outage without needing me to take action. Results could be better if a human took notice and action, but it was good enough most of the time, and partial outages were much easier for the automation to handle it. Even if there's no way to do something like that, at least automation can take care of 3rd party API is failing hard, so mostly return errors quickly without trying the API and only let a small fraction of requests go through to sample if the service came back online. That can keep your servers from getting overwhelmed, as well as drive the alert that helps you wake up, yell at the vendor, and decide if you can go back to sleep while the system takes care of itself. When on-call is disconnected from development, that's when it gets really miserable. If you can swing a shift-work/follow the sun operator job, that's certainly better than on-call where incidents are common and there's no feedback loop to reduce things. It may well be better even if there is a feedback loop, but the feedback loop in that case requires explicit communication and effort; if I'm on call for my own work, I don't have to tell me to not push shit code right before I leave for the day or go to sleep, I'll get that message from myself right away. If someone else is cleaning up after my messes and doesn't communicate the effects to me, I might never know.
Animats 2y ago
For "non-exempt" employees, that's paid "stand-by time" California.[1] Also see this case involving on-call coroners.[2] The way this works in most unionized jobs is that there's a stand-by rate paid for on-call hours, plus a minimum number of hours at full or overtime pay, usually four, when someone is called to duty. This is useful to management - if the call frequency is too high, it becomes cheaper to hire an additional person. [1] https://www.dir.ca.gov/dlse/CallBackAndStandbyTime.pdf https://www.dir.ca.gov/dlse/CallBackAndStandbyTime.pdf [2] https://casetext.com/case/berry-v-county-of-sonoma https://casetext.com/case/berry-v-county-of-sonoma
mjcarden 2y ago
This article gave me unpleasant flashbacks to the first half of 2023. I resigned from planet.com in mid 2023 due to the stress caused by being on-call every second week. It took me six months to get my head into a healthy state again. Now I have a much better job, better paid and no possibility of on-call, ever.
purplejacket 2y ago
Here's an idea: Compensate any on-call work received during off hours at 10X the normal hourly rate. E.g., if my salary is $150K per year, then my hourly pay rate is about $75 per hour, so compensate my on call work at a rate of $750 per hour. Thus if I get a call at 10pm, log in to my laptop and work for 30 minutes to resolve the issue to a satisfactory level, then I pocket $375. That puts a financial incentive on companies to structure their on call protocols so that only the most important calls are handled. And I can envision variations on this theme. Different sorts of on-call disasters could offer bids for how much they're worth to fix based on some automated rubrick, and anyone on the ENG team could pick these up on a first-come, first-serve basis. Or various combinations of the above for a guaranteed backup person. But the companies should offer enough incentive to make it worthwhile. And this is in the companies' own best interest. To maintain a workforce that can think clearly during the normal work, to have a good reputation in the industry, to get good reviews on Glassdoor, etc.
- mortar 2y ago
  Good suggestion and I can see the benefit for honest people, but unfortunately it’s as equally a system for financial abuse for others - sometimes enough to prevent people fixing things during their regular hours just to benefit at other times. A good counter balance to this might be to offer even more compensation for no incidents, or otherwise well handled incidents that go on to squash types of that incident now and into the future.
  - alwa 2y ago
    If I’m rewarded for no incidents, doesn’t that mean I’m punished for there being incidents?
    - mortar 2y ago
      Not necessarily, but agree that bad managers or organisations have a tedency to do this. I guess the ultimate goal is to keep everyone happy right? Everyone has different ideals, you can probably assume the business wants everything to work by spending as little as possible, employees want to be paid as much as possible while enjoying their job. Striking that balance is always a challenge.
dadkins 2y ago
I just want to point out that the answer is shift work. Here's an example of an SRE job at a national lab: https://lbl.referrals.selectminds.com/jobs/site-reliability-engineer-7041 https://lbl.referrals.selectminds.com/jobs/site-reliability-... "Work 5 shifts per week to monitor the NERSC HPC Facility, which includes 2 - 3 OWL (midnight - 8am) shifts. Some days may be onsite, some may be offsite. The schedule will be determined by staffing needs." 40 hours per week, full salary, full disclosure about the night shifts, but none of this 24x7 wake up in the middle of the night on top of your regular job bullshit that the tech industry insists on.
- lolinder 2y ago
  Shift work sucks more than on call for me. I like having a flexible schedule that I can work whenever I want, and I'll happily take the remote risk that I'll get paged during my one week per quarter on call rotation in order to guarantee that no one expects me to be on during set hours the rest of the year. I think the problem isn't on call itself, it's that a lot of companies suck at on call. If your rotation is every 3 weeks and you get woken up in the night at least once per rotation, then yeah, that's awful. But the problem you have in that case is that stuff is always on fire and you don't have enough people on the rotation, not with on-call as a concept.
  - hx8 2y ago
    I'd take a night-shift job over being on call ontop of a 9-5. I think the real answer here is to have team members on multiple continents. Have some team members NA/EU/Asia then it's always reasonable hours for someone to deal with the production problem. High priority issues can be worked on around the clock without anyone working overtime.
    - fsckboy 2y ago
      I know plenty of friends (mostly medical) who've had "on call" shifts on top of their 9 to 5, but in those cases it was pretty exceptional for them to actually receive a call, and the "interruptions" would be a small number of hours.
smackeyacky 2y ago
That article made me shudder with echos of having what we used to call “beeper madness” back in the 1990s. After a while of being on a roster of on call weeks, anything that beeped would make you reach for that pager on your belt. As a kid the first few weeks were kind of exciting as it felt like you had been elevated to a new level of responsibility. Once that wore off it was obvious what a cage it was. I don’t miss pagers.
- hinkley 2y ago
  I refused a contract to hire that was talking pager duty. I saw how they freaked out about things outside the team’s control during business hours. The first time someone called me after 11 pm I was going to get myself fired talking to them.
chris_wot 2y ago
There is another option. The on-call person just does a deliberately piss-poor job of resolving the problem. I mean, they resolve it but they make sure it takes a hour longer than necessary. What are they going to do, fire you? If they make life hard for you, then get another job. The shoddier your work outside of your normal hours, the better. You can have quality, speed and cheapness, but you can only pick two.
- sitzkrieg 2y ago
  at a prev us tech job, a few years in they made all engineer have oncall, with no compensation added whatsoever, and on call for other teams code in india. i got pages a few times and didnt exactly rush to ack. not slow but didnt rush. the rotations increased more and more (mostly from other US employees quitting) till i quit. scummy company
- nijave 2y ago
  I think there's also some middle ground where you don't go out of your way to carry a laptop but you do best effort while maintaining a normal outside-work life.
denkmoon 2y ago
I was on call in a 4 man startup for a 1 week rotation for about 9 months, 6 years ago. I still have an anxiety reaction when my phone rings. Can very much relate to the author's thoughts about PTSD.
nhumrich 2y ago
The difference with dev oncall vs doctor on call is that it is self inflicted. Why are you getting paged? Because you built the system. Either your system isn't resilient enough, or you have noisy alerts. Both are problems you should be motivated to fix. I have been on call 24/7/52 in SRE roles most of my career. It has either sucked hard, or not at all. And the time it sucked the most was because every single practice was bad. And now, I build better things because of if. Paying me more for on call wouldn't have changed how much it sucked. It wouldn't have made any material impact on my actual quality of life. But it would have done two things: 1) made me feel like I can't complain 2) give me less motivation to fix it Paying for on call doesn't seem like a win. I want happy employees, not disgruntled but silent ones.
- croshan 2y ago
  This works if you're on call for your systems. In many situations (ranging from small startups to big tech), you're also on-call for the systems of sister teams. Not that there aren't other ways to fix that. But fixing the erroring service isn't practical in all cases.
- cempaka 2y ago
  We have integrations with dozens of external vendor APIs for which it's essentially impossible to disambiguate ahead of time whether any given error might be on their end or ours.
- eestrada 2y ago
  I think paid oncall could work if oncall is voluntary. The more oncall sucks, the less likely team members are to volunteer because the pay isn't worth it, then the company/team needs to pay more as an incentive to get people to volunteer for oncall. Eventually the price is so high that it becomes cheaper to just build the system correctly and stop shoehorning features in with no regard for stability. If oncall burden is light, then everyone volunteers because it is an easy way to make a bit of money. However, it is a huge systemic change to move towards a voluntary model. Not sure how feasible this really is.
  - 2y ago
lopatin 2y ago
The OP needs to write with some more focus. Most of this reads like a very long rant by someone who was woken up too many times recently. > We need to talk about Kafka No we don't, that entire section was irrelevant.
- lazide 2y ago
  Well yeah, do you think people who are well rested and non stressed have oncall rotations they want to shove up someone’s nether regions?
nullorempty 2y ago
Yeah, I can relate to people saying they nearly got PTSD, I sure did get it. Paging apps use seriously offensive alarm sounds. I hated every sound they had in the options. It made me instantly sick. Fuck that!
- linsomniac 2y ago
  The right answer is, in my experience, something that starts soft and intermittent, and then gradually ramps up in loudness and annoyance. In bed, it's enough to have a harpsichord gently playing, but if I'm watching TV or out at a restaurant I might need it to eventually escalate to a klaxon to get my attention. My first experience with this was Groovy Blue: https://youtu.be/WCbyp_UuyoM?si=s0ZpfcXZKw7dP17i https://youtu.be/WCbyp_UuyoM?si=s0ZpfcXZKw7dP17i
- smcameron 2y ago
  The (in)offensiveness of the sound has little to do with it. For quite a few years after I left my job as an SRE at google, this gentle little tune called "morning flower"[1] which I used as my pager alarm sound would give me a little jolt of adrenaline. Even now, it's a bit startling, but it's old enough now that at least I do not hear it in the wild anymore. [1] https://www.youtube.com/watch?v=4hr8lhO3k5o https://www.youtube.com/watch?v=4hr8lhO3k5o
tootie 2y ago
There's a great option for small companies that aren't amenable to on-call that is underappreciated. Hire an MSP. There are companies whose entire business model is having a geo distributed team with a stack of automated monitoring and run books for multiple clients. You train them, pay a set fee and never ask anyone on your team to be on call.
srhtftw 2y ago
This is an excellent article. I've commented before¹ why "on-call" as described is bad because it conflates roles and responsibilities and robs developers of resources, but this goes quite a bit further and explains why it's a bad practice that eventually leads to burnout. With the exception of companies like PagerDuty, the sooner this practice is ended the better off we'll be. ¹ https://news.ycombinator.com/item?id=43400898 https://news.ycombinator.com/item?id=43400898
staplung 2y ago
One of the biggest problems with on-call rotations is that you're actually incentivized to do it poorly. Every minute you spend doing on-call work is time that you can't spend on the things you've actually been assigned to do. You're never going to put your on-call work into your performance reviews; doing so might actively work against you. "I see that you spent time tuning alerts and updating the runbook. That's time you didn't spend on the actual tasks that were assigned to you." If it's better to spend the least amount of time doing on-call work then the logical conclusion is that it's best to snooze as many alerts as possible until they either go away on their own or roll over past your rotation. Fixing the underlying problem might be worthwhile if it's something that you can fairly easily fix but if the on-call rotation is more than 2 people, the underlying problem is mathematically unlikely to be of your making and is it really a good idea to make a habit of fixing other people's broken code? What's crazy is that I've never seen anyone with on call duties acting in this worst case bad faith manner. Companies basically abuse the work ethic of their employees because it's the cheapest possible way to check that box.
- xigency 2y ago
  It's just more hazing so the underclass of engineers don't realize they actually build everything big tech is selling and more.
- chronid 2y ago
  > Every minute you spend doing on-call work is time that you can't spend on the things you've actually been assigned to do. In my experience at least if you're oncall during a sprint you would have less work assigned to you than otherwise (2 week sprint and 1 week you are oncall? 50% allocation) as the expectation is that week you will spend responding to alerts, or investigating issues, or even improving alerting and dashboard and fixing bugs. If this does not happen, devs don't push for it and management is completely blind to it you have an organization issue. If leadership does not care about the problem it's time to jump ship ASAP. But I've seen people stubbornly defending an alert on >60% CPU usage of their 1 CPU allocated kubernetes pods where there was no impact in p99.9 latency (which was measured and was the actual metric that mattered as agreed with the rest of the business and internal customers of the service). Or alerting on each single pod restart. That is self inflicted pain.
neilv 2y ago
I want to armchair quarterback this: > I suffered through a particularly acute week of on-call pain. At one point I was in my third or fourth video call about the same long and protracted smoldering SEV and, in a moment of frustrated weakness, I made an offhand comment about just being tired of repeatedly handling the same problem. [...] With the utterance of a single sentence, I opened a rift in the relationship with my manager that remained until the day I left that job. It's possible that the manager dropped the ball in two ways: 1. Perma-haranguing an employee, when manager should instead have had a talk with employee about what was bothering the employee. (And split it, between the little that really needs to be said right now, while employee is already overextended, and what can happen after they recharge.) 2. The root cause of the recurring problem might be something management should've tackled already. Which is additional reason not to blame the employee, for calling out the problem in frustration.
neilv 2y ago
I recall "Soul of a New Machine" having parts that were fatiguing to read, incidentally (or intentionally) conveying the miserable mood of the characters, slogging through the project's trials.
neilv 2y ago
I had the same reaction as the author, the first time I heard the Kafka name, and then the same reaction when I heard the reasoning behind the name. (Then I wondered when would be a sufficiently good occasion to say "Kafkaesque" regarding the software, since that's not something you want to waste.)
neilv 2y ago
This article form might be cathartic for the writer, and actionable recommendations aren't the main point. But they are sprinkled throughout, and a small management slide deck could be distilled from the piece.
Artoooooor 2y ago
In my country if you are on employee agreement, you are just paid for overtime. My company never made any problems with that payment. And surprise...there were always people who chose to take the additional hours. I was often among them - the additional money/vacation was just too nice to resist.
- srhtftw 2y ago
  This sounds like a healthy arrangement. Compensate fairly for overtime and those who want it will take it.
tikhonj 2y ago
> My manager was present on the call, and my statement seemed to really set him off. I was essentially told that my feelings about the situation—perhaps the only authentic part of myself I ever expressed there—were wrong. In the days that followed I was made to feel like I was not a team player, that I was not pulling my weight, and that I was not meeting the bare minimum of what was expected of a person bearing the torch of on-call. With the utterance of a single sentence, I opened a rift in the relationship with my manager that remained until the day I left that job. So, this is just plainly bad leadership, right? Totally believable too, of course, but just really bad. Bad for the employee, but also self-defeating for the manager. It seems like this would be an awful manager reaction to anything short of a quasi-fireable offense. If that's your response to an employee to not being enthusiastic about a part of work that sucks, what are you even doing as a leader?
llsf 2y ago
PTSD. I was that guy dragging his bagpack everywhere. A drink at a bar on Saturday ? Sunday lunch with the in-laws ? with my bag pack, ready to bust the laptop in case of emergency. My two personal lows... I had to pull the laptop at my own birthday party, in a restaurant. And at a funeral (not proud on that one). So sad.
asdf6969 2y ago
I’m quitting my job with nothing lined up because our oncall is such a piece of shit. House arrest every 5 weeks with no compensation. If it wasn’t for this I would just quiet quit but there’s no way I can make it through another shift without getting fired. Fuck oncall and anyone who has such little respect they think it’s ok
linsomniac 2y ago
Counterpoint: I'm at a small business and I'm primary for 24x7 oncall. I don't even take shifts with my coworker. But, this is because I'm empowered to make out of hours (overnight, weekend, holiday) calls STOP. I get woken up by something about once or twice a quarter. When something wakes me up, the next day I start a process to ensure I don't get that same alert again: bugfixes, adjusting thresholds or time-to-critical, detecting problems and auto-remediation, determining it can be a "business hours" response. This also requires buy-in from development. Literally yesterday I had an education opportunity with one of the developers about a ticket slated to go into production that evening that would have immediately eliminated one of our leading monitoring indicators, because it would have started creating hundreds or thousands of Sentry issues an hour. "I was thinking it was more like logging, where more information is better, where with monitoring we want the fewest messages possible." Always, always, look at every pager hit and ask "what can prevent that from happening again?"
saagarjha 2y ago
(Former) iOS app developer here. I was oncall once and it was actually not that bad because every change had to go through app review, which put the lower bar on response times at days if not weeks. I hate app review but it was actually very nice that "oncall" really just meant "check the Slack channel in the morning" because there was no point in doing anything faster.
Jean-Papoulos 2y ago
TLDR; Author doesn't like being on-call when there's no concrete effort being made to solve the issue that results in calls.
notacoward 2y ago
I was on as many as three on-call rotations for a few years. One had only two people for a while, so I was on every other week. The two things I most remember are: * Arranging my whole life around on-call requirements. Bringing my laptop and backpack every time I went out. Designing new running routes that would use every street in a neighborhood and keep me close to home so I could respond within a 15-minute window. And yeah, the drinking thing. It pervaded my life in many ways I hadn't expected. * Time zones and geography. These were always problematic, but especially during on-call. Often I'd narrow a problem down to a particular component that I didn't know well, so I'd try to contact the sub-team responsible for it, but nobody would respond. Then I'd try to turn the right knobs myself, and as often as not get yelled at for it in the morning. No, my afternoon, because my coworkers were three hours behind and late-commuters to boot. Of course they'd never hesitate to schedule meetings or ping me for trivial things well after my dinner. I had taken the job, initially working on a project for which I was already a maintainer, because I wanted to avoid becoming an "architecture astronaut" by getting closer to operational reality. Indeed, I did learn a lot about how my own code behaved in real life. I don't have a problem with on-call requirements in and of themselves, but the way people and organizations handle the details is kind of <vomit emoji>.
kayo_20211030 2y ago
OP is right, but apart from that, this is one of the best written pieces I've read in an age. Agree, disagree, but it's so well written it's mesmerizing.
kaycebasques 2y ago
Highly recommend listening to the 70s country gem that the title is an homage to: Take This Job And Shove It [1] by (the aptly named) Johnny Paycheck [1] https://youtu.be/gj2iGAifSNI https://youtu.be/gj2iGAifSNI
ramseytyson17 2y ago
[dead]
fewdaysto2025 2y ago
[dead]

191 points by mirawelner 2y ago

" insure against every possible thing that could ever go wrong, they would have to build a second studio on a separate part of the city’s electric grid, with redundant copies of all the equipment and broadcast content, along with a full crew of understudies ready to take over at a moment’s notice." WTH?? I guess this person has never heard of backup generators? Every broadcast TV station has them.

deleted 2y ago
[deleted]
beepboopboop 2y ago
That just covers electricity. They seem to be implying coverage of a multi-failure scenario.
cfraenkel 2y ago
To begin with, airplanes do fall out of they sky, sometimes right on your backup generator. (speaking from military experience where yes, there is an entire backup studio waiting to take over, just in case. Or rather, the 'studio' is geographically dispersed, with 100% redundancy, which is another way of saying the same thing.) But more importantly, *this* is what you noticed from that article?
- dylan604 2y ago
  No, but this was the point where the "let's make something up" got to be too much. I'm not talking about low budget UHF channels, but TV stations I've been in and around all have multiple studios. If the switcher in Studio-A goes down, the signals can be routed to the control room for Studio-B. Also, Alex the know-it-all is such a forced thing that is just ridiculous and eye roll inducing. Anybody that is a jack of that many trades is a master of none of them. The entire forced analogy just got to be too much and I lost interest before a point was ever made.
deleted 2y ago
[deleted]
Macha 2y ago
Also, I don't think broadcast TV news is quite as reliable as the post makes it out to be. Like this exchange happens all the time: "and now we're going to our on site reporter, Onda Premises" <45s silence> "Oh, it appears we've lost them for now, we'll cycle back later. In our next story...."
- dylan604 2y ago
  Going to a remote is not the same thing as backup/redundancy within the studio. The broadcast was never interrupted. The latency between remotes can be mindnumbing, and with inexperienced reporter/anchor stepping over each other unable to sit through the delay it's pain inducing for the viewer. But that's an unrelated tangent

boznz 2y ago

What they don't tell you about working for yourself is the fact you can be effectively on-call 24x7 every day. I am currently supporting four wineries that are processing thousands of tonnes of receivals 24x7. It happens for two months of the year and I am expected to be available from 06:00 to 22:00 during that time, there is no phoning in sick or having a lazy day, I work alone and only have one reputation. I don't want to be that contractor forever known for destroying a clients business. You can only do this for so long though, when two or three problems come in simultaneously it can cause issues as you drop something halfway through when something more important comes in. I once executed an SQL update query without a where clause under this kind of pressure, and ended up working until the next morning to recover, only to start again at 6AM. I have even had land-line calls at 2AM to bypass my mobile restrictions. The rewards are great, but don't let anyone tell you it is always easy. My current system is 16 years old now and I know all the ins and outs so it has been pretty easy to keep on top of things the last several years, however I am glad the replacement system is nearly written and it will be somebody else problem in 2026.

netruk44 2y ago
There's a big difference, though. In your case, you're the only employee of your business. And if you're not there the business will literally go under. And you also get directly rewarded for being there. I would guess that being 'on call' in this manner is possibly less draining on a person's soul (depending on how well they tolerate the risk of owning a business). Contrast that with being 'on call' for your megacorporation, who isn't giving you anything extra for your on call time because they 'already pay you enough'. And where the only negative consequence for the company if you fail to immediately respond within 15 minutes is that some executive in the company is kept waiting longer than 15 minutes, or some ads aren't being shown for 15 minutes. But if you aren't there, your boss is going to get a phone call and that's definitely not going to look good on you. And there's no bonus for fixing the problem, that was already your job in the first place. Sucks that you had to do it outside of scheduled hours, oh well. I'm with the author of this article. Take your on-call rotation and shove it (if you're a large corporation). I'm fortunate enough to be able to take a firm stance on this point, and do so happily.

flerchin 2y ago

Jeez I guess what we do is industry standard best practice, and it sucks.

darioush 2y ago
deferring to best practice instead of best judgement is a major plague of the software industry these days. best practices usually come from giant companies with tens of thousands of engineers like google (who doesn't seem to be keeping up with competition btw) and amazon (which is notorious for burning out people). what science or evidence drives the best practices?
- cratermoon 2y ago
  > what science or evidence drives the best practices? management "science"

yodsanklai 2y ago

Excellent article. I can relate to a lot of it. The sad part is that we can't even control the quality of the systems we're oncall for. We're pushed by management for new features, not for robustness of the tools. Also some systems have no clear ownership, so nobody has an incentive to fix them. It'll be next oncall's business. Oncall is really the worst part of my job. I can stand long hours but this is something else.

mortar 2y ago
I feel for you, I’ve also suffered through this a lot over the years, and am finally at the stage of career and wisdom to start pushing back on the quality that I can’t control and ensuring that others are equally as accountable for their mess. For one particular occasion , once we took blame out of the equation (at least within the engineering team) and started doing Post Incident Reports, the incentives finally became clear for the business as we were able to compile a list of recurrent issues during every issue, calculate a financial loss and present it for inspection each and every time they either began a witch hunt for downtime or refused to allocate time to backlog. Small wins.
cratermoon 2y ago
One of the sidebars mentions that: "The production system in question is almost certainly a schizophrenic box of compromises brought about through poor decision-making, unaddressed technical debt, design-by-committee, and impossible timelines and budgets. This is not a system that any single rational human being on the team would’ve chosen to build if permitted to do so alone. Trying to assert ownership over an environment like that is just begging to get your shit rocked."
- sheepscreek 2y ago
  That’s basically every company and every system ever. Things are always in a state of flux, constantly being worked on. People come and go, priorities change and technologies evolve.
  - cratermoon 2y ago

Kwpolska 2y ago

That seems like a very long-winded way to say you hate on-call, which is a completely normal thing to do. That said, is on-call effectively mandatory or very popular in the US startup world? Because here, in the European established company world, I can’t really recall seeing a job posting with on-call listed.

spongebobstoes 2y ago
on-call is ubiquitous in the US tech industry. I've never had a job without it.
Carrok 2y ago
I've been on-call (sometimes on a rotation, sometimes always) for the past 10+ years. So, yes. It's common.
qwerpy 2y ago
Curious how this is the case. Do European companies not provide 24/7 services? Or staff a "follow the sun" model so no one has to answer pages outside of working hours? Do Europeans write better code so they don't need on-call? I've worked at a few FAANGs in the US, and every single one after 2010 had an on-call rotation.
- Macha 2y ago
  I think it's mostly down to three factors: 1. Uncompensated oncall is legally tricky in many EU companies, so a lot of midsize companies look at the cost of paying for oncall, or sometimes just the time of administering paying for oncall, and decide they can do without. High frequency oncall is also often restricted (e.g. more than 1 week in 6 is not legal here) 2. A lot of the smaller companies are europeans selling to europeans, and are much more used to a business culture of availability during office hours. Especially there's a bigger share of like b2b back office stuff in europe compared to like, restaurant POS systems. 3. Larger companies do seem more into follow the sun. A lot of the big tech in europe are subsidiaries of US companies, so if they're in Europe it means they've already opened one remote location, and therefore are more likely to have another (California, Europe, India is a super common arrangement)

slt2021 2y ago

being oncall forces the quality of software to improve. if you want fewer incidents: ensure better QA, monitoring, smaller rollouts usually developers start becoming more conservative after they do few oncall shifts and suddenly prioritize important reliability improvements, instead of shiny new features nobody will use

geoffpado 2y ago
Being on-call forces the *desire* for the quality of software to improve. Shitty management can and will override that. We don't have time for QA or to waste an engineer adding monitoring, we gotta ship ship ship.
darioush 2y ago
this doesn't always work. many things can go wrong in distributed systems and you cannot test for all of them. also you have no control of your dependencies like when AWS networking degrades or a 3rd party API provider changes their APIs without letting you know.
- toast0 2y ago
  I think if you have a reasonable environment, where on-call feeds back to development (which is what OP is suggesting, more or less), you will absolutely get woken up for networking problems, because there's not really an alternative. Maybe some thresholding to allow for minor problems without alerting, but you know. If it's a big enough problem, someone has to fix it, and it doesn't matter if it's your problem or your dependency's problem, it breaks your service so it's your problem. If it happens a lot, you look for another network to run on. For 3rd party APIs, if they're not critical, you start to develop kill switches. So yeah, someone has to wake up and handle it, but all they have to do is set the kill switch and go to sleep. Personally, I did dev and on-call for SMS/Voice verification codes. Most of the time, that's in the nasty corner of it's super critical to the application (users can't use the product if they can't get a verification code) and it depends on 3rd parties that have three nines on a good year. In my case, I got tired of dealing with the disruption and developed automated routing that could manage most providers taking an outage without needing me to take action. Results could be better if a human took notice and action, but it was good enough most of the time, and partial outages were much easier for the automation to handle it. Even if there's no way to do something like that, at least automation can take care of 3rd party API is failing hard, so mostly return errors quickly without trying the API and only let a small fraction of requests go through to sample if the service came back online. That can keep your servers from getting overwhelmed, as well as drive the alert that helps you wake up, yell at the vendor, and decide if you can go back to sleep while the system takes care of itself. When on-call is disconnected from development, that's when it gets really miserable. If you can swing a shift-work/follow the sun operator job, that's certainly better than on-call where incidents are common and there's no feedback loop to reduce things. It may well be better even if there is a feedback loop, but the feedback loop in that case requires explicit communication and effort; if I'm on call for my own work, I don't have to tell me to not push shit code right before I leave for the day or go to sleep, I'll get that message from myself right away. If someone else is cleaning up after my messes and doesn't communicate the effects to me, I might never know.

Animats 2y ago

For "non-exempt" employees, that's paid "stand-by time" California.[1] Also see this case involving on-call coroners.[2] The way this works in most unionized jobs is that there's a stand-by rate paid for on-call hours, plus a minimum number of hours at full or overtime pay, usually four, when someone is called to duty. This is useful to management - if the call frequency is too high, it becomes cheaper to hire an additional person. [1] https://www.dir.ca.gov/dlse/CallBackAndStandbyTime.pdf https://www.dir.ca.gov/dlse/CallBackAndStandbyTime.pdf [2] https://casetext.com/case/berry-v-county-of-sonoma https://casetext.com/case/berry-v-county-of-sonoma

mjcarden 2y ago

This article gave me unpleasant flashbacks to the first half of 2023. I resigned from planet.com in mid 2023 due to the stress caused by being on-call every second week. It took me six months to get my head into a healthy state again. Now I have a much better job, better paid and no possibility of on-call, ever.

purplejacket 2y ago

Here's an idea: Compensate any on-call work received during off hours at 10X the normal hourly rate. E.g., if my salary is $150K per year, then my hourly pay rate is about $75 per hour, so compensate my on call work at a rate of $750 per hour. Thus if I get a call at 10pm, log in to my laptop and work for 30 minutes to resolve the issue to a satisfactory level, then I pocket $375. That puts a financial incentive on companies to structure their on call protocols so that only the most important calls are handled. And I can envision variations on this theme. Different sorts of on-call disasters could offer bids for how much they're worth to fix based on some automated rubrick, and anyone on the ENG team could pick these up on a first-come, first-serve basis. Or various combinations of the above for a guaranteed backup person. But the companies should offer enough incentive to make it worthwhile. And this is in the companies' own best interest. To maintain a workforce that can think clearly during the normal work, to have a good reputation in the industry, to get good reviews on Glassdoor, etc.

mortar 2y ago
Good suggestion and I can see the benefit for honest people, but unfortunately it’s as equally a system for financial abuse for others - sometimes enough to prevent people fixing things during their regular hours just to benefit at other times. A good counter balance to this might be to offer even more compensation for no incidents, or otherwise well handled incidents that go on to squash types of that incident now and into the future.
- alwa 2y ago
  If I’m rewarded for no incidents, doesn’t that mean I’m punished for there being incidents?
  - mortar 2y ago
    Not necessarily, but agree that bad managers or organisations have a tedency to do this. I guess the ultimate goal is to keep everyone happy right? Everyone has different ideals, you can probably assume the business wants everything to work by spending as little as possible, employees want to be paid as much as possible while enjoying their job. Striking that balance is always a challenge.

dadkins 2y ago

I just want to point out that the answer is shift work. Here's an example of an SRE job at a national lab: https://lbl.referrals.selectminds.com/jobs/site-reliability-engineer-7041 https://lbl.referrals.selectminds.com/jobs/site-reliability-... "Work 5 shifts per week to monitor the NERSC HPC Facility, which includes 2 - 3 OWL (midnight - 8am) shifts. Some days may be onsite, some may be offsite. The schedule will be determined by staffing needs." 40 hours per week, full salary, full disclosure about the night shifts, but none of this 24x7 wake up in the middle of the night on top of your regular job bullshit that the tech industry insists on.

lolinder 2y ago
Shift work sucks more than on call for me. I like having a flexible schedule that I can work whenever I want, and I'll happily take the remote risk that I'll get paged during my one week per quarter on call rotation in order to guarantee that no one expects me to be on during set hours the rest of the year. I think the problem isn't on call itself, it's that a lot of companies suck at on call. If your rotation is every 3 weeks and you get woken up in the night at least once per rotation, then yeah, that's awful. But the problem you have in that case is that stuff is always on fire and you don't have enough people on the rotation, not with on-call as a concept.
- hx8 2y ago
  I'd take a night-shift job over being on call ontop of a 9-5. I think the real answer here is to have team members on multiple continents. Have some team members NA/EU/Asia then it's always reasonable hours for someone to deal with the production problem. High priority issues can be worked on around the clock without anyone working overtime.
  - fsckboy 2y ago
    I know plenty of friends (mostly medical) who've had "on call" shifts on top of their 9 to 5, but in those cases it was pretty exceptional for them to actually receive a call, and the "interruptions" would be a small number of hours.

smackeyacky 2y ago

That article made me shudder with echos of having what we used to call “beeper madness” back in the 1990s. After a while of being on a roster of on call weeks, anything that beeped would make you reach for that pager on your belt. As a kid the first few weeks were kind of exciting as it felt like you had been elevated to a new level of responsibility. Once that wore off it was obvious what a cage it was. I don’t miss pagers.

hinkley 2y ago
I refused a contract to hire that was talking pager duty. I saw how they freaked out about things outside the team’s control during business hours. The first time someone called me after 11 pm I was going to get myself fired talking to them.

chris_wot 2y ago

There is another option. The on-call person just does a deliberately piss-poor job of resolving the problem. I mean, they resolve it but they make sure it takes a hour longer than necessary. What are they going to do, fire you? If they make life hard for you, then get another job. The shoddier your work outside of your normal hours, the better. You can have quality, speed and cheapness, but you can only pick two.

sitzkrieg 2y ago
at a prev us tech job, a few years in they made all engineer have oncall, with no compensation added whatsoever, and on call for other teams code in india. i got pages a few times and didnt exactly rush to ack. not slow but didnt rush. the rotations increased more and more (mostly from other US employees quitting) till i quit. scummy company
nijave 2y ago
I think there's also some middle ground where you don't go out of your way to carry a laptop but you do best effort while maintaining a normal outside-work life.

denkmoon 2y ago

I was on call in a 4 man startup for a 1 week rotation for about 9 months, 6 years ago. I still have an anxiety reaction when my phone rings. Can very much relate to the author's thoughts about PTSD.

nhumrich 2y ago

The difference with dev oncall vs doctor on call is that it is self inflicted. Why are you getting paged? Because you built the system. Either your system isn't resilient enough, or you have noisy alerts. Both are problems you should be motivated to fix. I have been on call 24/7/52 in SRE roles most of my career. It has either sucked hard, or not at all. And the time it sucked the most was because every single practice was bad. And now, I build better things because of if. Paying me more for on call wouldn't have changed how much it sucked. It wouldn't have made any material impact on my actual quality of life. But it would have done two things: 1) made me feel like I can't complain 2) give me less motivation to fix it Paying for on call doesn't seem like a win. I want happy employees, not disgruntled but silent ones.

croshan 2y ago
This works if you're on call for your systems. In many situations (ranging from small startups to big tech), you're also on-call for the systems of sister teams. Not that there aren't other ways to fix that. But fixing the erroring service isn't practical in all cases.
cempaka 2y ago
We have integrations with dozens of external vendor APIs for which it's essentially impossible to disambiguate ahead of time whether any given error might be on their end or ours.
eestrada 2y ago
I think paid oncall could work if oncall is voluntary. The more oncall sucks, the less likely team members are to volunteer because the pay isn't worth it, then the company/team needs to pay more as an incentive to get people to volunteer for oncall. Eventually the price is so high that it becomes cheaper to just build the system correctly and stop shoehorning features in with no regard for stability. If oncall burden is light, then everyone volunteers because it is an easy way to make a bit of money. However, it is a huge systemic change to move towards a voluntary model. Not sure how feasible this really is.
- 2y ago

lopatin 2y ago

The OP needs to write with some more focus. Most of this reads like a very long rant by someone who was woken up too many times recently. > We need to talk about Kafka No we don't, that entire section was irrelevant.

lazide 2y ago
Well yeah, do you think people who are well rested and non stressed have oncall rotations they want to shove up someone’s nether regions?

nullorempty 2y ago

Yeah, I can relate to people saying they nearly got PTSD, I sure did get it. Paging apps use seriously offensive alarm sounds. I hated every sound they had in the options. It made me instantly sick. Fuck that!

linsomniac 2y ago
The right answer is, in my experience, something that starts soft and intermittent, and then gradually ramps up in loudness and annoyance. In bed, it's enough to have a harpsichord gently playing, but if I'm watching TV or out at a restaurant I might need it to eventually escalate to a klaxon to get my attention. My first experience with this was Groovy Blue: https://youtu.be/WCbyp_UuyoM?si=s0ZpfcXZKw7dP17i https://youtu.be/WCbyp_UuyoM?si=s0ZpfcXZKw7dP17i
smcameron 2y ago
The (in)offensiveness of the sound has little to do with it. For quite a few years after I left my job as an SRE at google, this gentle little tune called "morning flower"[1] which I used as my pager alarm sound would give me a little jolt of adrenaline. Even now, it's a bit startling, but it's old enough now that at least I do not hear it in the wild anymore. [1] https://www.youtube.com/watch?v=4hr8lhO3k5o https://www.youtube.com/watch?v=4hr8lhO3k5o

tootie 2y ago

There's a great option for small companies that aren't amenable to on-call that is underappreciated. Hire an MSP. There are companies whose entire business model is having a geo distributed team with a stack of automated monitoring and run books for multiple clients. You train them, pay a set fee and never ask anyone on your team to be on call.

srhtftw 2y ago

This is an excellent article. I've commented before¹ why "on-call" as described is bad because it conflates roles and responsibilities and robs developers of resources, but this goes quite a bit further and explains why it's a bad practice that eventually leads to burnout. With the exception of companies like PagerDuty, the sooner this practice is ended the better off we'll be. ¹ https://news.ycombinator.com/item?id=43400898 https://news.ycombinator.com/item?id=43400898

staplung 2y ago

One of the biggest problems with on-call rotations is that you're actually incentivized to do it poorly. Every minute you spend doing on-call work is time that you can't spend on the things you've actually been assigned to do. You're never going to put your on-call work into your performance reviews; doing so might actively work against you. "I see that you spent time tuning alerts and updating the runbook. That's time you didn't spend on the actual tasks that were assigned to you." If it's better to spend the least amount of time doing on-call work then the logical conclusion is that it's best to snooze as many alerts as possible until they either go away on their own or roll over past your rotation. Fixing the underlying problem might be worthwhile if it's something that you can fairly easily fix but if the on-call rotation is more than 2 people, the underlying problem is mathematically unlikely to be of your making and is it really a good idea to make a habit of fixing other people's broken code? What's crazy is that I've never seen anyone with on call duties acting in this worst case bad faith manner. Companies basically abuse the work ethic of their employees because it's the cheapest possible way to check that box.

xigency 2y ago
It's just more hazing so the underclass of engineers don't realize they actually build everything big tech is selling and more.
chronid 2y ago
> Every minute you spend doing on-call work is time that you can't spend on the things you've actually been assigned to do. In my experience at least if you're oncall during a sprint you would have less work assigned to you than otherwise (2 week sprint and 1 week you are oncall? 50% allocation) as the expectation is that week you will spend responding to alerts, or investigating issues, or even improving alerting and dashboard and fixing bugs. If this does not happen, devs don't push for it and management is completely blind to it you have an organization issue. If leadership does not care about the problem it's time to jump ship ASAP. But I've seen people stubbornly defending an alert on >60% CPU usage of their 1 CPU allocated kubernetes pods where there was no impact in p99.9 latency (which was measured and was the actual metric that mattered as agreed with the rest of the business and internal customers of the service). Or alerting on each single pod restart. That is self inflicted pain.

neilv 2y ago

I want to armchair quarterback this: > I suffered through a particularly acute week of on-call pain. At one point I was in my third or fourth video call about the same long and protracted smoldering SEV and, in a moment of frustrated weakness, I made an offhand comment about just being tired of repeatedly handling the same problem. [...] With the utterance of a single sentence, I opened a rift in the relationship with my manager that remained until the day I left that job. It's possible that the manager dropped the ball in two ways: 1. Perma-haranguing an employee, when manager should instead have had a talk with employee about what was bothering the employee. (And split it, between the little that really needs to be said right now, while employee is already overextended, and what can happen after they recharge.) 2. The root cause of the recurring problem might be something management should've tackled already. Which is additional reason not to blame the employee, for calling out the problem in frustration.

neilv 2y ago

I recall "Soul of a New Machine" having parts that were fatiguing to read, incidentally (or intentionally) conveying the miserable mood of the characters, slogging through the project's trials.

neilv 2y ago

I had the same reaction as the author, the first time I heard the Kafka name, and then the same reaction when I heard the reasoning behind the name. (Then I wondered when would be a sufficiently good occasion to say "Kafkaesque" regarding the software, since that's not something you want to waste.)

neilv 2y ago

This article form might be cathartic for the writer, and actionable recommendations aren't the main point. But they are sprinkled throughout, and a small management slide deck could be distilled from the piece.

Artoooooor 2y ago

In my country if you are on employee agreement, you are just paid for overtime. My company never made any problems with that payment. And surprise...there were always people who chose to take the additional hours. I was often among them - the additional money/vacation was just too nice to resist.

srhtftw 2y ago
This sounds like a healthy arrangement. Compensate fairly for overtime and those who want it will take it.

tikhonj 2y ago

> My manager was present on the call, and my statement seemed to really set him off. I was essentially told that my feelings about the situation—perhaps the only authentic part of myself I ever expressed there—were wrong. In the days that followed I was made to feel like I was not a team player, that I was not pulling my weight, and that I was not meeting the bare minimum of what was expected of a person bearing the torch of on-call. With the utterance of a single sentence, I opened a rift in the relationship with my manager that remained until the day I left that job. So, this is just plainly bad leadership, right? Totally believable too, of course, but just really bad. Bad for the employee, but also self-defeating for the manager. It seems like this would be an awful manager reaction to anything short of a quasi-fireable offense. If that's your response to an employee to not being enthusiastic about a part of work that sucks, what are you even doing as a leader?

llsf 2y ago

PTSD. I was that guy dragging his bagpack everywhere. A drink at a bar on Saturday ? Sunday lunch with the in-laws ? with my bag pack, ready to bust the laptop in case of emergency. My two personal lows... I had to pull the laptop at my own birthday party, in a restaurant. And at a funeral (not proud on that one). So sad.

asdf6969 2y ago

I’m quitting my job with nothing lined up because our oncall is such a piece of shit. House arrest every 5 weeks with no compensation. If it wasn’t for this I would just quiet quit but there’s no way I can make it through another shift without getting fired. Fuck oncall and anyone who has such little respect they think it’s ok

linsomniac 2y ago

Counterpoint: I'm at a small business and I'm primary for 24x7 oncall. I don't even take shifts with my coworker. But, this is because I'm empowered to make out of hours (overnight, weekend, holiday) calls STOP. I get woken up by something about once or twice a quarter. When something wakes me up, the next day I start a process to ensure I don't get that same alert again: bugfixes, adjusting thresholds or time-to-critical, detecting problems and auto-remediation, determining it can be a "business hours" response. This also requires buy-in from development. Literally yesterday I had an education opportunity with one of the developers about a ticket slated to go into production that evening that would have immediately eliminated one of our leading monitoring indicators, because it would have started creating hundreds or thousands of Sentry issues an hour. "I was thinking it was more like logging, where more information is better, where with monitoring we want the fewest messages possible." Always, always, look at every pager hit and ask "what can prevent that from happening again?"

saagarjha 2y ago

(Former) iOS app developer here. I was oncall once and it was actually not that bad because every change had to go through app review, which put the lower bar on response times at days if not weeks. I hate app review but it was actually very nice that "oncall" really just meant "check the Slack channel in the morning" because there was no point in doing anything faster.

Jean-Papoulos 2y ago

TLDR; Author doesn't like being on-call when there's no concrete effort being made to solve the issue that results in calls.

notacoward 2y ago

I was on as many as three on-call rotations for a few years. One had only two people for a while, so I was on every other week. The two things I most remember are: * Arranging my whole life around on-call requirements. Bringing my laptop and backpack every time I went out. Designing new running routes that would use every street in a neighborhood and keep me close to home so I could respond within a 15-minute window. And yeah, the drinking thing. It pervaded my life in many ways I hadn't expected. * Time zones and geography. These were always problematic, but especially during on-call. Often I'd narrow a problem down to a particular component that I didn't know well, so I'd try to contact the sub-team responsible for it, but nobody would respond. Then I'd try to turn the right knobs myself, and as often as not get yelled at for it in the morning. No, my afternoon, because my coworkers were three hours behind and late-commuters to boot. Of course they'd never hesitate to schedule meetings or ping me for trivial things well after my dinner. I had taken the job, initially working on a project for which I was already a maintainer, because I wanted to avoid becoming an "architecture astronaut" by getting closer to operational reality. Indeed, I did learn a lot about how my own code behaved in real life. I don't have a problem with on-call requirements in and of themselves, but the way people and organizations handle the details is kind of <vomit emoji>.

kayo_20211030 2y ago

OP is right, but apart from that, this is one of the best written pieces I've read in an age. Agree, disagree, but it's so well written it's mesmerizing.

kaycebasques 2y ago

Highly recommend listening to the 70s country gem that the title is an homage to: Take This Job And Shove It [1] by (the aptly named) Johnny Paycheck [1] https://youtu.be/gj2iGAifSNI https://youtu.be/gj2iGAifSNI

ramseytyson17 2y ago

[dead]

fewdaysto2025 2y ago

[dead]