15 ms·

Oncall shift should be Tuesday to Tuesday

190 points by RyeCombinator 2y ago

ludwigvan 2y ago
Mon-Mon has the advantage that it is a single week and you are done. Tue-Tue means it bleeds into the second week.
julianeon 2y ago
He makes a sound argument.
applecrazy 2y ago
> Most places take after hours paging pretty seriously. LOL i wish
- tail_exchange 2y ago
  My team has a meeting to hand off the on-call to the next person, and we discuss all pages we got during the week. Primarily two things: whether the page was for a good reason or not (good: our on-call person had an something actionable to fix. bad: non-actionable pages, pages because someone else's system was broken, false alarms, etc), and also whether there is something we can do so we never get paged for this again. I find it very effective at reducing pages.
- kdazzle 2y ago
  Lol yeah. My old team had oncall pages in the middle of the night pretty often where nothing was actually the matter. My manager was only nominally on call. In the handoff meetings every week he was basically just like “that sucks”.
N8works 2y ago
I never understood why companies didn't simply leverage 24x7 internet MSPs. They are able to staff 24x7 by spreading the cost over multiple customers and working through the process of making your application manageable by a 3rd party is super beneficial. Most of these companies will also do performance monitoring and analysis as well. They see issues and optimization opportunities across multiple applications and know more than a single team who's only built one.
- 0_____0 2y ago
  Are you speaking from personal experience having worked with one? What was the feedback between application management back to engineering like?
- danpalmer 2y ago
  That works well for generic IT systems and running the desktop/laptop fleets, but doesn’t work at all for running the software a company builds. We typically split our teams, so we have ~16 split across two time zones so that our shifts are just 12 hours during the day. It works well, but it is expensive, so we support a lot of services (or a small number of very high priority services) as a result.
- RaoulP 2y ago
  I hadn't heard of Managed Service Providers before, but you make a good case for them. I'm finding surprisingly little discussion on HN regarding the costs/benefits of MSPs. Or rather, under which conditions (such as company size) they make sense. Any big players or companies you would recommend?
- sgarland 2y ago
  If an MSP can effectively manage your company’s product, then your problems are simple enough to have automated detection and recovery.
losteric 2y ago
I have occasionally convinced teams to adopt both oncall and sprint cycles aligned with Tuesday [1] - the dev teams all loved it. Management was a harder sell, but by and large were happier with the extra days to communicate results/get metrics before their own Friday deadlines. [1] also Wednesday/Thursdays. Wednesdays were my favorite in good working environments, it felt like running a successful marathon, but it was more prone to falling apart due to short-term thinking.
- superfrank 2y ago
  I'm curious why Tues to Tues for sprints was a hard sell to management?
  - dijksterhuis 2y ago
    likewise, be interested to hear more about that situation
- andrewaylett 2y ago
  We have our sprints start on Tuesdays, and our in-hours on-call also runs Tuesday to Monday. Out of hours on-call starts at 5pm Monday.
hnbear 2y ago
I’ve always been partial for Friday night through Friday night. You start off over the weekend, when you have energy and can survive the two days alone. Ideally no Friday releases so the transition is calm, but as the writer says the batches might fail. You spend the week fixing whatever breaks. You’re cleanly off the Monday to Monday sprint, just doing on-call/ops. You finish Friday evening and immediately get Friday night and the weekend to recover when you need it most.
- taberiand 2y ago
  That was exactly my reasoning too when I set up our on call roster as Friday to Friday, though for us Saturday is the busiest day in terms of customer activity, so it was a no-brainer.
- superfrank 2y ago
  This post was discussed somewhere else and I saw someone say that their work does Firday noon to Friday noon and their work gives the outgoing on call the rest of Friday off. I feel like that's even better because 1) it recognizes the hard work that the outgoing on call put in 2) it give the incoming on call a few hours to get up to speed while they still have the support of the other engineers on the team.
- wging 2y ago
  Maybe I'm taking you too literally, but I wouldn't want to have a handoff sync-up (or any meeting, really) on a Friday night, nor push that earlier so significant things can happen between sync-up and the actual shift in responsibility from person to person. Friday-to-Friday does sound good. One thing I really liked in a previous job was a split daytime-vs-nighttime rotation. It was well worth a little annoyance to set up in our tools. One week you'd be the 'daytime' oncall for business hours (something like 9-5 Mon-Fri, though we might have tweaked those hours a bit; it might have been 10-6 or something). The next you'd be on call for the complementary time (5-9, weekends). You were on call for the same total amount of time, just smeared over two different weeks. It ended up being less of a burden to optimize your schedule for a reasonable response time, but operational work still got done. And in practice awareness of operational issues was not too hard to maintain between the two members of the split. (I think the best thing, if you can swing it, is probably a follow-the-sun rotation where there are three teams distributed 8 hours apart around the globe, and they trade off 8-hour workday shifts. But a lot of uncommon things probably have to be true of your organization for that idea to even be on the radar.)
ljoshua 2y ago
My team does Wednesday to Wednesday for many of the same reasons mentioned in the article, and it works great. We switch at 11am and hold a hand-off meeting at that time, and invite the whole team. Hand-off meetings with the whole team work really well (in my opinion!) when you have a relatively small team--we have 9 FT teammates. Often someone else may have been delegated the page or bug that arose and can discuss how they handled it, or someone who wasn't involved may have insight for how to handle a situation better the next time. Since we're all going to be on rotation at least once during a quarter, it's great to know what happened in case a similar page pops up later. Finally, we also fill out a running Doc before/during the meeting with links to the pages/bugs, along with short descriptions of how they were handled. This forms a great living memory of how to deal with incidents, and is also often the birthplace of new playbooks for handling new types of incidents.
- l8nite 2y ago
  Same here. Except we do a two week rotation, and it aligns with our sprints. The active on-call engineer doesn’t have any assigned sprint work and focuses their effort on fixing bugs or cleaning up the backlog when they’re not actively triaging an incident.
thuanao 2y ago
Is anyone getting compensated for being on-call? If you are paged and work outside of business hours, do you receive additional compensation?
- tdeck 2y ago
  At Google we used to get paid an oncall bonus which was calculated at something like 1/3 your prorated salary for the non-working hours you were oncall (IIRC), up to some limit per quarter. For my team a week of oncall per quarter would max it out and net you a few thousand dollars bonus.
  - thaumasiotes 2y ago
    > up to some limit per quarter. For my team a week of oncall per quarter would max it out That reminds me of Amazon's abysmally bad employee discount, which was "10% off anything on the site, up to $100 / year".
  - kevinventullo 2y ago
    Google still does this. Roughly speaking, hitting the limit in a quarter means you have <= 5 people on the rotation.
- themenomen 2y ago
  Yes. And not only for responding to a page, but also for being stand by outside working hours.
- losteric 2y ago
  On my teams, if someone got paged off-hours they would just work less the day after the event. imo it should just be part of the regular salary/work expectations, incentivizing keeping oncall low
  - rk06 2y ago
    No, it should be compensated, so Management prioritises fixing issues, instead of adding new bugs
AaronM 2y ago
We are on-call for 48hrs at a time, about once every 12 days or so, one day as backup, and one as primary. It's nice because it doesn't interrupt your week too much. The downside being that complex issues might require extra work while not on-call
Charon77 2y ago
My company does this
nikolay 2y ago
Ours starts at 5 PM on Tuesday and I think it's great.
siliconc0w 2y ago
We do daily shifts with a follow the sun rotation, makes it easier to handle persistent commitments and ensures a bad week doesn't all land on the same person.
- tgma 2y ago
  Daily might be okay for more ops/SRE types, but it is a hell for a primarily dev team. Can't focus on building shit.
  - crossroadsguy 2y ago
    In some cases it might help. Because then it becomes “natural” - on-call thing. It’s not something someone dreads as in “god, that week is coming”. Also, it spreads the fuck-ups and peaceful times better.
    - tgma 2y ago
      Increasing hand-offs by 7x is sub-optimal and interferes with folks wanting to take continuous vacation time. Again, I can see for ops teams that could be true, but very much disagree that on-call should be a "natural" thing for dev teams in the first place. It can be a necessary evil that should be minimized (the personality of people who like firefighting and quiet development are quite distinct and there are people who actually like the former.) On the latter point, I think that benefit is very much a mirage. If there's a flaw in the system causing a "bad week," it actually might be easier for the first person who gets the hang of it to deal with it than to try handing off and teaching the next one in the rotation.
- crossroadsguy 2y ago
  This! Whenever someone talk about on-call this aspect of that rotation gets swept under the carpet. Whenever I interview I always ask whether they have on-call system (they must if there are servers and apps involved) and if they do whether they have follow the sun. Most don’t even like the question. For them such questions are red flags or the candidate is not “motivated enough”. Rarely some even have follow the sun policy. They might have one in their HQ, true for a lot of US/EU firms, but their offices in a developing country like India - it’s always something on the lines of “oh, engineers here take full ownership; they are the owners”. Also, I have seen — 2-3 days rotation with follow the sun is best, week long or longer being worst. Then there are companies where it could be forever on-call with no follow the sun - e.g. Amazon, Uber (in India at least). That’s another world altogether.
polack 2y ago
We do Thursday to Thursday and then you get Friday off after completed on-call. Being on-call gives you no extra pay by itself, but if you get paged off hours and need to work you get paid 150 to 200% of your normal hourly wage depending on what time of day you need to work. Best on-call I’ve had.
- mrweasel 2y ago
  That's good the hear. We're currently redesigning our on-call and plan to make it Thursday to Thursday, and then Friday off.
- einichi 2y ago
  No pay for being on call by itself is still poor, particularly when it comes to swapping rotations between team members to provide flexibility amongst each other. You’re making yourself available 24/7. That has a non trivial lifestyle impact which I’ve always thought deserves more than is typically rewarded.
  - zgeor 2y ago
    Not to mention that there is incentive to keep having oncall pages, because that's how you get paid. Or not participate at all. On the other hand, with a flat payment, there is a big incentive to prevent issues and not have(reduce) ooh incidents, and participate in the rota.
  - sokoloff 2y ago
    As long as the on-call coverage is as specified at the time of hiring, this is just a difference in form of payment. If I receive 100 total units of compensation, I'd way rather get 100 units of base pay (and 0 on-call pay) than 90 units of base pay and 10 units of specific on-call pay. (What if the company eliminates on-call? What if I get injured and my insurance only covers base pay? Severance is usually based only on base pay; I would not be paid on-call while I'm on PTO or other paid leave, annual raise percentages typically apply to base pay, etc...)
    - 2y ago
bckr 2y ago
> But websites need to be up 24/7, cron jobs need to run on the weekend and backend servers need to be up to support both Tech entrepreneurs should give more weight to choosing markets that don’t require this
- vineyardmike 2y ago
  It used to be somewhat common for websites/services to go down for a few minutes every so often for maintenance/migrations/etc. Tech entrepreneurs should give no weight to this. The market seems to support engineers doing on-call rotations, and a service that can’t tolerate any downtime is (theoretically) a service that is worth a lot to a lot of people- which is perfect for monetizing. Tech entrepreneurs should stop giving excessive “nines” of availability. Even 99% is probably enough for most customers to never notice, and significantly easier to engineer than 99.999….
  - portaouflop 2y ago
    The issue is less giving out excessive SLAs - it’s more that even a tiny ass startup thinks they need high availability and four nines and scale to billions - when in reality almost no one actually needs it. But those are cool engineering problem so we’d rather work on them than on building a business.
  - bckr 2y ago
    > and a service that can’t tolerate any downtime is (theoretically) a service that is worth a lot to a lot of people- which is perfect for monetizing. The connection makes sense but one must not think in this order. One must think “people will pay for this” and then consider “does this need to be highly available?” If you have more than one road to choose from, and one of them doesn’t require high availability, then give that one some bonus points for that.
  - SoftTalker 2y ago
    It's still common. Nobody really cares if a site is offline for a few minutes. You try again later (or not, but so what). Heck, nobody cares if they are offline for half the day, it gets fixed and at the end of it it's just a post-mortem for the nerds to read and a shrug and life goes on for everyone else. People vastly overestimate the importance of anything that is on the public internet. None of it is life-critical (if it is, it certainly should not depend on an internet connection or a web server being up).
thebigspacefuck 2y ago
On a past team I set up on-call to be: - Mon/Tue - Wed/Thu - Fri - Sat/Sun Original reason for this schedule was that on-call was paid by days per quarter in a tiered system so this guaranteed that all members got the 5% on-call for 10 days/quarter rather than one person hitting 9 days and dropping to 3%, but I stand by this as a better on-call rotation. The number of people does need to be not wholly divisible so the days rotate so if you run into this you can combine Fri into Sat/Sun or break Sat/Sun apart. It’s a bit complex to set up but the mental impact of on-call is greatly reduced and if you need a week for vacation you can much more easily find someone to cover your shift for a couple days in a nearby week rather than ending up with 2 weeks back to back 6 weeks from now. And if you pull a weekend you get the week off rather than losing your weekend to on-call and going into a work week still on-call.
- bigiain 2y ago
  Any company that makes it an employee's responsibility to find "someone to cover" their on call time while they're on vacation is a company worth quitting. I'm pretty sure that'd be illegal here in .au On call coverage while an employee is on vacation is a management problem, not an employee problem.
  - glitchcrab 2y ago
    Could not agree more, any company I've worked at with an on-call rotation has always ensured that staff are not scheduled when they have holiday booked. The only time an employee needed to find their own cover is if something unexpected came up during their on-call period and they needed a few hours out (like an emergency visit to the doctor with a child etc). At my current job we have an automated scheduler which uses our gcal to ensure that it never schedules if people have an AFK entry. It also schedules fairly based on how long since the person was last on-call, not putting them on on a weekend if they were on last weekend etc (we do 24hr shifts).
    - jeduardo 2y ago
      Are you using an in-house scheduler or is this a feature of a particular tool?
zeroonetwothree 2y ago
In my 20ish years I’ve done every possible day for oncall schedules. I would say each have pros/cons but overall I found it to be a minor difference. Mon-Mon is nice because it’s a logical time to start something fresh at the start of the week. Tuesday is good for the reasons in the post, Wednesday is similar. Thursday is nice because after you’re done you can relax on Friday. Friday-Friday is less common but can be nice because you get the satisfaction of being done on the last day of the week.
- natebc 2y ago
  Where i work we do Friday 8AM -> Friday 8AM. We changed to that from a Monday->Monday a few years ago. Feedback has been postive. Coming off of on-call on Monday morning was just a major bummer.
pmayrgundter 2y ago
This is a strong positive imhe: "- Step 1: handling it - Step 2: making sure it doesn’t happen again So when a major issue happens over the weekend only Step 1 happens during the weekend. Step 2 involves following up with other teams, creating new alarms and updating the runbook. And all that usually happen during the week. The oncall is going to spend at minimum their Monday doing that so it’s better if the schedule reflects that."
sanzs 2y ago
[flagged]
Simon_ORourke 2y ago
100% agree, especially when you have to deal with distributed teams in the UK with all their "bank holidays" which all seem to land on Mondays.
throwaway240403 2y ago
Each person on my team has a day of the week they own, and then we have a rotation for weekends, and negotiate holiday/pto trades. I guess it really only maps correctly for a 5 person team. We previously had a week long rotation, and some folks were initially skeptical of the idea to change, saying they were worried they'd feel like they were "oncall all the time". But, they agreed to try it for a month. That was a bit over a year ago now, and no complaints. I think it ends up being a lower stress configuration, because it just becomes part of your normal expected work-week routine, and generally isn't as mentally draining. It does make end of year PTO/holiday time a bit more complex to work out, but so far my team has been okay with that tradeoff.
- hnlmorg 2y ago
  how do you work around bank holidays? Which in some countries are almost always on the same weekday? Does the person who has Mondays just deal with not having a longer weekend like everyone else? What about the person who has Friday? Do they never go out on a Friday evening? Sounds a nice idea in theory but not all week days are equally inconvenient.
coding123 2y ago
It makes it slightly harder to go on vacation, I think a lot of people won't like that.
canergl 2y ago
wrong, oncall shifts should not even exist
- guessmyname 2y ago
  I agree with the sentiment, but on-call support is an unavoidable necessity given the critical nature of many systems that underpin modern society. When we talk about on-call, we’re not referring to systems like Netflix streaming a major fight for 65 million users, but rather essential infrastructure like healthcare systems, nuclear power plants, military operations, financial markets, and the vast array of SCADA (Supervisory Control and Data Acquisition) systems that monitor and control industrial processes. These systems are crucial to our safety, economy, and everyday lives, and downtime or failure is not an option. Before Apple, I worked at Microsoft in the Azure team, where I logged over 2,016 hours of on-call support each year. This involved six 24/7 on-call rotations, each lasting two weeks, with responsibilities alternating between primary and secondary support. While there were certainly tough moments and challenging issues during those rotations, they also provided valuable learning experiences and helped me develop problem-solving skills under pressure. On-call support is a necessary evil.
  - harimau777 2y ago
    Then the people who have to work on call should be compensated accordingly. In my experience they are not.
  - smitelli 2y ago
    I’ve always felt it should be split among a geographically distributed team where support hours follow the sun. It really sucks to be awake at 3am, alone, groggy and unsupported and responsible for saving the world. If the company/product isn’t large enough to be distributed, is it really important that it have a 10 minute time-to-acknowledge?
  - sed_zeppelin 2y ago
    I was once paged over thirty times in a span of 24 hours while working for a website that, in the grand scheme of things, could unilaterally improve life in the United States by shutting itself down.
elmerfuddwalz 2y ago
[dead]
MisterBastahrd 2y ago
Here's my on call schedule: never, and don't ask. It's my responsibility to do my job when I am scheduled, and it's management's responsibility to staff properly. If we can't agree, then we can't have a business relationship.
- sed_zeppelin 2y ago
  I wish more devs had the gumption to refuse it. I can understand on-call hours if you're a literal firefighter or paramedic who saves lives. I understand that, as a building superintendent, every once in a long while you have to run out and fix a burst pipe before property is destroyed. I don't understand why some of these tech companies have on-call responsibilities like there was some hazard to life or property. They need five nines of availability to make sure they don't lose one cent of potential ad revenue? Good luck with that, I guess, but I'll be over here actually sleeping through the night.
sgarland 2y ago
I’ve also done Wednesday to Wednesday, though Tuesday seems better mentally if only because there is one less day after the new week starts. What is much better, though, is splitting the week into a 4/3 or 5/2 split, with a primary and backup on-call. Primary takes the weekdays, then switches with Backup for the weekend. You’re still sharp and aware of any current issues should the need arise, but the odds of a weekend page are (hopefully) lower, so you can relax a bit. This of course requires enough people to have a reasonable rotation; 6 at a minimum, but 8 is better.
nvarsj 2y ago
We started a split shift for a really busy oncall and it works out really well. It's Th->Tu, Tu->Th. So basically weekend+2 working days vs 3 working days. Expectation is you are 100% oncall during the working day, so it works out pretty well between weekend vs non-weekend shifts. I much prefer the shorter shifts to a full week. A full week on-call usually means delaying important project work, etc. for a full week.
harimau777 2y ago
On call should be reasonably compensated. IMHO all other discussions of on call should take place after that is resolved. Instead, developers are expected to work unreasonable hours and are then fired when they start to burn out.
- metaltyphoon 2y ago
  Seriously don’t understand why devs want to do free work.
  - s1artibartfast 2y ago
    If that is what you think they are want, you are seriously confused.
    - metaltyphoon 2y ago
      Please enlighten me as to how on call is not free work when companies do not pay for this outside of your regular salary .
      - s1artibartfast 2y ago
        1) Salary can include tasks like this. 2) You said they want to. They dont. If you offer same pay for a job with and without it, exactly nobody would choose the job with extra on call duties. The obvious part you are missing is that people do it because they are paid to do it, and they like money.
        metaltyphoon 2y ago
        > Salary can include tasks like this So lets say that it just magically happens that when YOU are on call, stuff breaks all the time but when its your coworkers it doesn’t. You are all paid the same, does it seem fair to you now? Unless it’s written in paper where a salaried worker will be getting X extra per hour you are just working for free. The definition of a salaried worker in the US is having 40hrs of total work time averaged throughout a year.
dmazin 2y ago
A few months ago we switched from Tuesday-through-Monday to Monday-through-Sunday and on call stress decreased. After a weekend of on call, it sucks to have yet another day of on call on Monday. This overpowered all other reasons (most of them listed in the blog post) for us.
looperhacks 2y ago
I'm not sure if my team is a crazy exception, but here's how our on-call works: We're usually on from Monday-Monday (with exceptions if we say, don't have time on Wednesday or something), but every team decides the time on their own. During work-hours, every team member is responsible for responding to alerts (but usually, only the on-call engineers will carry their company-provided phones and are the first to respond). Outside work-hours? Most alarms (if they happen) are due to bad alarm configurations. Because nothing ever happens. There was one alert this month, and it was because a randomly generated ID contained the string "ERROR" and was logged due to a warning. I know that my company isn't the "biggest" (only a few hundred requests per minute) and traffic amount is mostly correlated to usual business hours in my country, so there's just not much happening at night (but never zero traffic). Still, I'm always surprised that other companies seem to have really stressful on-call shifts, because the most annoying part to me is having to carry my laptop if I leave my home for more than 20 minutes.
sed_zeppelin 2y ago
I just want to jump in as a minority voice here. In case anybody is reading the other comments and feeling... alienated. I refuse to accept on-call duties, full stop. If a job posting expects it, I don't apply. If a hiring manager says they have it, I do not accept the offer. If management starts talking about maybe implementing it, I protest. If it becomes enacted, I resign. There is absolutely no situation in which I will ever participate in another on-call shift. I've been there, I've done it, now that chapter of my life is closed. Find some younger kid, pay them better than you paid me for the miserable intrusion on their life. I'm done. Just wanted to be the voice who says what, hopefully, some of the more seasoned and battle-scarred readers here are thinking.
- convolvatron 2y ago
  on call is like hiring civil and structural engineers to build you a bridge over a canyon, and then when they show up to do a site inspection you just push them in. eventually maybe you'll be able to cross.
  - ddingus 2y ago
    I love your comment on this. Perfection. Once, while traveling in an RV for some work related marketing thing, the discussion turned to the lack of fuel economy... The RV might perform better if the engine powered the RV by blowing fuel right out the tail pipe. Horrible efficiency, terrible for the planet, and, and all the negatives packed right into a quick expression. Your comment is on point. Solid and I just felt like sharing my appreciation for the morbid fun it contains. Nice work. Worth a healthy chuckle. Thanks.
  - stackskipton 2y ago
    As SRE, strongly disagree. On Call is like hiring civil and structural engineers then holding them responsible when their poor bridge collapses under the weight of all the traffic. Sometimes, yes, Devs get called out for stuff outside their control like infrastructure failing. However, at my job, we just had two devs that quit over on call and guess what, their service was one of worst offenders in "Opps, we pushed bug to production."
bradleyjg 2y ago
I’ve had to schedule on calls for a team and this would make it much harder to make a schedule. Every week’s vacation now means two weeks that person can’t be scheduled for on-call.
corytheboyd 2y ago
I’ve been in so many meetings endlessly searching for the Best Day for on-call shifts to start/end, and feel I have heard it all at this point, every day of the week seems to have pros and cons. Current team just does Monday, because that’s when the week usually starts, and I really don’t want to talk about it anymore lol. If it conflicts with one of the abysmally few US holidays, adjust accordingly, you’re a bunch of smart clever adults, you’ll figure it out.
dpcx 2y ago
We used to do two week on-call rotations (to follow along with sprint/release cadence). At the beginning of '23, one of our devs asked to be full-time on-call, both because they had sleep issues and so would regularly be awake at 2 am, but also because they didn't like that the other team members wouldn't follow the documented processes. They don't get paid extra, but they seem to be very happy with the setup.
mnahkies 2y ago
The point about holidays resonates with me. Our setup you get paid extra for being on-call for a public holiday, but given we do Mon-Mon shifts in practice that means two people can't take advantage of a long weekend and only one of them gets extra compensation for it. Different people deal with being on-call differently but personally I don't do what I normally would when on-call, whether that's long motorbike rides or hiking etc because it's not practical to guarantee cell coverage and also the threat of a page ruins the experience. A "day off" whilst on-call isn't equal to a day off
seusscat 2y ago
I hate on-call shifts, but if they must exist, I like the way my team handles them. We have split day and night shifts. 7-18 day shift, and 18-07 night shift. All non-work hours compensated with standby at 10% of hourly pay. Any pages outside of work hours earn you an additional 150% in base pay. Each page guarantees a minimum of 3 hours of pay even if you spent only 5 mins on it. And since in my country, you must gave at least 11 hours between shifts, if you get paged at night, you get PTO for the next 11 hours on top.
- BHSPitMonkey 2y ago
  I like the idea of added compensation based on hours covered as it incentivizes the business to avoid very small rotation sizes, but paying extra per page seems like a perverse incentive favoring instability.
  - lolinder 2y ago
    It depends on who has the largest amount of influence on how noisy the on call is. If engineers have blanket control to define what is important enough to get interrupted and to prioritize fixing frequent offenders, then sure, it's a perverse incentive. If, on the other hand, engineering doesn't have very much control over the roadmap and/or isn't allowed to make their own judgment calls about what really matters for pages, then the arrangement that OP describes makes a ton of sense—it gets gets pages onto the budget as a separate line item, which is a good way to get the people who are really in charge on board with investing in permanent fixes.
    - seusscat 2y ago
      It also becomes a good deterrent against useless requests. You get pinged on Slack at 10pm? Just ask them to file a ticket with a page-worthy severity. When its not nearly as important as that, even external managers will hesitate to do that since they need to explain if the ticket was worth 150% base pay for 3 hours plus the extra PTO next day. Significantly reduces the number of pages.
  - seusscat 2y ago
davidjfelix 2y ago
At a previous employer we had a pair on call for each of: front end, back end, and infra. We had on-call lasting from Monday midday - Friday midday. Handing off to a "weekend on-call" from the same pool of people from Friday midday to Monday midday. Weekend on-call paid 100 per day, weekday on-call paid 50 per day. You were generally expected to take normal time "off" (but still on call) if paged off hours. Many people would still work if it was just a blip (rare). I thought this was a pretty good system and despite the cycles being shorter, we had enough engineers to fill a rotation pretty well so that at most you were on call once a month, alternating months between weekend and weekday on-call cycles. I still do not enjoy being forced into on call and wish I could opt-in. We traded weeks a lot but with smaller rotations or really finicky paging its awful. I still have a sinking feeling in my gut when I hear the work phone ringtone from somebody else's phone in public, and murphy's law definitely applies to being on call -- you always get paged the minute after your beer gets delivered at a restaurant.
bleuarff 2y ago
Here in France, we have strict laws, and on-calls MUST be paid in some form or another. When we were bought by a US company, mgmt tried to set up on-call shifts for us - we had never needed them for the 10 years prior -, until they learnt of labor laws and went "fuck it, you're on call mon-fri, 10am - 6pm". I'm forty, have a family, and no amount of money could justify that I can't shutoff my phone at night, or prevent from going on a walk on weekends because "uptime". I've never been so glad of french worker protections.
- pb7 2y ago
  [flagged]
regularfry 2y ago
> Highly scrum/agile focused people have brought up that sprints start on Monday and that starting the oncall on Tuesday makes sprint planning harder Wait, what? Don't run your sprints Monday to Monday either. That's been the eventual conclusion on all scrum teams I've been on.
phamilton4 2y ago
When did on-call become so accepted and demanded from employers? Currently I am "Release Captain" for a week: So I have to setup any releases and manage all the related tasks, do automated/manual testing of the release, release (enabling toggles and any config changes). Then Backup to secondary and primary for a week: About once or twice I am asked to help with tickets. Then for 14 days we alternate primary / secondary. Thursday to Thursday is our deal. Every ~40 days I am in one of the above. It's absolutely miserable. I have never had this much time spent doing non-development related tasks. For 4 weeks every 1.5 months I can't have a life at all. This just screams to me that we are forcing broken software/not complete software out the gate a building huge piles of technical debt that will never get the focus. I remember a time when I would start at 9am and end at 6pm every day and never heard a peep about production issues unless the support engineers couldn't figure it out. Which maybe happened twice a year. To make matters worse most things are not allowed to be touched in production with the risk of being fired for making changes. So if you want to "fix" any data or call xyz service you need high ranking approval. It's like being tortured!
- rr808 2y ago
  > When did on-call become so accepted and demanded from employers? As a 50 something year old software engineer. Its always been like this. I'm kinda shocked at how reluctant the new generation is to support the systems. Sure we'd all prefer strict 9-5 hours but most companies rely on software to stay in business and you need experts available in case things go wrong.
  - szszrk 2y ago
    If you need experts 24/7, you should have had shifts that cover that timeframe. Oncall is a source of so many ways for abuse, don't even ask me how I know. Saying that rejecting Oncall is denying support for you system is bollocks. I'm happy that younger engineers mostly laught at that concept and leave. Once of the few lessons they are teaching us (the old pricks), especially in self care and respect space.
    - rr808 2y ago
gwillz 2y ago
I work in a small team, we have ~30 active clients and other ~100 or so that are low maintenance or dormant. We don't have an on-call rotation but I desperately want one. Because if no one is on-call - then all of us are on-call. Any one of us could be called at any time if one of our larger commerce projects falls over. To me on-call is a necessary burden that means when I'm not on duty I am completely free to ignore my phone. I'll certainly feel more positive about helping outside of the 9-5. I do like to be helpful, but perhaps that'll wear off like some kind of honeymoon period, juxtaposed to my current situation. I'm always looking for more positives in such a system because I want it to work. Tuesday to Tuesday sounds great. Other comments here highlight the difference between critical fixes and patch it laters. Any other insights are welcome.
chungus 2y ago
When I just started out, I used to take everybody's on-call service because it paid well. Also had the advantage that I learned the ropes quicker. Now-a-days I'm very happy to not be awoken at 3am, even if 1 weekend incident would amount to a couple grand.
renewiltord 2y ago
I always do Wednesday to Wednesday. I think I’d rather fix something than having it hanging over my head. That said, I prefer systems that don’t require constant attendance. But when I was young, I reveled in that. You get to have a lot of fun firefighting and using each chance to make sure the fire can’t be lit next time. Thoroughly enjoyable and eventually the fires reduce. I still think of each error as a possibility of improvement to asymptotic zero error and I prefer working with people like that. Others prefer other systems and I think that it’s fine for each of these groups to select for members appropriately aligned.
dillydogg 2y ago
I'm not IT, but the hospital I work for has the resident physicans on call Tuesday to Tuesday and the attending physicians Monday to Monday. I have found it to be a good system, with the weekends being covered by people who have a good idea of their patients.
71bw 2y ago
Oncall for barely any calls. If I have to do anything I just add that as overtime, simple as. Was a bit odd to have to carry my work phone everywhere for the first two weeks or so but now it's just standard procedure

190 points by RyeCombinator 2y ago

ludwigvan 2y ago

Mon-Mon has the advantage that it is a single week and you are done. Tue-Tue means it bleeds into the second week.

julianeon 2y ago

He makes a sound argument.

applecrazy 2y ago

> Most places take after hours paging pretty seriously. LOL i wish

tail_exchange 2y ago
My team has a meeting to hand off the on-call to the next person, and we discuss all pages we got during the week. Primarily two things: whether the page was for a good reason or not (good: our on-call person had an something actionable to fix. bad: non-actionable pages, pages because someone else's system was broken, false alarms, etc), and also whether there is something we can do so we never get paged for this again. I find it very effective at reducing pages.
kdazzle 2y ago
Lol yeah. My old team had oncall pages in the middle of the night pretty often where nothing was actually the matter. My manager was only nominally on call. In the handoff meetings every week he was basically just like “that sucks”.

N8works 2y ago

I never understood why companies didn't simply leverage 24x7 internet MSPs. They are able to staff 24x7 by spreading the cost over multiple customers and working through the process of making your application manageable by a 3rd party is super beneficial. Most of these companies will also do performance monitoring and analysis as well. They see issues and optimization opportunities across multiple applications and know more than a single team who's only built one.

0_____0 2y ago
Are you speaking from personal experience having worked with one? What was the feedback between application management back to engineering like?
danpalmer 2y ago
That works well for generic IT systems and running the desktop/laptop fleets, but doesn’t work at all for running the software a company builds. We typically split our teams, so we have ~16 split across two time zones so that our shifts are just 12 hours during the day. It works well, but it is expensive, so we support a lot of services (or a small number of very high priority services) as a result.
RaoulP 2y ago
I hadn't heard of Managed Service Providers before, but you make a good case for them. I'm finding surprisingly little discussion on HN regarding the costs/benefits of MSPs. Or rather, under which conditions (such as company size) they make sense. Any big players or companies you would recommend?
sgarland 2y ago
If an MSP can effectively manage your company’s product, then your problems are simple enough to have automated detection and recovery.

losteric 2y ago

I have occasionally convinced teams to adopt both oncall and sprint cycles aligned with Tuesday [1] - the dev teams all loved it. Management was a harder sell, but by and large were happier with the extra days to communicate results/get metrics before their own Friday deadlines. [1] also Wednesday/Thursdays. Wednesdays were my favorite in good working environments, it felt like running a successful marathon, but it was more prone to falling apart due to short-term thinking.

superfrank 2y ago
I'm curious why Tues to Tues for sprints was a hard sell to management?
- dijksterhuis 2y ago
  likewise, be interested to hear more about that situation
andrewaylett 2y ago
We have our sprints start on Tuesdays, and our in-hours on-call also runs Tuesday to Monday. Out of hours on-call starts at 5pm Monday.

hnbear 2y ago

I’ve always been partial for Friday night through Friday night. You start off over the weekend, when you have energy and can survive the two days alone. Ideally no Friday releases so the transition is calm, but as the writer says the batches might fail. You spend the week fixing whatever breaks. You’re cleanly off the Monday to Monday sprint, just doing on-call/ops. You finish Friday evening and immediately get Friday night and the weekend to recover when you need it most.

taberiand 2y ago
That was exactly my reasoning too when I set up our on call roster as Friday to Friday, though for us Saturday is the busiest day in terms of customer activity, so it was a no-brainer.
superfrank 2y ago
This post was discussed somewhere else and I saw someone say that their work does Firday noon to Friday noon and their work gives the outgoing on call the rest of Friday off. I feel like that's even better because 1) it recognizes the hard work that the outgoing on call put in 2) it give the incoming on call a few hours to get up to speed while they still have the support of the other engineers on the team.
wging 2y ago
Maybe I'm taking you too literally, but I wouldn't want to have a handoff sync-up (or any meeting, really) on a Friday night, nor push that earlier so significant things can happen between sync-up and the actual shift in responsibility from person to person. Friday-to-Friday does sound good. One thing I really liked in a previous job was a split daytime-vs-nighttime rotation. It was well worth a little annoyance to set up in our tools. One week you'd be the 'daytime' oncall for business hours (something like 9-5 Mon-Fri, though we might have tweaked those hours a bit; it might have been 10-6 or something). The next you'd be on call for the complementary time (5-9, weekends). You were on call for the same total amount of time, just smeared over two different weeks. It ended up being less of a burden to optimize your schedule for a reasonable response time, but operational work still got done. And in practice awareness of operational issues was not too hard to maintain between the two members of the split. (I think the best thing, if you can swing it, is probably a follow-the-sun rotation where there are three teams distributed 8 hours apart around the globe, and they trade off 8-hour workday shifts. But a lot of uncommon things probably have to be true of your organization for that idea to even be on the radar.)

ljoshua 2y ago

My team does Wednesday to Wednesday for many of the same reasons mentioned in the article, and it works great. We switch at 11am and hold a hand-off meeting at that time, and invite the whole team. Hand-off meetings with the whole team work really well (in my opinion!) when you have a relatively small team--we have 9 FT teammates. Often someone else may have been delegated the page or bug that arose and can discuss how they handled it, or someone who wasn't involved may have insight for how to handle a situation better the next time. Since we're all going to be on rotation at least once during a quarter, it's great to know what happened in case a similar page pops up later. Finally, we also fill out a running Doc before/during the meeting with links to the pages/bugs, along with short descriptions of how they were handled. This forms a great living memory of how to deal with incidents, and is also often the birthplace of new playbooks for handling new types of incidents.

l8nite 2y ago
Same here. Except we do a two week rotation, and it aligns with our sprints. The active on-call engineer doesn’t have any assigned sprint work and focuses their effort on fixing bugs or cleaning up the backlog when they’re not actively triaging an incident.

thuanao 2y ago

Is anyone getting compensated for being on-call? If you are paged and work outside of business hours, do you receive additional compensation?

tdeck 2y ago
At Google we used to get paid an oncall bonus which was calculated at something like 1/3 your prorated salary for the non-working hours you were oncall (IIRC), up to some limit per quarter. For my team a week of oncall per quarter would max it out and net you a few thousand dollars bonus.
- thaumasiotes 2y ago
  > up to some limit per quarter. For my team a week of oncall per quarter would max it out That reminds me of Amazon's abysmally bad employee discount, which was "10% off anything on the site, up to $100 / year".
- kevinventullo 2y ago
  Google still does this. Roughly speaking, hitting the limit in a quarter means you have <= 5 people on the rotation.
themenomen 2y ago
Yes. And not only for responding to a page, but also for being stand by outside working hours.
losteric 2y ago
On my teams, if someone got paged off-hours they would just work less the day after the event. imo it should just be part of the regular salary/work expectations, incentivizing keeping oncall low
- rk06 2y ago
  No, it should be compensated, so Management prioritises fixing issues, instead of adding new bugs

AaronM 2y ago

We are on-call for 48hrs at a time, about once every 12 days or so, one day as backup, and one as primary. It's nice because it doesn't interrupt your week too much. The downside being that complex issues might require extra work while not on-call

Charon77 2y ago

My company does this

nikolay 2y ago

Ours starts at 5 PM on Tuesday and I think it's great.

siliconc0w 2y ago

We do daily shifts with a follow the sun rotation, makes it easier to handle persistent commitments and ensures a bad week doesn't all land on the same person.

tgma 2y ago
Daily might be okay for more ops/SRE types, but it is a hell for a primarily dev team. Can't focus on building shit.
- crossroadsguy 2y ago
  In some cases it might help. Because then it becomes “natural” - on-call thing. It’s not something someone dreads as in “god, that week is coming”. Also, it spreads the fuck-ups and peaceful times better.
  - tgma 2y ago
    Increasing hand-offs by 7x is sub-optimal and interferes with folks wanting to take continuous vacation time. Again, I can see for ops teams that could be true, but very much disagree that on-call should be a "natural" thing for dev teams in the first place. It can be a necessary evil that should be minimized (the personality of people who like firefighting and quiet development are quite distinct and there are people who actually like the former.) On the latter point, I think that benefit is very much a mirage. If there's a flaw in the system causing a "bad week," it actually might be easier for the first person who gets the hang of it to deal with it than to try handing off and teaching the next one in the rotation.
crossroadsguy 2y ago
This! Whenever someone talk about on-call this aspect of that rotation gets swept under the carpet. Whenever I interview I always ask whether they have on-call system (they must if there are servers and apps involved) and if they do whether they have follow the sun. Most don’t even like the question. For them such questions are red flags or the candidate is not “motivated enough”. Rarely some even have follow the sun policy. They might have one in their HQ, true for a lot of US/EU firms, but their offices in a developing country like India - it’s always something on the lines of “oh, engineers here take full ownership; they are the owners”. Also, I have seen — 2-3 days rotation with follow the sun is best, week long or longer being worst. Then there are companies where it could be forever on-call with no follow the sun - e.g. Amazon, Uber (in India at least). That’s another world altogether.

polack 2y ago

We do Thursday to Thursday and then you get Friday off after completed on-call. Being on-call gives you no extra pay by itself, but if you get paged off hours and need to work you get paid 150 to 200% of your normal hourly wage depending on what time of day you need to work. Best on-call I’ve had.

mrweasel 2y ago
That's good the hear. We're currently redesigning our on-call and plan to make it Thursday to Thursday, and then Friday off.
einichi 2y ago
No pay for being on call by itself is still poor, particularly when it comes to swapping rotations between team members to provide flexibility amongst each other. You’re making yourself available 24/7. That has a non trivial lifestyle impact which I’ve always thought deserves more than is typically rewarded.
- zgeor 2y ago
  Not to mention that there is incentive to keep having oncall pages, because that's how you get paid. Or not participate at all. On the other hand, with a flat payment, there is a big incentive to prevent issues and not have(reduce) ooh incidents, and participate in the rota.
- sokoloff 2y ago
  As long as the on-call coverage is as specified at the time of hiring, this is just a difference in form of payment. If I receive 100 total units of compensation, I'd way rather get 100 units of base pay (and 0 on-call pay) than 90 units of base pay and 10 units of specific on-call pay. (What if the company eliminates on-call? What if I get injured and my insurance only covers base pay? Severance is usually based only on base pay; I would not be paid on-call while I'm on PTO or other paid leave, annual raise percentages typically apply to base pay, etc...)
  - 2y ago

bckr 2y ago

> But websites need to be up 24/7, cron jobs need to run on the weekend and backend servers need to be up to support both Tech entrepreneurs should give more weight to choosing markets that don’t require this

vineyardmike 2y ago
It used to be somewhat common for websites/services to go down for a few minutes every so often for maintenance/migrations/etc. Tech entrepreneurs should give no weight to this. The market seems to support engineers doing on-call rotations, and a service that can’t tolerate any downtime is (theoretically) a service that is worth a lot to a lot of people- which is perfect for monetizing. Tech entrepreneurs should stop giving excessive “nines” of availability. Even 99% is probably enough for most customers to never notice, and significantly easier to engineer than 99.999….
- portaouflop 2y ago
  The issue is less giving out excessive SLAs - it’s more that even a tiny ass startup thinks they need high availability and four nines and scale to billions - when in reality almost no one actually needs it. But those are cool engineering problem so we’d rather work on them than on building a business.
- bckr 2y ago
  > and a service that can’t tolerate any downtime is (theoretically) a service that is worth a lot to a lot of people- which is perfect for monetizing. The connection makes sense but one must not think in this order. One must think “people will pay for this” and then consider “does this need to be highly available?” If you have more than one road to choose from, and one of them doesn’t require high availability, then give that one some bonus points for that.
- SoftTalker 2y ago
  It's still common. Nobody really cares if a site is offline for a few minutes. You try again later (or not, but so what). Heck, nobody cares if they are offline for half the day, it gets fixed and at the end of it it's just a post-mortem for the nerds to read and a shrug and life goes on for everyone else. People vastly overestimate the importance of anything that is on the public internet. None of it is life-critical (if it is, it certainly should not depend on an internet connection or a web server being up).

thebigspacefuck 2y ago

On a past team I set up on-call to be: - Mon/Tue - Wed/Thu - Fri - Sat/Sun Original reason for this schedule was that on-call was paid by days per quarter in a tiered system so this guaranteed that all members got the 5% on-call for 10 days/quarter rather than one person hitting 9 days and dropping to 3%, but I stand by this as a better on-call rotation. The number of people does need to be not wholly divisible so the days rotate so if you run into this you can combine Fri into Sat/Sun or break Sat/Sun apart. It’s a bit complex to set up but the mental impact of on-call is greatly reduced and if you need a week for vacation you can much more easily find someone to cover your shift for a couple days in a nearby week rather than ending up with 2 weeks back to back 6 weeks from now. And if you pull a weekend you get the week off rather than losing your weekend to on-call and going into a work week still on-call.

bigiain 2y ago
Any company that makes it an employee's responsibility to find "someone to cover" their on call time while they're on vacation is a company worth quitting. I'm pretty sure that'd be illegal here in .au On call coverage while an employee is on vacation is a management problem, not an employee problem.
- glitchcrab 2y ago
  Could not agree more, any company I've worked at with an on-call rotation has always ensured that staff are not scheduled when they have holiday booked. The only time an employee needed to find their own cover is if something unexpected came up during their on-call period and they needed a few hours out (like an emergency visit to the doctor with a child etc). At my current job we have an automated scheduler which uses our gcal to ensure that it never schedules if people have an AFK entry. It also schedules fairly based on how long since the person was last on-call, not putting them on on a weekend if they were on last weekend etc (we do 24hr shifts).
  - jeduardo 2y ago
    Are you using an in-house scheduler or is this a feature of a particular tool?

zeroonetwothree 2y ago

In my 20ish years I’ve done every possible day for oncall schedules. I would say each have pros/cons but overall I found it to be a minor difference. Mon-Mon is nice because it’s a logical time to start something fresh at the start of the week. Tuesday is good for the reasons in the post, Wednesday is similar. Thursday is nice because after you’re done you can relax on Friday. Friday-Friday is less common but can be nice because you get the satisfaction of being done on the last day of the week.

natebc 2y ago
Where i work we do Friday 8AM -> Friday 8AM. We changed to that from a Monday->Monday a few years ago. Feedback has been postive. Coming off of on-call on Monday morning was just a major bummer.

pmayrgundter 2y ago

This is a strong positive imhe: "- Step 1: handling it - Step 2: making sure it doesn’t happen again So when a major issue happens over the weekend only Step 1 happens during the weekend. Step 2 involves following up with other teams, creating new alarms and updating the runbook. And all that usually happen during the week. The oncall is going to spend at minimum their Monday doing that so it’s better if the schedule reflects that."

sanzs 2y ago

[flagged]

Simon_ORourke 2y ago

100% agree, especially when you have to deal with distributed teams in the UK with all their "bank holidays" which all seem to land on Mondays.

throwaway240403 2y ago

Each person on my team has a day of the week they own, and then we have a rotation for weekends, and negotiate holiday/pto trades. I guess it really only maps correctly for a 5 person team. We previously had a week long rotation, and some folks were initially skeptical of the idea to change, saying they were worried they'd feel like they were "oncall all the time". But, they agreed to try it for a month. That was a bit over a year ago now, and no complaints. I think it ends up being a lower stress configuration, because it just becomes part of your normal expected work-week routine, and generally isn't as mentally draining. It does make end of year PTO/holiday time a bit more complex to work out, but so far my team has been okay with that tradeoff.

hnlmorg 2y ago
how do you work around bank holidays? Which in some countries are almost always on the same weekday? Does the person who has Mondays just deal with not having a longer weekend like everyone else? What about the person who has Friday? Do they never go out on a Friday evening? Sounds a nice idea in theory but not all week days are equally inconvenient.

coding123 2y ago

It makes it slightly harder to go on vacation, I think a lot of people won't like that.

canergl 2y ago

wrong, oncall shifts should not even exist

guessmyname 2y ago
I agree with the sentiment, but on-call support is an unavoidable necessity given the critical nature of many systems that underpin modern society. When we talk about on-call, we’re not referring to systems like Netflix streaming a major fight for 65 million users, but rather essential infrastructure like healthcare systems, nuclear power plants, military operations, financial markets, and the vast array of SCADA (Supervisory Control and Data Acquisition) systems that monitor and control industrial processes. These systems are crucial to our safety, economy, and everyday lives, and downtime or failure is not an option. Before Apple, I worked at Microsoft in the Azure team, where I logged over 2,016 hours of on-call support each year. This involved six 24/7 on-call rotations, each lasting two weeks, with responsibilities alternating between primary and secondary support. While there were certainly tough moments and challenging issues during those rotations, they also provided valuable learning experiences and helped me develop problem-solving skills under pressure. On-call support is a necessary evil.
- harimau777 2y ago
  Then the people who have to work on call should be compensated accordingly. In my experience they are not.
- smitelli 2y ago
  I’ve always felt it should be split among a geographically distributed team where support hours follow the sun. It really sucks to be awake at 3am, alone, groggy and unsupported and responsible for saving the world. If the company/product isn’t large enough to be distributed, is it really important that it have a 10 minute time-to-acknowledge?
- sed_zeppelin 2y ago
  I was once paged over thirty times in a span of 24 hours while working for a website that, in the grand scheme of things, could unilaterally improve life in the United States by shutting itself down.

elmerfuddwalz 2y ago

[dead]

MisterBastahrd 2y ago

Here's my on call schedule: never, and don't ask. It's my responsibility to do my job when I am scheduled, and it's management's responsibility to staff properly. If we can't agree, then we can't have a business relationship.

sed_zeppelin 2y ago
I wish more devs had the gumption to refuse it. I can understand on-call hours if you're a literal firefighter or paramedic who saves lives. I understand that, as a building superintendent, every once in a long while you have to run out and fix a burst pipe before property is destroyed. I don't understand why some of these tech companies have on-call responsibilities like there was some hazard to life or property. They need five nines of availability to make sure they don't lose one cent of potential ad revenue? Good luck with that, I guess, but I'll be over here actually sleeping through the night.

sgarland 2y ago

I’ve also done Wednesday to Wednesday, though Tuesday seems better mentally if only because there is one less day after the new week starts. What is much better, though, is splitting the week into a 4/3 or 5/2 split, with a primary and backup on-call. Primary takes the weekdays, then switches with Backup for the weekend. You’re still sharp and aware of any current issues should the need arise, but the odds of a weekend page are (hopefully) lower, so you can relax a bit. This of course requires enough people to have a reasonable rotation; 6 at a minimum, but 8 is better.

nvarsj 2y ago

We started a split shift for a really busy oncall and it works out really well. It's Th->Tu, Tu->Th. So basically weekend+2 working days vs 3 working days. Expectation is you are 100% oncall during the working day, so it works out pretty well between weekend vs non-weekend shifts. I much prefer the shorter shifts to a full week. A full week on-call usually means delaying important project work, etc. for a full week.

harimau777 2y ago

On call should be reasonably compensated. IMHO all other discussions of on call should take place after that is resolved. Instead, developers are expected to work unreasonable hours and are then fired when they start to burn out.

metaltyphoon 2y ago
Seriously don’t understand why devs want to do free work.
- s1artibartfast 2y ago
  If that is what you think they are want, you are seriously confused.
  - metaltyphoon 2y ago
    Please enlighten me as to how on call is not free work when companies do not pay for this outside of your regular salary .
    - s1artibartfast 2y ago
      1) Salary can include tasks like this. 2) You said they want to. They dont. If you offer same pay for a job with and without it, exactly nobody would choose the job with extra on call duties. The obvious part you are missing is that people do it because they are paid to do it, and they like money.
      - metaltyphoon 2y ago
        > Salary can include tasks like this So lets say that it just magically happens that when YOU are on call, stuff breaks all the time but when its your coworkers it doesn’t. You are all paid the same, does it seem fair to you now? Unless it’s written in paper where a salaried worker will be getting X extra per hour you are just working for free. The definition of a salaried worker in the US is having 40hrs of total work time averaged throughout a year.

dmazin 2y ago

A few months ago we switched from Tuesday-through-Monday to Monday-through-Sunday and on call stress decreased. After a weekend of on call, it sucks to have yet another day of on call on Monday. This overpowered all other reasons (most of them listed in the blog post) for us.

looperhacks 2y ago

I'm not sure if my team is a crazy exception, but here's how our on-call works: We're usually on from Monday-Monday (with exceptions if we say, don't have time on Wednesday or something), but every team decides the time on their own. During work-hours, every team member is responsible for responding to alerts (but usually, only the on-call engineers will carry their company-provided phones and are the first to respond). Outside work-hours? Most alarms (if they happen) are due to bad alarm configurations. Because nothing ever happens. There was one alert this month, and it was because a randomly generated ID contained the string "ERROR" and was logged due to a warning. I know that my company isn't the "biggest" (only a few hundred requests per minute) and traffic amount is mostly correlated to usual business hours in my country, so there's just not much happening at night (but never zero traffic). Still, I'm always surprised that other companies seem to have really stressful on-call shifts, because the most annoying part to me is having to carry my laptop if I leave my home for more than 20 minutes.

sed_zeppelin 2y ago

I just want to jump in as a minority voice here. In case anybody is reading the other comments and feeling... alienated. I refuse to accept on-call duties, full stop. If a job posting expects it, I don't apply. If a hiring manager says they have it, I do not accept the offer. If management starts talking about maybe implementing it, I protest. If it becomes enacted, I resign. There is absolutely no situation in which I will ever participate in another on-call shift. I've been there, I've done it, now that chapter of my life is closed. Find some younger kid, pay them better than you paid me for the miserable intrusion on their life. I'm done. Just wanted to be the voice who says what, hopefully, some of the more seasoned and battle-scarred readers here are thinking.

convolvatron 2y ago
on call is like hiring civil and structural engineers to build you a bridge over a canyon, and then when they show up to do a site inspection you just push them in. eventually maybe you'll be able to cross.
- ddingus 2y ago
  I love your comment on this. Perfection. Once, while traveling in an RV for some work related marketing thing, the discussion turned to the lack of fuel economy... The RV might perform better if the engine powered the RV by blowing fuel right out the tail pipe. Horrible efficiency, terrible for the planet, and, and all the negatives packed right into a quick expression. Your comment is on point. Solid and I just felt like sharing my appreciation for the morbid fun it contains. Nice work. Worth a healthy chuckle. Thanks.
- stackskipton 2y ago
  As SRE, strongly disagree. On Call is like hiring civil and structural engineers then holding them responsible when their poor bridge collapses under the weight of all the traffic. Sometimes, yes, Devs get called out for stuff outside their control like infrastructure failing. However, at my job, we just had two devs that quit over on call and guess what, their service was one of worst offenders in "Opps, we pushed bug to production."

bradleyjg 2y ago

I’ve had to schedule on calls for a team and this would make it much harder to make a schedule. Every week’s vacation now means two weeks that person can’t be scheduled for on-call.

corytheboyd 2y ago

I’ve been in so many meetings endlessly searching for the Best Day for on-call shifts to start/end, and feel I have heard it all at this point, every day of the week seems to have pros and cons. Current team just does Monday, because that’s when the week usually starts, and I really don’t want to talk about it anymore lol. If it conflicts with one of the abysmally few US holidays, adjust accordingly, you’re a bunch of smart clever adults, you’ll figure it out.

dpcx 2y ago

We used to do two week on-call rotations (to follow along with sprint/release cadence). At the beginning of '23, one of our devs asked to be full-time on-call, both because they had sleep issues and so would regularly be awake at 2 am, but also because they didn't like that the other team members wouldn't follow the documented processes. They don't get paid extra, but they seem to be very happy with the setup.

mnahkies 2y ago

The point about holidays resonates with me. Our setup you get paid extra for being on-call for a public holiday, but given we do Mon-Mon shifts in practice that means two people can't take advantage of a long weekend and only one of them gets extra compensation for it. Different people deal with being on-call differently but personally I don't do what I normally would when on-call, whether that's long motorbike rides or hiking etc because it's not practical to guarantee cell coverage and also the threat of a page ruins the experience. A "day off" whilst on-call isn't equal to a day off

seusscat 2y ago

I hate on-call shifts, but if they must exist, I like the way my team handles them. We have split day and night shifts. 7-18 day shift, and 18-07 night shift. All non-work hours compensated with standby at 10% of hourly pay. Any pages outside of work hours earn you an additional 150% in base pay. Each page guarantees a minimum of 3 hours of pay even if you spent only 5 mins on it. And since in my country, you must gave at least 11 hours between shifts, if you get paged at night, you get PTO for the next 11 hours on top.

BHSPitMonkey 2y ago
I like the idea of added compensation based on hours covered as it incentivizes the business to avoid very small rotation sizes, but paying extra per page seems like a perverse incentive favoring instability.
- lolinder 2y ago
  It depends on who has the largest amount of influence on how noisy the on call is. If engineers have blanket control to define what is important enough to get interrupted and to prioritize fixing frequent offenders, then sure, it's a perverse incentive. If, on the other hand, engineering doesn't have very much control over the roadmap and/or isn't allowed to make their own judgment calls about what really matters for pages, then the arrangement that OP describes makes a ton of sense—it gets gets pages onto the budget as a separate line item, which is a good way to get the people who are really in charge on board with investing in permanent fixes.
  - seusscat 2y ago
    It also becomes a good deterrent against useless requests. You get pinged on Slack at 10pm? Just ask them to file a ticket with a page-worthy severity. When its not nearly as important as that, even external managers will hesitate to do that since they need to explain if the ticket was worth 150% base pay for 3 hours plus the extra PTO next day. Significantly reduces the number of pages.
- seusscat 2y ago

davidjfelix 2y ago

At a previous employer we had a pair on call for each of: front end, back end, and infra. We had on-call lasting from Monday midday - Friday midday. Handing off to a "weekend on-call" from the same pool of people from Friday midday to Monday midday. Weekend on-call paid 100 per day, weekday on-call paid 50 per day. You were generally expected to take normal time "off" (but still on call) if paged off hours. Many people would still work if it was just a blip (rare). I thought this was a pretty good system and despite the cycles being shorter, we had enough engineers to fill a rotation pretty well so that at most you were on call once a month, alternating months between weekend and weekday on-call cycles. I still do not enjoy being forced into on call and wish I could opt-in. We traded weeks a lot but with smaller rotations or really finicky paging its awful. I still have a sinking feeling in my gut when I hear the work phone ringtone from somebody else's phone in public, and murphy's law definitely applies to being on call -- you always get paged the minute after your beer gets delivered at a restaurant.

bleuarff 2y ago

Here in France, we have strict laws, and on-calls MUST be paid in some form or another. When we were bought by a US company, mgmt tried to set up on-call shifts for us - we had never needed them for the 10 years prior -, until they learnt of labor laws and went "fuck it, you're on call mon-fri, 10am - 6pm". I'm forty, have a family, and no amount of money could justify that I can't shutoff my phone at night, or prevent from going on a walk on weekends because "uptime". I've never been so glad of french worker protections.

pb7 2y ago
[flagged]

regularfry 2y ago

> Highly scrum/agile focused people have brought up that sprints start on Monday and that starting the oncall on Tuesday makes sprint planning harder Wait, what? Don't run your sprints Monday to Monday either. That's been the eventual conclusion on all scrum teams I've been on.

phamilton4 2y ago

When did on-call become so accepted and demanded from employers? Currently I am "Release Captain" for a week: So I have to setup any releases and manage all the related tasks, do automated/manual testing of the release, release (enabling toggles and any config changes). Then Backup to secondary and primary for a week: About once or twice I am asked to help with tickets. Then for 14 days we alternate primary / secondary. Thursday to Thursday is our deal. Every ~40 days I am in one of the above. It's absolutely miserable. I have never had this much time spent doing non-development related tasks. For 4 weeks every 1.5 months I can't have a life at all. This just screams to me that we are forcing broken software/not complete software out the gate a building huge piles of technical debt that will never get the focus. I remember a time when I would start at 9am and end at 6pm every day and never heard a peep about production issues unless the support engineers couldn't figure it out. Which maybe happened twice a year. To make matters worse most things are not allowed to be touched in production with the risk of being fired for making changes. So if you want to "fix" any data or call xyz service you need high ranking approval. It's like being tortured!

rr808 2y ago
> When did on-call become so accepted and demanded from employers? As a 50 something year old software engineer. Its always been like this. I'm kinda shocked at how reluctant the new generation is to support the systems. Sure we'd all prefer strict 9-5 hours but most companies rely on software to stay in business and you need experts available in case things go wrong.
- szszrk 2y ago
  If you need experts 24/7, you should have had shifts that cover that timeframe. Oncall is a source of so many ways for abuse, don't even ask me how I know. Saying that rejecting Oncall is denying support for you system is bollocks. I'm happy that younger engineers mostly laught at that concept and leave. Once of the few lessons they are teaching us (the old pricks), especially in self care and respect space.
  - rr808 2y ago

gwillz 2y ago

I work in a small team, we have ~30 active clients and other ~100 or so that are low maintenance or dormant. We don't have an on-call rotation but I desperately want one. Because if no one is on-call - then all of us are on-call. Any one of us could be called at any time if one of our larger commerce projects falls over. To me on-call is a necessary burden that means when I'm not on duty I am completely free to ignore my phone. I'll certainly feel more positive about helping outside of the 9-5. I do like to be helpful, but perhaps that'll wear off like some kind of honeymoon period, juxtaposed to my current situation. I'm always looking for more positives in such a system because I want it to work. Tuesday to Tuesday sounds great. Other comments here highlight the difference between critical fixes and patch it laters. Any other insights are welcome.

chungus 2y ago

When I just started out, I used to take everybody's on-call service because it paid well. Also had the advantage that I learned the ropes quicker. Now-a-days I'm very happy to not be awoken at 3am, even if 1 weekend incident would amount to a couple grand.

renewiltord 2y ago

I always do Wednesday to Wednesday. I think I’d rather fix something than having it hanging over my head. That said, I prefer systems that don’t require constant attendance. But when I was young, I reveled in that. You get to have a lot of fun firefighting and using each chance to make sure the fire can’t be lit next time. Thoroughly enjoyable and eventually the fires reduce. I still think of each error as a possibility of improvement to asymptotic zero error and I prefer working with people like that. Others prefer other systems and I think that it’s fine for each of these groups to select for members appropriately aligned.

dillydogg 2y ago

I'm not IT, but the hospital I work for has the resident physicans on call Tuesday to Tuesday and the attending physicians Monday to Monday. I have found it to be a good system, with the weekends being covered by people who have a good idea of their patients.

71bw 2y ago

Oncall for barely any calls. If I have to do anything I just add that as overtime, simple as. Was a bit odd to have to carry my work phone everywhere for the first two weeks or so but now it's just standard procedure