28 ms·

I Don't Want to Be on Call Anymore

230 points by kiyanwang 5y ago

draw_down 5y ago
> I believe it’s reasonable to expect to be woken up 2-3 times a year, as an engineer for that system. Reasonable that may be, but the problem is you don’t know which 2-3 nights it will happen. No thanks.
yuppie_scum 5y ago
Extremely important article. I have been a part of some hell rotations and it is not sustainable at all.
PicassoCTs 5y ago
On call is a dysfunctional company culture attacking its own workforce in a battle of attrition. Its the equivalent of rheumatic fever to a company process. If you need it, your reliability engineering is failing and the process that should mend it, is instead busy fixing symptoms. Such a company is already dieing and will do once the team "on call" is used up, burned out and gone.It will be replaced in the market by something more vital, still capable to value its workforce.
- ckdarby 5y ago
  Are you saying, every fortune 100 tech company's reliability engineers are failing to do their jobs? MAANG all have on-call individuals even with the fact they're across all timezones and have been growing steady.
  - nzmsv 5y ago
    There are different ways of handling on-call and the practices at the companies you mention are very different, as is the on-call load. There is a world of difference between someone voluntarily accepting on-call as part of their duties in exchange for higher pay or a 4 day work week, and on-call being forced on everyone.
  - supernovae 5y ago
    He's saying we're all suckers... and sometimes, it takes a long time to find out how true that really may be.
dopylitty 5y ago
This misses the key point that on-call itself is an abusive practice regardless of how well an organization practices it. If an organization thinks their systems should be available 24x7 they should staff people 24x7. There is no situation where it’s ok to contact someone to do work outside of their work hours.
- echelon 5y ago
  > There is no situation where it’s ok to contact someone to do work outside of their work hours. This lacks perspective. Imagine that your system runs banking, healthcare, telecom, etc. Of course someone needs to be on call 24/7. The best people to hold the pager are the engineers that work on the system.
  - shoo 5y ago
    that's a pretty selective quote. you skipped over: > If an organization thinks their systems should be available 24x7 they should staff people 24x7. One way to achieve 24x7 coverage without anyone being structurally on-call outside of standard office hours could be "follow the sun" support -- have offices in multiple timezones around the world and run 3 x 8 hour shifts each day of support working their local office hours to provide 24/7 coverage. Still need enough staff at each site to handle rotation for covering weekends and for people to go on holiday and get sick or quit for new jobs etc. I can understand why many companies would prefer not to pay for that if they can get away with cheaper alternatives, and why they might prefer to understaff and push the burden onto employees, but that's the company's problem, not the problem of the individual worker who trades their life by the hour in exchange for money.
    - echelon 5y ago
      If you're not a global empire with offices all over the world yet, the simpler thing is to pay your engineers well and tell them being oncall is one of the responsibilities.
      - 5y ago
ckdarby 5y ago
Author lists off group of people who should not get pager duty and I think it is unfair to the rest of the team that individuals who fall into the groups outlined are able to escape the duties and responsibilities of the job. If the job posting or interviewing was upfront about the role will have on-call then the interviewee should make the decision to not apply or accept the offer if they're unable to fulfill the responsibilities.
- barrkel 5y ago
  That attitude leads directly to ageism, sexism and other discriminatory practices, and has second order effects on society which are contrary to the long term interests of capitalism - a stable consumer base with disposable income. Fairness isn't a straightforward concept, but your conception is more naive and - I'd guess - self-serving than most. How do you feel about progressive income taxes? Under a simplistic model of fairness, flat rates seem fair, but in reality the marginal ability to pay increases, and perceived pain of paying decreases, with increased income and wealth. Now expand that concept from the monetary domain to the time domain, and consider how fair it is for people who are particularly time poor - for non-selfish reasons - to need to spend more of their time on the job. Affordances can be made for inclusiveness and they can be fair.
  - kayodelycaon 5y ago
    Fairness, as used by the parent, means “if someone else can do something, I should be able to too.” The other person’s situation is never a factor. They see someone getting things for free and they think they should have them for free too.
    - barrkel 5y ago
      Do you feel the same way about maternity and paternity leave?
- kayodelycaon 5y ago
  Many programming jobs require on-call and overtime when they shouldn’t. I wouldn’t be able to have a job if that requirement was truly valid. It’s usually not. I can’t do on-call for medical reasons, nor can I work overtime. Neither of these are considered to be an essential job function for programmers, regardless of job descriptions, under ADA. A company must justify that job requirement. They can’t. By the way, the “fairness” reason is explicitly excluded by law. It’s the entire reason ADA exists.
deleted 5y ago
[deleted]
willcipriano 5y ago
My argument is that as the tech side of the organization takes on more and more of the operation of the business, it also needs to be taking on more of the rewards. For example if the expectation is that a engineering team designs, builds and maintains the machine twenty four hours a day that is responsible for the majority (if not all) of the firms profits then the majority of the rewards must flow there. One the other hand, traditional methods for doing this have fallen by the wayside. With performance overhangs and other chicanery, what was once a lottery ticket is now a lottery ticket that someone will almost certainly try to steal from you once it becomes worth something. Instead of rewarding engineers who keep the lights on, we make them look for another job every 2 years to keep with the market. People like to say tech is eating the world, tech is doing virtually all the killing but the hangers on seem to do the bulk of the eating.
- shoo 5y ago
  Ah, capitalism! The company makes surplus profits that can be extracted by shareholders (and executives and employees, although these get labelled as expenses, not profits) if it is able to sell goods or services at a price point much higher than the cost of producing the service. The cost of keeping the machine running is an operational expense. There is supply and demand in the labour market. If the team keeping the machine running generates $100m of profits for the company and costs $5m in compensation, but it is possible to hire a similar team at market rates to do a comparable job for $5m , then it doesn't make much sense to pay the team more than $5m. The excess value of $95 m generated by the team's activities can be captured as profit. It's also important not to neglect the contributions of other roles. The machine is only able to produce revenue because it is matched with customers who are a good fit for the service the machine delivers and are willing to pay for it. If there are no customers then the machine and the engineering team that support it produce no revenue. So arguably the sales team that is able to identify potential customers and help them understand if renting access to the machine could help them also provide a large amount of value to the company. If you take the same high performing engineering team that builds and operates a valuable service in one company, and drop them into a different business context where there are simply much fewer paying customers, or the sales team isn't able to deliver, then the value the engineering team are able to generate in the new situation may be zero or negative. Maybe most of the value generated by the service isn't due to the team who built and maintain it, but the surrounding business context. Leverage is a big thing. The highest paying roles are often in situations with more leverage (in huge orgs or huge projects), not because the work is necessary more skilled or harder than work in other smaller scale situations. If some of the salespeople and engineers believe they are getting paid disproportionately little compared to the value they create, they could consider banding together and starting their own venture where they would be the owners, not the employees. Much easier said than done, and much easier to stomach if starting in a situation where they have the financial safety net and career connections so they'll be okay if the business fails.
mindvirus 5y ago
I think the challenge that organizations run into with on call is that the costs of it are hidden and delayed - reduced productivity and lower retention. I think that because of this, organizations do not invest in keeping their systems in order. The question is, what do you do about it? At some point things do have to escalate to an engineer, and I don't think it's practical to have a 24/7 engineering team staffed for every service (although having 24/7 non-eng support is very reasonable). The best solution I've seen is having error budgets and on call compensation commensurate with required response times. Error budgets since they balance maintaining and expanding the service, compensation to respect people's time. What have you seen that works?
- supernovae 5y ago
  If your business requires 24x7 coverage, you should pay for 24x7 coverage and not expect the same people working 8x5 are the ones who will do it "because it has to be done". I once worked for a tech company that said they had follow the sun rotation and i was dumb enough to take the job only to find out that the other countries had strict labor laws and follow the sun didn't mean the US folks would have their nights and weekends, but that the US folks were doing off hour support for the rest of the world because we don't have labor laws that protect our workers rights. I literally had to keep a seat warm from 8-5 my time to do the bulk of my work from 5-9pm and on weekends to "work around the customers" and they failed miserably at making tech work... (fin tech.. where they through money and soul sucking work at every problem)
- redisman 5y ago
  Shift work has been a thing for decades. I would imagine hiring some percentage of your devs for the night shift wouldn’t be that hard as many people enjoy that lifestyle. Then you already cover the majority of the 24 hours with normal work
  - JCharante 5y ago
    Additionally there are a lot of devs who are from East Asia but started working in the US after undergrad or grad school. I'm sure there are many of them who would love to return to their home countries and work during the day time (for example, Vietnam timezone is currently a simple 12 hour difference from NYC tz). Even with a salary reduction they'd still come out ahead through lower taxes & the lower cost of living. Additionally you could still hire devs from western backgrounds. While studying abroad my cookies have led to heavy advertising from turing.com, which seems to hire overseas US devs to work remote for US companies. There's plenty of American devs in Chiang Mai working on their stealth phase startup. edit: It seems turing.com is wasting their ad budget, because they have a 4 hour overlap with US TZ requirement. Not sure why they're advertising to IP addresses coming from Asia.
cletus 5y ago
So I've seen personally been involved in this at both Google and Facebook. They have very different approaches. One works reasonably well. One... doesn't. At Google, when you develop a service, you as a software team are responsible for maintaining it. This includes being oncall. But, depending on how critical the service is, you'll get bonus pay for this oncall time. This can amount to >$10k a year. And your oncall may not be that noisy. That is something and (IMHO) significant. Generally you'll be oncall for a week but this can vary. For sufficiently high profile services, oncall may end up being owned by an SRE team. That's not really something that can be thrown over the fence by SWE teams. SRE teams have to accept that ownership. To do this, it requires meeting standards like having an oncall runbook, a sufficiently long history (~6 months), adequate metrics and so on. At that point, SWEs will still be second level support for something SREs may need help resolving. You'll still get paid for this. SREs don't generally have week long oncalls. For the highest profile services, SRE support is global, meaning you'll have team members in 3 time zones such that whoever is oncall is in their normal working hours. So you might have an 8 hour oncall every week or something. At Facebook, generally as a software team you'll be responsible for that service forever. Some key services may be fully or partially supported by PEs (Production Engineers). PEs are less common than Google SREs. You don't get oncall pay at Facebook. It's simply expected. Depending on your management, you may be expected to do that oncall and everything else you're supposed to be doing. So here's why Google's system is better than Facebook's: 1. Oncalls, in my experience, tend to be a lot less noisy at Google. Services tend to be much more mature and stable at Google than Facebook; 2. Teams are generally larger at Google. This means that not everyone needs to be oncall. This is good. There is an optimum size for oncalls. IMHO it is about 8-12. Fewer than 8 and people are oncall too much. More than 12 and you lose skills by not using them often enough. 3. The pay is really important to (2) because it means that those who are oncall at least get compensated for it. At a minimum this sends the message that oncall is important and rewarded. It also means those oncall are less likely to resent those not oncall, which might well be the case if you're not compensated for the extra work; 4. There are a ton of orphan projects at Facebook (IME). By this I mean some product team (in particular) will develop something, ship it and then... forget about it. Onwership of source code trees and projects tends to be fairly strictly enforced at Google. These orphan projects tend to create a number of support issues and tasks that are a drain on oncall as there is no one to pass those tasks on to. 5. Culturally, Google seems to acknowledge and respect oncall work load more than Facebook does. For example, tasks at Facebook have SLAs and I saw plenty of games played to avoid dealing with tasks. Examples: changing priority to increase or remove the SLA, silently closing tasks, passing tasks back months later to the reporter asking "is this still an issue?", etc. So my point is that not wanting to be oncall is understandable but for many engineers, it's going to be part of your job. If so, you need to make sure your organization does it well. There's a world of difference between an oncall you don't get paid for that's 40 hours of work vs one where you get 2 tasks a week and you're getting paid for possibly having to answer an alert. Hating oncall is indicative of a bad engineering culture IME. Things like prioritizing shipping above all else, not rewarding fixing things, unreasonable management driven deadlines and so on. Oncall is the canary in the coal mine for all of that.
throwaway55421 5y ago
Most jobs I've had there was an implicit expectation that if your shit breaks, you fix it. This resulted in systems which didn't break because, well, that keeps work to working hours. A few years back I took a job with no mention of on call and then was told there would be a fortnightly 24 hour slot, which later turned into ~weekly as team members left. I left too. I have no idea why they felt it reasonable to not mention this before I joined, they effectively just wasted a few months pay on me on a gamble that I had no life outside of work. They seemed to find the idea that I don't take my laptop to the pub, or up a mountain, or on holiday, some sort of bizarre way of living. Sick system effect? Who knows.
- Wiseacre 5y ago
  Sign of a wider lack of trust between employer and employee. Yet the idea of unionization is still very unpopular.
  - throwaway55421 5y ago
    I don't follow. Most jobs I've had were high trust, this was the weird exception. I didn't need a union to fix that.
    - Wiseacre 5y ago
      It's not a rare phenomenon across the industry. Plenty of my colleagues and I have experienced something similar.
- cameronh90 5y ago
  > if your shit breaks, you fix it What if you're drunk?
  - throwaway55421 5y ago
    You think about that when you design it. I used to work on trading systems which were managed in real time by a rotating team of traders (i.e. mathematically oriented people as opposed to software engineers). You design the system so that if it breaks there are at least workarounds that anyone can employ without needing to write code. For example, a big stop button, manual adjustments, manual trading, etc. Maybe it stops printing money overnight and you take an opportunity cost loss. That's fine, post mortem at work, fix it. In the worst case if you're not available then someone with ownership steps in like a CTO/founder level (who are of course always on call almost by definition, though they generally have the executive power to say - sod this, we'll just leave it down for a while).
DoneWithAllThat 5y ago
I used to not mind oncall as much, before managerialism and corporate attitudes took over tech. It was fun (I’d also stressful) to feel in the trenches so to speak, while being treated as someone mildly crazy for taking on the responsibility but also not held to the same silly and stupid bureaucratic attitudes that afflicted the rest of the company. Now I’m supposed to talk to businessspeak talk and walk the promo game walk while also supposed to answer the phone at one am and stay up all night saving the business from melting down. Yes I know I’m ranting. The industry has turned into a miserable corporate grind like finance or something, only with the added obligation of someone has to mop up to slop in the middle of the night and that someone is me.
- deleted 5y ago
  [deleted]
habeebtc 5y ago
This model of oncall (what I currently operate off) is less abusive than how DevOps teams were run at my last company. I would get called on days off, evenings, weekends. As late as 1am. I once brought my laptop to a Cubs game, because I knew I would get called (and I did). It was mostly because Devs broke the build, and their managers refused to have them learn how to triage that themselves because it would be extra responsibility.
indymike 5y ago
At some point, there has to be someone to push the right button at 3AM. That can be done by shift work, it can also be handled by on call. In general, a job with on-call requirements should disclose that up front, and compensation should be better than the same position without the on-call requirement. For small teams, on-call may be the only way to do it because there isn't going to be a night shift.
dmitrygr 5y ago
I ask every team I interview with whether they have any sort of on call. A “yes” answer terminates the interview, unless quickly followed by “but we can make an exception”. Call me a monster if you wish, but my time is mine and my family’s.
teeray 5y ago
> My best idea is ‘pay people more,’ but that doesn’t sound so compelling as a pitch. I would not accept a position that paid for on-call, even if they paid more. That just legitimizes encroaching on personal time and likely will make it happen more often. Pay me instead in equivalent PTO (or greater) so that there is a stronger forcing function to reduce pages.
- b3morales 5y ago
  I like this, but of course it collides with the trend towards only offering unspecified (so-called "unlimited") PTO.
  - teeray 5y ago
    I also add a mandatory annual minimum PTO rider when it’s “unlimited”
- dilyevsky 5y ago
  At google you could choose either 1/3-2/3 of your hourly pay (base) for each hour of being oncall outside of 9-5 or equivalent number of hours of additional pto. Best oncall comp structure I’ve seen ever since. Somehow people like to parrot only employee unfriendly practices from big corps…
kayodelycaon 5y ago
I can’t even be on-call if I wanted to. ADA requires a company justify any essential job functions. Justifying a programmer needing to be on call is difficult because it is a secondary role. Programmers aren’t hired to do on-call. The company just decides they can skip hiring anyone for those roles by reusing “existing assets”. Same with single sources of knowledge. They don’t invest in having sufficient capacity. Basically, companies hire people who can do multiple jobs as cheaply as possible and with has as little support as possible. Which is not sustainable. ADA can really highlight dysfunctional systems. For anyone saying a small company can’t afford this, ADA doesn’t apply to companies under 15 employers or that can demonstrate undo-hardship given their resources. (By the way, trying to hire employees as contractors does not work.)
- erik_seaberg 5y ago
  Troubleshooting and mitigating an outage always take priority over anything else. In that sense on call is my job, and I could wake my entire team or ask them to drop everything else if I really needed to. But if I understand the design and code well enough to do that, why wouldn’t I also work on features and fixes in between outages? Why divide the roles when you have to choose among the same small group of people for either?
hirundo 5y ago
There's a salutary effect on developer conscientiousness to put them on call for problems with their own code on production. If a missing nil check means that you get dragged out of bed at peak REM sleep, you check for missing nils harder. You test more. You design your whole project around the requirement of getting uninterrupted sleep. For years I was in this position for a trading company that had pre-market downloads at 4am my time. Bad data and my own bad code woke me up a lot ... which contributed a lot to making that code wake-me-up proof for further years. Of course you should get paid for that time, and there's a diminishing return to such character building exercises. But eating your own dog food is worthwhile particularly in the middle of the night.
- deleted 5y ago
  [deleted]
- kayodelycaon 5y ago
  This assumes your company is okay with spending time to write reliable systems.
Buttons840 5y ago
> It is engineering’s job to own their software I'll accept this responsibility so long as it comes with the corresponding authority. Now tell me, do I really own the software?
- datavirtue 5y ago
  Ahh the ownership argument, dashed upon the rocks.
- redisman 5y ago
  I have denied your request for a new feature because I have ownership over the project and decided we should work on X. Oh I don’t actually have any power but you think I’m an idiot who can’t see that? Cool
- joshuamorton 5y ago
  Of course, but you don't own the business, so the software still needs to meet the businesses goals. This isn't the gotcha you seem to think it is?
cushychicken 5y ago
Hell no, you're not a monster for not wanting to be on call. You're normal. Being on call is fundamentally not that fun. You're also totally normal if you respond to an incentive structure of offering senior level and above employees the perk of not having to carry a pager. Some companies use this to hire senior talent because, hey, it works. As a side note: there's a shitload of negative solidarity in this thread. That's the attitude of: "If I must eat shit, everyone else must eat some too". I think this article (while a little flawed) makes a good pivot towards positive solidarity, and asking the question: "Is there a way to achieve this goal where everyone eats less shit?"
- wildrhythms 5y ago
  When I worked in IT we had a mantra: Don't be a superhero. This was at a massive company of 70k+ employees. Don't stay after hours to fix things. Don't take on more issues than you can handle. Don't agree to unforeseeable deadlines. Backlogs represent a staffing failure. Customers are waiting 2 days for an email response? That's a staffing failure. Working harder to cover up a backlog leads to staffing issues never being addressed, and only passes that burden on to the other IT staff.
- throwaway984393 5y ago
  The best way to keep people from eating shit on-call is to make technology that is failure-resistant. Most software today is not designed to resist failure, and most operations are not designed to resist failure. But it could get better. For example: a trend in recent years was static site generators. You can still find the internet littered with websites that return only errors due to SQL servers breaking long ago. But those static sites, as long as they don't depend on some other precarious tech, will keep working as long as someone's paying the bills. No security patches needed, no logs filling up disks. The servers could still go down, but it won't be your code's fault, so you shouldn't get a call. The majority of failure risk today comes down to security and mutable state. Both of them are easier when the tech is read-only. For security, you can reduce your attack surface until no reasonable attack would work. For mutable state, you have a couple options. First, all changes to both code and data should be versioned and immutable with a working rollback mechanism. Second, never allow a change without verification, and rollback should happen immediately if verification fails. Third, do not depend on technology whose state can or does mutate regularly. Fourth, do not depend on technology that will stop operating once resource limits are exceeded. And fifth, never build something if you can buy it. (The latter has to do with investment in robustness. If you have to build an entire car, you will never have enough time or money or expertise to design a better brake system/suspension system/muffler/etc as someone who specializes in manufacturing just that one part) Another positive solidarity take would be for everyone to stop taking jobs where 24/7 is a requirement. How do you do this? Before signing a contract, require that they put in language that says you will not be required to work outside of normal business hours. If you're a developer, have it say explicitly that you will only be required to maintain application code and not maintain servers. If they balk, don't take the job. Have a lawyer look it over. This doesn't stop the company from firing you in at-will employment states if you pull out the contract one day, so it's better to just walk if they challenge putting it in writing. Some industries just do not expect their users to be 24/7. And a final note: if you are a developer, please don't just wing your app's design in terms of how it will work in production. Don't let the ops person take care of running it for you, and also don't try to figure it out by yourself. Put your brain together with the brain of somebody who specializes in system architecture and operations - at the start of your project, not the end. Design your app along with this other brain, to work in tandem with a very robust system architecture. And then as you build your app, ask the other brain to review it periodically and give you feedback. In terms of car analogies, this is like asking a mechanic for feedback on the parts you are designing. They've seen what fails when and why, what's painful to maintain and what isn't, what's expensive and what's not. I promise you this will result in fewer calls.
snicker7 5y ago
My team is distributed across the world. We ensure that we have multiple SME’s for the most critical systems in each region. No one has to wake up at 3 AM.
JoshTko 5y ago
Not an SWE, but how common is on call duty, and are there no companies that pay spot bonuses per issue?
- SpicyLemonZest 5y ago
  It’s hard to quantify exactly how common it is. Common enough that you’d need to deliberately search for jobs that don’t have it, but not so common that you won’t find any. I’ve never heard of spot bonuses per issue, although I imagine it would create a perverse incentive to have noisy alerting that files a lot of issues. Flat bonuses per oncall shift exist (and I think they’re a great idea) but they’re relatively rare.
- mathteddybear 5y ago
  It is common in internet software companies, less common in other kinds of software companies. Bigger companies tend to have multiple levels of compensation for different oncall rotations.
varispeed 5y ago
Nothing wrong being on call as long as you are being paid to adequately. I always refuse being on-call if I am not being paid full hours plus extra for outside 9 to 5 for the entire duration regardless if anyone called.
ToddWBurgess 5y ago
I was on call for 5 years at an old job (non IT). Getting paged and yanked out of bed at all hours messes with you in so many ways. Being sleep deprived you turn to high sugar drinks and foods to keep you awake. The sleep deprivation and bad diets causes you to put on weight. Going to bed with the idea you could get paged does terrible things for your anxiety. On top of that all the sleep deprivation changes your personality. You become highly irritable and it really brings down your mood. Eventually I couldn't deal with it after 5 years and I pretty much walked off the job. I didn't realize the toll it was taking on me and what being on call was doing to me until I was no longer on call.
jmmv 5y ago
Really liked the article. One thing I'd add (and that I wrote about before here: https://jmmv.dev/2021/07/principal-engineers-oncall.html https://jmmv.dev/2021/07/principal-engineers-oncall.html) is that high-level engineers should be on-call. These engineers (along with management) are the people that can most easily make a difference in making the rotation better: and not because of their technical skills, but because if they "suffer through the pain", they may be able to set priorities to fix things. As some other people have alluded here, though, in some organizations engineers seem to "escape" on-call rotations the more senior they get, which is unfortunate...
- redisman 5y ago
  Get the VPs and Directors on call and see how quickly we’ll change priorities lol
  - mavelikara 5y ago
    In many orgs, the pager escalation path is to VPs and Directors.
marcus_holmes 5y ago
> In a healthy engineering organization, there are no gaps in coverage. Every critical component is owned by a TEAM, not a person. I just joined a large organisation that practices this, and it's driving me nuts. Because no individual is taking ownership, nothing happens. There's no accountability, no drive to improve anything. We ask questions on Slack about "can someone deal with this problem, please?" and nothing happens. Rather than an alarm going off at 2am and the owner dealing with it (again), the alarm goes off and nothing happens, no-one fixes it. Stuff stays broken until it gets a ticket and assigned to a sprint (and even then, people self-assign to tickets, and there's no consequences to not finishing all the tickets on a sprint, so it's still relying on someone to be vaguely interested enough to pick it up and deal with it). I moved from being tech co-founder of a startup (where everything is my responsibility and it's all on me to fix it) to this. It's doing my head in. I think I actually prefer being "on-call" for code I built and have complete control over.
- chii 5y ago
  > I moved from being tech co-founder of a startup (where everything is my responsibility and it's all on me to fix it) to this likely because the startup you co-founded would benefit hugely if you worked to fix stuff, which means you personally would benefit since you're a co-founder. In an employee relationship, this benefit doesn't flow on to the employee, and thus everyone is incentivized to do the least possible that they can get away with. It doesn't gel with your personality, and that is why it's doing your head in.
- ryan_lane 5y ago
  Your team dynamic feels broken, and your manager isn't managing well. On-call is a shared responsibility, and so is ensuring that on-call isn't awful. That's one of the main arguments in this blog post. There's lots of ways to fix this, but one of the more popular approaches is to make the on-call responsible for fixing this kind of stuff during normal working hours, and for that person to not have sprint-work during that period of time. The goal is to ensure the next person's on-call is better than yours, which leads to a continuous improvement mentality, with an end-goal of never being paged after-hours, and for all pages to be actionable.
- 5y ago
Der_Einzige 5y ago
We need federal legislation that bans super strong Service level agreements in cloud tech. Amazon has the best cloud entirely because of their hellish on-call powered by their absurd SLAs. I'm sure this is one of the most important components behind why there is so much oncall. I'd love to see amazon and bezos get punished instead of rewarded just once for being slave drivers. Please pass federal legislation against "999" or other extremely absurd SLAs.
- yuppie_scum 5y ago
  You can’t walk in here and compare slavery to working at a FAANG. Demonstrates either supreme hubris or supreme ignorance. If you don’t like it, fucking quit? Slaves can’t do that
l0b0 5y ago
I and another person are taking over an old batch processing system at work. The system itself is fairly stable, but interacts with some other systems which keep falling over or changing things without adequate notification. Currently there are (mostly false) alarms almost every day, usually for stupid reasons like the network falling over for a few minutes or disk use going above a fairly arbitrary level. So when the subject of weekend on-call came up we just went hard "no", and we've been discussing solutions to that ever since (adjusting cron scheduling, making alarms more forgiving, adjusting customer expectations, improving automation, etc). Importantly, we can do this because we live somewhere with employee protection, and our contracts do not mention being on call or working weekends. If you don't have that you're pretty much SOL.
- physicles 5y ago
  You probably know this already, but getting rid of those false alarms is absolutely critical. Instead of alerting on disk use, alert on user-facing metrics when possible (unless it’s disk space going below a certain threshold, since that’s one of the worst things that can happen).
  - l0b0 5y ago
    Problem is, the only user facing metric is basically "Did you get the last week's updates?", which currently is shipped on Friday and the user usually notices on Monday.
znpy 5y ago
I used to be on call, and now I'm not anymore (but I changed jobs in between). I'm not against being on-call but in my opinion if I get called tonight then tomorrow morning I want to be able to start a process that prevents the problem from happening again, and I want to be able to put EVERYTHING in discussion. I don't want to hear anything about development or operations, that thing must be root caused and it must whoever can implement a fix (either me, my team or the development team, either on our side or on customer side) must drop everything they're doing and implement a correction of error. TL;DR: If I can't fix stuff and prevent the problem from happening again then I don't want be on call.
- deleted 5y ago
  [deleted]
angarg12 5y ago
> Being paged out of hours is as serious as a fucking heart attack. As someone who is oncall regularly and had close ones getting heart attacks I find this offensive. But hey, at the same time I understand the need for hyperbole to grab attention, and generally I think people get offended too easily. Other than that my feelings about oncall are complicated. Most everyone hates it as engineers, but as customers we expect systems to be online all the time and recover quickly if they fail. Managed properly, oncall is a necessity for many products that doesn't need to be too disruptive.
- deleted 5y ago
  [deleted]
poulsbohemian 5y ago
I see a lot of commenters saying it is part of the job and to be expected... that's fine, but in my own career what I observed was that the reason an on call rotation was even necessary was that the companies knew things would fail regularly, because they refused to invest in good practices up front. If a company wants to do everything it can to not turn on call time into a shit show and they want to fairly compensate for a person being available nights and weekends, then cool- we can all be team players. But, too often they want to squeeze you into doing the work of three people if they can get away with it.
steele 5y ago
Having a young kid is an edge-case while on-call is the norm? Software Engineering has jumped the shark and is now commodotized, exploited labor. Laptops and smartphones don't turn engineers into ER physicians.
adxl 5y ago
Never again. I’m out.
mkl95 5y ago
I don't accept jobs that want me to be on call unless fuck you money is involved, and so far nobody has offered it to me. The product that drives most of my employer's revenue is a gigantic pile of tech debt, and there are some people on call whose lives are kind of miserable. What some people struggle to realize is how easily the responsibility of an individual can scale at certain companies, without their compensation improving in the slightest. If you are working on-call and not making big bucks your company is likely milking you dry.
- Consultant32452 5y ago
  I bill my hourly rate for all on-call hours regardless of if I'm called. If my employer wants any amount of control over my time (sobriety, proximity, etc) then they pay my regular hourly rate for it. Most will not accept these terms, and that's the goal.
  - spaetzleesser 5y ago
    That's the way to go. On-call is bearable if you get paid for it. As a contractor who charged by the hour I was perfectly fine with sitting in useless meetings or fixing things at night that weren't even my fault. Doing this without extra compensation is just not OK.
  - atoav 5y ago
    And of course it should be that way: if they want your time (you not being able to do other jobs, you not being able to relax, etc) they should pay for it. Forcing them to think in your hours can also have the nice side effect that they get their communications in order.
- burade 5y ago
  what would you say is big bucks?
  - maerF0x0 5y ago
    levels.fyi
Sophistifunk 5y ago
I'm sure you could pay me enough to be on call again, but so far nobody's offered.
emodendroket 5y ago
Me neither, but I'd like to keep getting paid to work on 24/7 availability systems, so I guess I'll put up with it, like all the other parts of my job I don't want to do.
dsizzle 5y ago
This problem highlights one advantage of having a distributed remote team. In principle, there may not need be a midnight shift.
rasengan 5y ago
Someone else is likely willing to be on call, though.
maerF0x0 5y ago
> you really want to encourage engineers to move around every 2-3 years I have to disagree, I can say I honestly don't hit true full stride until ~1-2 yrs of working with a complex system. A big part of this people really don't like know how to make non-complex systems, and there's next to no incentive in it with the 12-18month new project/promotion cycle. (ie, long-termism is rarely the original writer's concern) In my experience by the time i'm 2-3 years experience with a set of systems I'm many multiples more productive and informed than someone with ~0.5-1 yrs experience on it. And I'm able to tell product/support people exactly how the system behaves in its edges, and how it would hypothetically behave if modified for new features/logic. So, plz managers consider allowing people to become true masters of a domain too.
- PragmaticPulp 5y ago
  I agree. Some degree of moving around between companies is good for a career as it exposes people to different companies’ ways of doing things and helps someone determine what they like (or don’t) about different companies. But there’s also something great about a company where people genuinely enjoy working for the long term and aren’t just using it as a springboard to the next opportunity. Having a lot of employees with short time horizons leads to a lot of sacrificing long-term objectives in favor of short-term line items that sound good on a resume. That’s not to say that people can’t accomplish great things in 1-3 years, but when nobody’s time horizon extends beyond that, every goal gets altered to fit within that time frame. Or worse, people start getting very good at starting ambitious projects, getting them to proof-of-concept phase, and then leaving because they can put it on their resume as an accomplishment. I’ve moved through some companies they were basically a graveyard of half-finished projects they nobody wanted to inherit because they all wanted to start their own thing rather than pick up the hard parts of someone else’s half finished project. Of course, this requires the company to reward employees appropriately to make it worthwhile to stay. Many companies have figured this out, but many smaller corps underpay as long as possible and count on a steady turnover as part of their compensation strategy. Not great, IMO. Become a workplace that people want to stick with for a long time.
- Consultant32452 5y ago
  I find that at the 2-3 year mark too many people have come to know that I will be able to solve their problem, either directly or indirectly. That's when I usually leave, because my day becomes filled with constant "distraction." This distraction may be good for the employer, because problems get solved and things the employer values get done. But it's miserable for me. The part of this career I enjoy is getting that "in the zone" feeling when neck deep in code. If I'm getting yanked in different directions constantly I don't get what I value out of the job anymore.
bastardoperator 5y ago
On call rotation = Company is unwilling to invest in global support of said product. If they're willing to cut corners with their employees precious time off just imagine what other foul business practices are happening behinds the scenes.
deleted 5y ago
[deleted]
debo_ 5y ago
> First, I’d like to say that pager duty isn’t something we should treat like chronic pain or diabetes, where you just constantly manage symptoms and tend to flare-ups day and night. Being paged out of hours is as serious as a fucking heart attack. It should be RARE and taken SERIOUSLY. Resources should be mustered, product cycles should be reassigned, until the problem is fixed. Having experienced both long-term chronic pain and long-term chronic on-call for some grumpy services, I find this comparison ... off-putting. On-call does not feel like it's even remotely close to as difficult.
greenyouse 5y ago
idk, here are some ideas for on call must haves based on other talk about on call things: - normal rotation has 5-8 people - on call lasts for one week - some institutional trigger (like an SLO) exists for changing team priorities when pages happen too often - the culture around on call is empowering, not heroic/dysfunctional - people can still carry on with normal life while on call (just keep your laptop in a backpack and don't get drunk) At least some practices could help make it suck a bit less too: - option for next day off after picking up a page (no questions asked) - frequent pages (more than a couple per quarter) trigger discussions about how to improve code quality or fix bug hot spots - during on call shift you can get a break from normal sprint work (do rewarding things that add value for the team) - there's no fear about being on call since problems are rare and everyone knows how the software systems work - if you're stuck while debugging on call there are more senior people you can page too to get help - work on app monitoring/debugging a little bit of the year to make debugging and triage easier when things do come up - ops team catches the page before your team to provide context and repro - reverting or deploying fixes isn't stressful because the deployment is stable and fully automated Do those seem reasonable? It doesn't seem like a radical idea that the team that wrote the bug has to get pulled into the call to help fix the problem. The anecdotes from this thread make it seem like a healthy on call is very much the exception not the norm though.
efitz 5y ago
Why do so many websites and services need to run 24/7? Put another way, if a non-internet business wanted to operate 24/7, they would have significant expense and logistical problems (to the extent that very few do so). If you’re a PaaS/IaaS provider, ok, you’re going to have to run 24x7. If you’re anything else, maybe you should have a serious discussion about what it’s worth to you to operate around the clock. Of course it’s weird having web sites that are unavailable 24x7 but it’s not unheard of - some government web sites like clerks of court, inspections and GIS already are “business hours only”.
SergeAx 5y ago
I am reading an article and comments and see that typical on-call schedule is weekly. That's really brutal. We have a daily rotation, 3 ops and 5 devs, additionaly paid about 1/2 daily wage no matter if it's weekend or not. We can swap our days if someone has some plans. Overall, being a part of on-call team is really sought after.
paulus_magnus2 5y ago
I think the situation is pretty clear. You either have a significant stake of your empoloyer's success upside or not. This you see by the equity amount given to you. Employers have zero loyality to you as an employee and care the same amount about your career progression so in the end if you're not in for five-six (seven?) figure equity stake you're effectively a contractor / freelancer. By being an employee you're taking unlimited liability for long tail risk of your employer that otherwise would be priced into the freelance or service contract at a very high amount.
mathteddybear 5y ago
I don't often want to be on-call, but when I do, it goes like this My team managed to get paid for oncall that we had set up with a number of false positives, click-to-fix alerts, "override if slightly above threshold", rules for not paging outside of work hours, and whatnot that makes it bearable. That extra compensation is at 1/3 of hourly rate, so effectively it is like paying for working during weekends (1/3 * 24h = 8h) and some smaller bonus for weekdays. Perhaps it might make up for the pandemic inflation

230 points by kiyanwang 5y ago

> I believe it’s reasonable to expect to be woken up 2-3 times a year, as an engineer for that system. Reasonable that may be, but the problem is you don’t know which 2-3 nights it will happen. No thanks.

yuppie_scum 5y ago

Extremely important article. I have been a part of some hell rotations and it is not sustainable at all.

PicassoCTs 5y ago

On call is a dysfunctional company culture attacking its own workforce in a battle of attrition. Its the equivalent of rheumatic fever to a company process. If you need it, your reliability engineering is failing and the process that should mend it, is instead busy fixing symptoms. Such a company is already dieing and will do once the team "on call" is used up, burned out and gone.It will be replaced in the market by something more vital, still capable to value its workforce.

ckdarby 5y ago
Are you saying, every fortune 100 tech company's reliability engineers are failing to do their jobs? MAANG all have on-call individuals even with the fact they're across all timezones and have been growing steady.
- nzmsv 5y ago
  There are different ways of handling on-call and the practices at the companies you mention are very different, as is the on-call load. There is a world of difference between someone voluntarily accepting on-call as part of their duties in exchange for higher pay or a 4 day work week, and on-call being forced on everyone.
- supernovae 5y ago
  He's saying we're all suckers... and sometimes, it takes a long time to find out how true that really may be.

dopylitty 5y ago

This misses the key point that on-call itself is an abusive practice regardless of how well an organization practices it. If an organization thinks their systems should be available 24x7 they should staff people 24x7. There is no situation where it’s ok to contact someone to do work outside of their work hours.

echelon 5y ago
> There is no situation where it’s ok to contact someone to do work outside of their work hours. This lacks perspective. Imagine that your system runs banking, healthcare, telecom, etc. Of course someone needs to be on call 24/7. The best people to hold the pager are the engineers that work on the system.
- shoo 5y ago
  that's a pretty selective quote. you skipped over: > If an organization thinks their systems should be available 24x7 they should staff people 24x7. One way to achieve 24x7 coverage without anyone being structurally on-call outside of standard office hours could be "follow the sun" support -- have offices in multiple timezones around the world and run 3 x 8 hour shifts each day of support working their local office hours to provide 24/7 coverage. Still need enough staff at each site to handle rotation for covering weekends and for people to go on holiday and get sick or quit for new jobs etc. I can understand why many companies would prefer not to pay for that if they can get away with cheaper alternatives, and why they might prefer to understaff and push the burden onto employees, but that's the company's problem, not the problem of the individual worker who trades their life by the hour in exchange for money.
  - echelon 5y ago
    If you're not a global empire with offices all over the world yet, the simpler thing is to pay your engineers well and tell them being oncall is one of the responsibilities.
    - 5y ago

ckdarby 5y ago

Author lists off group of people who should not get pager duty and I think it is unfair to the rest of the team that individuals who fall into the groups outlined are able to escape the duties and responsibilities of the job. If the job posting or interviewing was upfront about the role will have on-call then the interviewee should make the decision to not apply or accept the offer if they're unable to fulfill the responsibilities.

barrkel 5y ago
That attitude leads directly to ageism, sexism and other discriminatory practices, and has second order effects on society which are contrary to the long term interests of capitalism - a stable consumer base with disposable income. Fairness isn't a straightforward concept, but your conception is more naive and - I'd guess - self-serving than most. How do you feel about progressive income taxes? Under a simplistic model of fairness, flat rates seem fair, but in reality the marginal ability to pay increases, and perceived pain of paying decreases, with increased income and wealth. Now expand that concept from the monetary domain to the time domain, and consider how fair it is for people who are particularly time poor - for non-selfish reasons - to need to spend more of their time on the job. Affordances can be made for inclusiveness and they can be fair.
- kayodelycaon 5y ago
  Fairness, as used by the parent, means “if someone else can do something, I should be able to too.” The other person’s situation is never a factor. They see someone getting things for free and they think they should have them for free too.
  - barrkel 5y ago
    Do you feel the same way about maternity and paternity leave?
kayodelycaon 5y ago
Many programming jobs require on-call and overtime when they shouldn’t. I wouldn’t be able to have a job if that requirement was truly valid. It’s usually not. I can’t do on-call for medical reasons, nor can I work overtime. Neither of these are considered to be an essential job function for programmers, regardless of job descriptions, under ADA. A company must justify that job requirement. They can’t. By the way, the “fairness” reason is explicitly excluded by law. It’s the entire reason ADA exists.

deleted 5y ago

[deleted]

willcipriano 5y ago

My argument is that as the tech side of the organization takes on more and more of the operation of the business, it also needs to be taking on more of the rewards. For example if the expectation is that a engineering team designs, builds and maintains the machine twenty four hours a day that is responsible for the majority (if not all) of the firms profits then the majority of the rewards must flow there. One the other hand, traditional methods for doing this have fallen by the wayside. With performance overhangs and other chicanery, what was once a lottery ticket is now a lottery ticket that someone will almost certainly try to steal from you once it becomes worth something. Instead of rewarding engineers who keep the lights on, we make them look for another job every 2 years to keep with the market. People like to say tech is eating the world, tech is doing virtually all the killing but the hangers on seem to do the bulk of the eating.

shoo 5y ago
Ah, capitalism! The company makes surplus profits that can be extracted by shareholders (and executives and employees, although these get labelled as expenses, not profits) if it is able to sell goods or services at a price point much higher than the cost of producing the service. The cost of keeping the machine running is an operational expense. There is supply and demand in the labour market. If the team keeping the machine running generates $100m of profits for the company and costs $5m in compensation, but it is possible to hire a similar team at market rates to do a comparable job for $5m , then it doesn't make much sense to pay the team more than $5m. The excess value of $95 m generated by the team's activities can be captured as profit. It's also important not to neglect the contributions of other roles. The machine is only able to produce revenue because it is matched with customers who are a good fit for the service the machine delivers and are willing to pay for it. If there are no customers then the machine and the engineering team that support it produce no revenue. So arguably the sales team that is able to identify potential customers and help them understand if renting access to the machine could help them also provide a large amount of value to the company. If you take the same high performing engineering team that builds and operates a valuable service in one company, and drop them into a different business context where there are simply much fewer paying customers, or the sales team isn't able to deliver, then the value the engineering team are able to generate in the new situation may be zero or negative. Maybe most of the value generated by the service isn't due to the team who built and maintain it, but the surrounding business context. Leverage is a big thing. The highest paying roles are often in situations with more leverage (in huge orgs or huge projects), not because the work is necessary more skilled or harder than work in other smaller scale situations. If some of the salespeople and engineers believe they are getting paid disproportionately little compared to the value they create, they could consider banding together and starting their own venture where they would be the owners, not the employees. Much easier said than done, and much easier to stomach if starting in a situation where they have the financial safety net and career connections so they'll be okay if the business fails.

mindvirus 5y ago

I think the challenge that organizations run into with on call is that the costs of it are hidden and delayed - reduced productivity and lower retention. I think that because of this, organizations do not invest in keeping their systems in order. The question is, what do you do about it? At some point things do have to escalate to an engineer, and I don't think it's practical to have a 24/7 engineering team staffed for every service (although having 24/7 non-eng support is very reasonable). The best solution I've seen is having error budgets and on call compensation commensurate with required response times. Error budgets since they balance maintaining and expanding the service, compensation to respect people's time. What have you seen that works?

supernovae 5y ago
If your business requires 24x7 coverage, you should pay for 24x7 coverage and not expect the same people working 8x5 are the ones who will do it "because it has to be done". I once worked for a tech company that said they had follow the sun rotation and i was dumb enough to take the job only to find out that the other countries had strict labor laws and follow the sun didn't mean the US folks would have their nights and weekends, but that the US folks were doing off hour support for the rest of the world because we don't have labor laws that protect our workers rights. I literally had to keep a seat warm from 8-5 my time to do the bulk of my work from 5-9pm and on weekends to "work around the customers" and they failed miserably at making tech work... (fin tech.. where they through money and soul sucking work at every problem)
redisman 5y ago
Shift work has been a thing for decades. I would imagine hiring some percentage of your devs for the night shift wouldn’t be that hard as many people enjoy that lifestyle. Then you already cover the majority of the 24 hours with normal work
- JCharante 5y ago
  Additionally there are a lot of devs who are from East Asia but started working in the US after undergrad or grad school. I'm sure there are many of them who would love to return to their home countries and work during the day time (for example, Vietnam timezone is currently a simple 12 hour difference from NYC tz). Even with a salary reduction they'd still come out ahead through lower taxes & the lower cost of living. Additionally you could still hire devs from western backgrounds. While studying abroad my cookies have led to heavy advertising from turing.com, which seems to hire overseas US devs to work remote for US companies. There's plenty of American devs in Chiang Mai working on their stealth phase startup. edit: It seems turing.com is wasting their ad budget, because they have a 4 hour overlap with US TZ requirement. Not sure why they're advertising to IP addresses coming from Asia.

cletus 5y ago

So I've seen personally been involved in this at both Google and Facebook. They have very different approaches. One works reasonably well. One... doesn't. At Google, when you develop a service, you as a software team are responsible for maintaining it. This includes being oncall. But, depending on how critical the service is, you'll get bonus pay for this oncall time. This can amount to >$10k a year. And your oncall may not be that noisy. That is something and (IMHO) significant. Generally you'll be oncall for a week but this can vary. For sufficiently high profile services, oncall may end up being owned by an SRE team. That's not really something that can be thrown over the fence by SWE teams. SRE teams have to accept that ownership. To do this, it requires meeting standards like having an oncall runbook, a sufficiently long history (~6 months), adequate metrics and so on. At that point, SWEs will still be second level support for something SREs may need help resolving. You'll still get paid for this. SREs don't generally have week long oncalls. For the highest profile services, SRE support is global, meaning you'll have team members in 3 time zones such that whoever is oncall is in their normal working hours. So you might have an 8 hour oncall every week or something. At Facebook, generally as a software team you'll be responsible for that service forever. Some key services may be fully or partially supported by PEs (Production Engineers). PEs are less common than Google SREs. You don't get oncall pay at Facebook. It's simply expected. Depending on your management, you may be expected to do that oncall and everything else you're supposed to be doing. So here's why Google's system is better than Facebook's: 1. Oncalls, in my experience, tend to be a lot less noisy at Google. Services tend to be much more mature and stable at Google than Facebook; 2. Teams are generally larger at Google. This means that not everyone needs to be oncall. This is good. There is an optimum size for oncalls. IMHO it is about 8-12. Fewer than 8 and people are oncall too much. More than 12 and you lose skills by not using them often enough. 3. The pay is really important to (2) because it means that those who are oncall at least get compensated for it. At a minimum this sends the message that oncall is important and rewarded. It also means those oncall are less likely to resent those not oncall, which might well be the case if you're not compensated for the extra work; 4. There are a ton of orphan projects at Facebook (IME). By this I mean some product team (in particular) will develop something, ship it and then... forget about it. Onwership of source code trees and projects tends to be fairly strictly enforced at Google. These orphan projects tend to create a number of support issues and tasks that are a drain on oncall as there is no one to pass those tasks on to. 5. Culturally, Google seems to acknowledge and respect oncall work load more than Facebook does. For example, tasks at Facebook have SLAs and I saw plenty of games played to avoid dealing with tasks. Examples: changing priority to increase or remove the SLA, silently closing tasks, passing tasks back months later to the reporter asking "is this still an issue?", etc. So my point is that not wanting to be oncall is understandable but for many engineers, it's going to be part of your job. If so, you need to make sure your organization does it well. There's a world of difference between an oncall you don't get paid for that's 40 hours of work vs one where you get 2 tasks a week and you're getting paid for possibly having to answer an alert. Hating oncall is indicative of a bad engineering culture IME. Things like prioritizing shipping above all else, not rewarding fixing things, unreasonable management driven deadlines and so on. Oncall is the canary in the coal mine for all of that.

throwaway55421 5y ago

Most jobs I've had there was an implicit expectation that if your shit breaks, you fix it. This resulted in systems which didn't break because, well, that keeps work to working hours. A few years back I took a job with no mention of on call and then was told there would be a fortnightly 24 hour slot, which later turned into ~weekly as team members left. I left too. I have no idea why they felt it reasonable to not mention this before I joined, they effectively just wasted a few months pay on me on a gamble that I had no life outside of work. They seemed to find the idea that I don't take my laptop to the pub, or up a mountain, or on holiday, some sort of bizarre way of living. Sick system effect? Who knows.

Wiseacre 5y ago
Sign of a wider lack of trust between employer and employee. Yet the idea of unionization is still very unpopular.
- throwaway55421 5y ago
  I don't follow. Most jobs I've had were high trust, this was the weird exception. I didn't need a union to fix that.
  - Wiseacre 5y ago
    It's not a rare phenomenon across the industry. Plenty of my colleagues and I have experienced something similar.
cameronh90 5y ago
> if your shit breaks, you fix it What if you're drunk?
- throwaway55421 5y ago
  You think about that when you design it. I used to work on trading systems which were managed in real time by a rotating team of traders (i.e. mathematically oriented people as opposed to software engineers). You design the system so that if it breaks there are at least workarounds that anyone can employ without needing to write code. For example, a big stop button, manual adjustments, manual trading, etc. Maybe it stops printing money overnight and you take an opportunity cost loss. That's fine, post mortem at work, fix it. In the worst case if you're not available then someone with ownership steps in like a CTO/founder level (who are of course always on call almost by definition, though they generally have the executive power to say - sod this, we'll just leave it down for a while).

DoneWithAllThat 5y ago

I used to not mind oncall as much, before managerialism and corporate attitudes took over tech. It was fun (I’d also stressful) to feel in the trenches so to speak, while being treated as someone mildly crazy for taking on the responsibility but also not held to the same silly and stupid bureaucratic attitudes that afflicted the rest of the company. Now I’m supposed to talk to businessspeak talk and walk the promo game walk while also supposed to answer the phone at one am and stay up all night saving the business from melting down. Yes I know I’m ranting. The industry has turned into a miserable corporate grind like finance or something, only with the added obligation of someone has to mop up to slop in the middle of the night and that someone is me.

deleted 5y ago
[deleted]

habeebtc 5y ago

This model of oncall (what I currently operate off) is less abusive than how DevOps teams were run at my last company. I would get called on days off, evenings, weekends. As late as 1am. I once brought my laptop to a Cubs game, because I knew I would get called (and I did). It was mostly because Devs broke the build, and their managers refused to have them learn how to triage that themselves because it would be extra responsibility.

indymike 5y ago

At some point, there has to be someone to push the right button at 3AM. That can be done by shift work, it can also be handled by on call. In general, a job with on-call requirements should disclose that up front, and compensation should be better than the same position without the on-call requirement. For small teams, on-call may be the only way to do it because there isn't going to be a night shift.

dmitrygr 5y ago

I ask every team I interview with whether they have any sort of on call. A “yes” answer terminates the interview, unless quickly followed by “but we can make an exception”. Call me a monster if you wish, but my time is mine and my family’s.

teeray 5y ago

> My best idea is ‘pay people more,’ but that doesn’t sound so compelling as a pitch. I would not accept a position that paid for on-call, even if they paid more. That just legitimizes encroaching on personal time and likely will make it happen more often. Pay me instead in equivalent PTO (or greater) so that there is a stronger forcing function to reduce pages.

b3morales 5y ago
I like this, but of course it collides with the trend towards only offering unspecified (so-called "unlimited") PTO.
- teeray 5y ago
  I also add a mandatory annual minimum PTO rider when it’s “unlimited”
dilyevsky 5y ago
At google you could choose either 1/3-2/3 of your hourly pay (base) for each hour of being oncall outside of 9-5 or equivalent number of hours of additional pto. Best oncall comp structure I’ve seen ever since. Somehow people like to parrot only employee unfriendly practices from big corps…

kayodelycaon 5y ago

I can’t even be on-call if I wanted to. ADA requires a company justify any essential job functions. Justifying a programmer needing to be on call is difficult because it is a secondary role. Programmers aren’t hired to do on-call. The company just decides they can skip hiring anyone for those roles by reusing “existing assets”. Same with single sources of knowledge. They don’t invest in having sufficient capacity. Basically, companies hire people who can do multiple jobs as cheaply as possible and with has as little support as possible. Which is not sustainable. ADA can really highlight dysfunctional systems. For anyone saying a small company can’t afford this, ADA doesn’t apply to companies under 15 employers or that can demonstrate undo-hardship given their resources. (By the way, trying to hire employees as contractors does not work.)

erik_seaberg 5y ago
Troubleshooting and mitigating an outage always take priority over anything else. In that sense on call is my job, and I could wake my entire team or ask them to drop everything else if I really needed to. But if I understand the design and code well enough to do that, why wouldn’t I also work on features and fixes in between outages? Why divide the roles when you have to choose among the same small group of people for either?

hirundo 5y ago

There's a salutary effect on developer conscientiousness to put them on call for problems with their own code on production. If a missing nil check means that you get dragged out of bed at peak REM sleep, you check for missing nils harder. You test more. You design your whole project around the requirement of getting uninterrupted sleep. For years I was in this position for a trading company that had pre-market downloads at 4am my time. Bad data and my own bad code woke me up a lot ... which contributed a lot to making that code wake-me-up proof for further years. Of course you should get paid for that time, and there's a diminishing return to such character building exercises. But eating your own dog food is worthwhile particularly in the middle of the night.

deleted 5y ago
[deleted]
kayodelycaon 5y ago
This assumes your company is okay with spending time to write reliable systems.

Buttons840 5y ago

> It is engineering’s job to own their software I'll accept this responsibility so long as it comes with the corresponding authority. Now tell me, do I really own the software?

datavirtue 5y ago
Ahh the ownership argument, dashed upon the rocks.
redisman 5y ago
I have denied your request for a new feature because I have ownership over the project and decided we should work on X. Oh I don’t actually have any power but you think I’m an idiot who can’t see that? Cool
joshuamorton 5y ago
Of course, but you don't own the business, so the software still needs to meet the businesses goals. This isn't the gotcha you seem to think it is?

cushychicken 5y ago

Hell no, you're not a monster for not wanting to be on call. You're normal. Being on call is fundamentally not that fun. You're also totally normal if you respond to an incentive structure of offering senior level and above employees the perk of not having to carry a pager. Some companies use this to hire senior talent because, hey, it works. As a side note: there's a shitload of negative solidarity in this thread. That's the attitude of: "If I must eat shit, everyone else must eat some too". I think this article (while a little flawed) makes a good pivot towards positive solidarity, and asking the question: "Is there a way to achieve this goal where everyone eats less shit?"

wildrhythms 5y ago
When I worked in IT we had a mantra: Don't be a superhero. This was at a massive company of 70k+ employees. Don't stay after hours to fix things. Don't take on more issues than you can handle. Don't agree to unforeseeable deadlines. Backlogs represent a staffing failure. Customers are waiting 2 days for an email response? That's a staffing failure. Working harder to cover up a backlog leads to staffing issues never being addressed, and only passes that burden on to the other IT staff.
throwaway984393 5y ago
The best way to keep people from eating shit on-call is to make technology that is failure-resistant. Most software today is not designed to resist failure, and most operations are not designed to resist failure. But it could get better. For example: a trend in recent years was static site generators. You can still find the internet littered with websites that return only errors due to SQL servers breaking long ago. But those static sites, as long as they don't depend on some other precarious tech, will keep working as long as someone's paying the bills. No security patches needed, no logs filling up disks. The servers could still go down, but it won't be your code's fault, so you shouldn't get a call. The majority of failure risk today comes down to security and mutable state. Both of them are easier when the tech is read-only. For security, you can reduce your attack surface until no reasonable attack would work. For mutable state, you have a couple options. First, all changes to both code and data should be versioned and immutable with a working rollback mechanism. Second, never allow a change without verification, and rollback should happen immediately if verification fails. Third, do not depend on technology whose state can or does mutate regularly. Fourth, do not depend on technology that will stop operating once resource limits are exceeded. And fifth, never build something if you can buy it. (The latter has to do with investment in robustness. If you have to build an entire car, you will never have enough time or money or expertise to design a better brake system/suspension system/muffler/etc as someone who specializes in manufacturing just that one part) Another positive solidarity take would be for everyone to stop taking jobs where 24/7 is a requirement. How do you do this? Before signing a contract, require that they put in language that says you will not be required to work outside of normal business hours. If you're a developer, have it say explicitly that you will only be required to maintain application code and not maintain servers. If they balk, don't take the job. Have a lawyer look it over. This doesn't stop the company from firing you in at-will employment states if you pull out the contract one day, so it's better to just walk if they challenge putting it in writing. Some industries just do not expect their users to be 24/7. And a final note: if you are a developer, please don't just wing your app's design in terms of how it will work in production. Don't let the ops person take care of running it for you, and also don't try to figure it out by yourself. Put your brain together with the brain of somebody who specializes in system architecture and operations - at the start of your project, not the end. Design your app along with this other brain, to work in tandem with a very robust system architecture. And then as you build your app, ask the other brain to review it periodically and give you feedback. In terms of car analogies, this is like asking a mechanic for feedback on the parts you are designing. They've seen what fails when and why, what's painful to maintain and what isn't, what's expensive and what's not. I promise you this will result in fewer calls.

snicker7 5y ago

My team is distributed across the world. We ensure that we have multiple SME’s for the most critical systems in each region. No one has to wake up at 3 AM.

JoshTko 5y ago

Not an SWE, but how common is on call duty, and are there no companies that pay spot bonuses per issue?

SpicyLemonZest 5y ago
It’s hard to quantify exactly how common it is. Common enough that you’d need to deliberately search for jobs that don’t have it, but not so common that you won’t find any. I’ve never heard of spot bonuses per issue, although I imagine it would create a perverse incentive to have noisy alerting that files a lot of issues. Flat bonuses per oncall shift exist (and I think they’re a great idea) but they’re relatively rare.
mathteddybear 5y ago
It is common in internet software companies, less common in other kinds of software companies. Bigger companies tend to have multiple levels of compensation for different oncall rotations.

varispeed 5y ago

Nothing wrong being on call as long as you are being paid to adequately. I always refuse being on-call if I am not being paid full hours plus extra for outside 9 to 5 for the entire duration regardless if anyone called.

ToddWBurgess 5y ago

I was on call for 5 years at an old job (non IT). Getting paged and yanked out of bed at all hours messes with you in so many ways. Being sleep deprived you turn to high sugar drinks and foods to keep you awake. The sleep deprivation and bad diets causes you to put on weight. Going to bed with the idea you could get paged does terrible things for your anxiety. On top of that all the sleep deprivation changes your personality. You become highly irritable and it really brings down your mood. Eventually I couldn't deal with it after 5 years and I pretty much walked off the job. I didn't realize the toll it was taking on me and what being on call was doing to me until I was no longer on call.

jmmv 5y ago

Really liked the article. One thing I'd add (and that I wrote about before here: https://jmmv.dev/2021/07/principal-engineers-oncall.html https://jmmv.dev/2021/07/principal-engineers-oncall.html) is that high-level engineers should be on-call. These engineers (along with management) are the people that can most easily make a difference in making the rotation better: and not because of their technical skills, but because if they "suffer through the pain", they may be able to set priorities to fix things. As some other people have alluded here, though, in some organizations engineers seem to "escape" on-call rotations the more senior they get, which is unfortunate...

redisman 5y ago
Get the VPs and Directors on call and see how quickly we’ll change priorities lol
- mavelikara 5y ago
  In many orgs, the pager escalation path is to VPs and Directors.

marcus_holmes 5y ago

> In a healthy engineering organization, there are no gaps in coverage. Every critical component is owned by a TEAM, not a person. I just joined a large organisation that practices this, and it's driving me nuts. Because no individual is taking ownership, nothing happens. There's no accountability, no drive to improve anything. We ask questions on Slack about "can someone deal with this problem, please?" and nothing happens. Rather than an alarm going off at 2am and the owner dealing with it (again), the alarm goes off and nothing happens, no-one fixes it. Stuff stays broken until it gets a ticket and assigned to a sprint (and even then, people self-assign to tickets, and there's no consequences to not finishing all the tickets on a sprint, so it's still relying on someone to be vaguely interested enough to pick it up and deal with it). I moved from being tech co-founder of a startup (where everything is my responsibility and it's all on me to fix it) to this. It's doing my head in. I think I actually prefer being "on-call" for code I built and have complete control over.

chii 5y ago
> I moved from being tech co-founder of a startup (where everything is my responsibility and it's all on me to fix it) to this likely because the startup you co-founded would benefit hugely if you worked to fix stuff, which means you personally would benefit since you're a co-founder. In an employee relationship, this benefit doesn't flow on to the employee, and thus everyone is incentivized to do the least possible that they can get away with. It doesn't gel with your personality, and that is why it's doing your head in.
ryan_lane 5y ago
Your team dynamic feels broken, and your manager isn't managing well. On-call is a shared responsibility, and so is ensuring that on-call isn't awful. That's one of the main arguments in this blog post. There's lots of ways to fix this, but one of the more popular approaches is to make the on-call responsible for fixing this kind of stuff during normal working hours, and for that person to not have sprint-work during that period of time. The goal is to ensure the next person's on-call is better than yours, which leads to a continuous improvement mentality, with an end-goal of never being paged after-hours, and for all pages to be actionable.
5y ago

Der_Einzige 5y ago

We need federal legislation that bans super strong Service level agreements in cloud tech. Amazon has the best cloud entirely because of their hellish on-call powered by their absurd SLAs. I'm sure this is one of the most important components behind why there is so much oncall. I'd love to see amazon and bezos get punished instead of rewarded just once for being slave drivers. Please pass federal legislation against "999" or other extremely absurd SLAs.

yuppie_scum 5y ago
You can’t walk in here and compare slavery to working at a FAANG. Demonstrates either supreme hubris or supreme ignorance. If you don’t like it, fucking quit? Slaves can’t do that

l0b0 5y ago

I and another person are taking over an old batch processing system at work. The system itself is fairly stable, but interacts with some other systems which keep falling over or changing things without adequate notification. Currently there are (mostly false) alarms almost every day, usually for stupid reasons like the network falling over for a few minutes or disk use going above a fairly arbitrary level. So when the subject of weekend on-call came up we just went hard "no", and we've been discussing solutions to that ever since (adjusting cron scheduling, making alarms more forgiving, adjusting customer expectations, improving automation, etc). Importantly, we can do this because we live somewhere with employee protection, and our contracts do not mention being on call or working weekends. If you don't have that you're pretty much SOL.

physicles 5y ago
You probably know this already, but getting rid of those false alarms is absolutely critical. Instead of alerting on disk use, alert on user-facing metrics when possible (unless it’s disk space going below a certain threshold, since that’s one of the worst things that can happen).
- l0b0 5y ago
  Problem is, the only user facing metric is basically "Did you get the last week's updates?", which currently is shipped on Friday and the user usually notices on Monday.

znpy 5y ago

I used to be on call, and now I'm not anymore (but I changed jobs in between). I'm not against being on-call but in my opinion if I get called tonight then tomorrow morning I want to be able to start a process that prevents the problem from happening again, and I want to be able to put EVERYTHING in discussion. I don't want to hear anything about development or operations, that thing must be root caused and it must whoever can implement a fix (either me, my team or the development team, either on our side or on customer side) must drop everything they're doing and implement a correction of error. TL;DR: If I can't fix stuff and prevent the problem from happening again then I don't want be on call.

deleted 5y ago
[deleted]

angarg12 5y ago

> Being paged out of hours is as serious as a fucking heart attack. As someone who is oncall regularly and had close ones getting heart attacks I find this offensive. But hey, at the same time I understand the need for hyperbole to grab attention, and generally I think people get offended too easily. Other than that my feelings about oncall are complicated. Most everyone hates it as engineers, but as customers we expect systems to be online all the time and recover quickly if they fail. Managed properly, oncall is a necessity for many products that doesn't need to be too disruptive.

deleted 5y ago
[deleted]

poulsbohemian 5y ago

I see a lot of commenters saying it is part of the job and to be expected... that's fine, but in my own career what I observed was that the reason an on call rotation was even necessary was that the companies knew things would fail regularly, because they refused to invest in good practices up front. If a company wants to do everything it can to not turn on call time into a shit show and they want to fairly compensate for a person being available nights and weekends, then cool- we can all be team players. But, too often they want to squeeze you into doing the work of three people if they can get away with it.

steele 5y ago

Having a young kid is an edge-case while on-call is the norm? Software Engineering has jumped the shark and is now commodotized, exploited labor. Laptops and smartphones don't turn engineers into ER physicians.

adxl 5y ago

Never again. I’m out.

mkl95 5y ago

I don't accept jobs that want me to be on call unless fuck you money is involved, and so far nobody has offered it to me. The product that drives most of my employer's revenue is a gigantic pile of tech debt, and there are some people on call whose lives are kind of miserable. What some people struggle to realize is how easily the responsibility of an individual can scale at certain companies, without their compensation improving in the slightest. If you are working on-call and not making big bucks your company is likely milking you dry.

Consultant32452 5y ago
I bill my hourly rate for all on-call hours regardless of if I'm called. If my employer wants any amount of control over my time (sobriety, proximity, etc) then they pay my regular hourly rate for it. Most will not accept these terms, and that's the goal.
- spaetzleesser 5y ago
  That's the way to go. On-call is bearable if you get paid for it. As a contractor who charged by the hour I was perfectly fine with sitting in useless meetings or fixing things at night that weren't even my fault. Doing this without extra compensation is just not OK.
- atoav 5y ago
  And of course it should be that way: if they want your time (you not being able to do other jobs, you not being able to relax, etc) they should pay for it. Forcing them to think in your hours can also have the nice side effect that they get their communications in order.
burade 5y ago
what would you say is big bucks?
- maerF0x0 5y ago
  levels.fyi

Sophistifunk 5y ago

I'm sure you could pay me enough to be on call again, but so far nobody's offered.

emodendroket 5y ago

Me neither, but I'd like to keep getting paid to work on 24/7 availability systems, so I guess I'll put up with it, like all the other parts of my job I don't want to do.

dsizzle 5y ago

This problem highlights one advantage of having a distributed remote team. In principle, there may not need be a midnight shift.

rasengan 5y ago

Someone else is likely willing to be on call, though.

maerF0x0 5y ago

> you really want to encourage engineers to move around every 2-3 years I have to disagree, I can say I honestly don't hit true full stride until ~1-2 yrs of working with a complex system. A big part of this people really don't like know how to make non-complex systems, and there's next to no incentive in it with the 12-18month new project/promotion cycle. (ie, long-termism is rarely the original writer's concern) In my experience by the time i'm 2-3 years experience with a set of systems I'm many multiples more productive and informed than someone with ~0.5-1 yrs experience on it. And I'm able to tell product/support people exactly how the system behaves in its edges, and how it would hypothetically behave if modified for new features/logic. So, plz managers consider allowing people to become true masters of a domain too.

PragmaticPulp 5y ago
I agree. Some degree of moving around between companies is good for a career as it exposes people to different companies’ ways of doing things and helps someone determine what they like (or don’t) about different companies. But there’s also something great about a company where people genuinely enjoy working for the long term and aren’t just using it as a springboard to the next opportunity. Having a lot of employees with short time horizons leads to a lot of sacrificing long-term objectives in favor of short-term line items that sound good on a resume. That’s not to say that people can’t accomplish great things in 1-3 years, but when nobody’s time horizon extends beyond that, every goal gets altered to fit within that time frame. Or worse, people start getting very good at starting ambitious projects, getting them to proof-of-concept phase, and then leaving because they can put it on their resume as an accomplishment. I’ve moved through some companies they were basically a graveyard of half-finished projects they nobody wanted to inherit because they all wanted to start their own thing rather than pick up the hard parts of someone else’s half finished project. Of course, this requires the company to reward employees appropriately to make it worthwhile to stay. Many companies have figured this out, but many smaller corps underpay as long as possible and count on a steady turnover as part of their compensation strategy. Not great, IMO. Become a workplace that people want to stick with for a long time.
Consultant32452 5y ago
I find that at the 2-3 year mark too many people have come to know that I will be able to solve their problem, either directly or indirectly. That's when I usually leave, because my day becomes filled with constant "distraction." This distraction may be good for the employer, because problems get solved and things the employer values get done. But it's miserable for me. The part of this career I enjoy is getting that "in the zone" feeling when neck deep in code. If I'm getting yanked in different directions constantly I don't get what I value out of the job anymore.

bastardoperator 5y ago

On call rotation = Company is unwilling to invest in global support of said product. If they're willing to cut corners with their employees precious time off just imagine what other foul business practices are happening behinds the scenes.

deleted 5y ago

[deleted]

debo_ 5y ago

> First, I’d like to say that pager duty isn’t something we should treat like chronic pain or diabetes, where you just constantly manage symptoms and tend to flare-ups day and night. Being paged out of hours is as serious as a fucking heart attack. It should be RARE and taken SERIOUSLY. Resources should be mustered, product cycles should be reassigned, until the problem is fixed. Having experienced both long-term chronic pain and long-term chronic on-call for some grumpy services, I find this comparison ... off-putting. On-call does not feel like it's even remotely close to as difficult.

greenyouse 5y ago

idk, here are some ideas for on call must haves based on other talk about on call things: - normal rotation has 5-8 people - on call lasts for one week - some institutional trigger (like an SLO) exists for changing team priorities when pages happen too often - the culture around on call is empowering, not heroic/dysfunctional - people can still carry on with normal life while on call (just keep your laptop in a backpack and don't get drunk) At least some practices could help make it suck a bit less too: - option for next day off after picking up a page (no questions asked) - frequent pages (more than a couple per quarter) trigger discussions about how to improve code quality or fix bug hot spots - during on call shift you can get a break from normal sprint work (do rewarding things that add value for the team) - there's no fear about being on call since problems are rare and everyone knows how the software systems work - if you're stuck while debugging on call there are more senior people you can page too to get help - work on app monitoring/debugging a little bit of the year to make debugging and triage easier when things do come up - ops team catches the page before your team to provide context and repro - reverting or deploying fixes isn't stressful because the deployment is stable and fully automated Do those seem reasonable? It doesn't seem like a radical idea that the team that wrote the bug has to get pulled into the call to help fix the problem. The anecdotes from this thread make it seem like a healthy on call is very much the exception not the norm though.

efitz 5y ago

Why do so many websites and services need to run 24/7? Put another way, if a non-internet business wanted to operate 24/7, they would have significant expense and logistical problems (to the extent that very few do so). If you’re a PaaS/IaaS provider, ok, you’re going to have to run 24x7. If you’re anything else, maybe you should have a serious discussion about what it’s worth to you to operate around the clock. Of course it’s weird having web sites that are unavailable 24x7 but it’s not unheard of - some government web sites like clerks of court, inspections and GIS already are “business hours only”.

SergeAx 5y ago

I am reading an article and comments and see that typical on-call schedule is weekly. That's really brutal. We have a daily rotation, 3 ops and 5 devs, additionaly paid about 1/2 daily wage no matter if it's weekend or not. We can swap our days if someone has some plans. Overall, being a part of on-call team is really sought after.

paulus_magnus2 5y ago

I think the situation is pretty clear. You either have a significant stake of your empoloyer's success upside or not. This you see by the equity amount given to you. Employers have zero loyality to you as an employee and care the same amount about your career progression so in the end if you're not in for five-six (seven?) figure equity stake you're effectively a contractor / freelancer. By being an employee you're taking unlimited liability for long tail risk of your employer that otherwise would be priced into the freelance or service contract at a very high amount.

mathteddybear 5y ago

I don't often want to be on-call, but when I do, it goes like this My team managed to get paid for oncall that we had set up with a number of false positives, click-to-fix alerts, "override if slightly above threshold", rules for not paging outside of work hours, and whatnot that makes it bearable. That extra compensation is at 1/3 of hourly rate, so effectively it is like paying for working during weekends (1/3 * 24h = 8h) and some smaller bonus for weekdays. Perhaps it might make up for the pandemic inflation