16 ms·

This misses the key point that on-call itself is an abusive practice regardless of how well an organization practices it. If an organization thinks their syste

This misses the key point that on-call itself is an abusive practice regardless of how well an organization practices it. If an organization thinks their systems should be available 24x7 they should staff people 24x7. There is no situation where it’s ok to contact someone to do work outside of their work hours.

echelon 5y ago
> There is no situation where it’s ok to contact someone to do work outside of their work hours. This lacks perspective. Imagine that your system runs banking, healthcare, telecom, etc. Of course someone needs to be on call 24/7. The best people to hold the pager are the engineers that work on the system.
- shoo 5y ago
  that's a pretty selective quote. you skipped over: > If an organization thinks their systems should be available 24x7 they should staff people 24x7. One way to achieve 24x7 coverage without anyone being structurally on-call outside of standard office hours could be "follow the sun" support -- have offices in multiple timezones around the world and run 3 x 8 hour shifts each day of support working their local office hours to provide 24/7 coverage. Still need enough staff at each site to handle rotation for covering weekends and for people to go on holiday and get sick or quit for new jobs etc. I can understand why many companies would prefer not to pay for that if they can get away with cheaper alternatives, and why they might prefer to understaff and push the burden onto employees, but that's the company's problem, not the problem of the individual worker who trades their life by the hour in exchange for money.
  - echelon 5y ago
    If you're not a global empire with offices all over the world yet, the simpler thing is to pay your engineers well and tell them being oncall is one of the responsibilities.
    - ipaddr 5y ago
      If you are not a global company being down at 3am doesn't matter either.
      - valzam 5y ago
        Because of course all companies exist in the same time zone as all their clients... Have you ever worked at a SaaS company? Having customers from around the world is literally a thing from day 1, especially if you are not US or (to a lesser extent already) EU based. Try starting a SaaS company in Australia and see how far "doesn't matter if it goes down at 3am" gets you.
        ipaddr 5y ago
        Starting a company in Australia and targeting a completely different timezone and not staff support at 3am when traffic is heaviest seems like gambling.
      - shoo 5y ago
        Arguably the importance of availability and time to restore service is less about if the company is global and more about the nature of the services they provide to their customers. Global ecommerce company letting people buy junk they don't need 24/7 at low low prices? An outage may frustrate customers and lose some sales but they might get most of the customers back in the long run, even if it takes them days to get the site to come back up and stay up. Bits of the local electricity grid get damaged? Communications for ambulance dispatch go down and stay down? Those outages may directly cause fatalities or greatly increase the risk of fatalities until problems are safely isolated and service can be restored.
- whateveracct 5y ago
  And when they hold the pager, they should get to do nothing else during the day - no feature dev etc. I don't care if there's a deadline..you should have accounted for on-call when planning. The only thing they should be doing besides responding to incidents is work to stabilize the system to reduce the need to page people. That's a fair way to make it work imo.
hirundo 5y ago
"There is no situation where it’s ok to contact someone to do work outside of their work hours." Here's one. I get paged for urgent fixes maybe three times per year. To staff all three shifts with senior engineers to cover for that would be enough to make the company unprofitable and lose many jobs. I don't much like being woken in the middle of the night for such things, but it's clearly necessary and not abusive.
- deleted 5y ago
  [deleted]
- whycombinatore 5y ago
  On call typically means you gotta be online and respond within 15 minutes if paged. That's abusive. If you want a person to do that, then pay them for that. As an example, on call in Amazon works the way I described, and yes it is abusive.
  - sokoloff 5y ago
    One argument is that you are being paid for it; it’s just not broken out into two lines, but the dollars are there. If you’re making $200K/yr, maybe $150K of that is for your job and $50K is for being on-call. That seems like a more than fair price (or at least one that’s in the ballpark) for periodic on-call service.
    - kelnos 5y ago
      In my experience, often people paid more end up not having on-call duties. I know several people who have negotiated high salaries from the start of their employment along with a stipulation that they will never be on the on-call rotation. In my own situation, I removed myself from on-call duties during a leave of absence, and never went back on after I returned to work. Since then, I've still gotten the same raises I got before.
      - lumost 5y ago
        An unfortunate reality of on call is that companies care about projects delivered, not disasters averted. Unless your stepping in to right the ship when a company is on fire operationally, and can point to specific action/results you delivered to right the ship - getting pages 8x per week does nothing for your career.
deleted 5y ago
[deleted]
exsmelliarmus 5y ago
> If an organization thinks their systems should be available 24x7 they should staff people 24x7. I’m a software engineer. I’m paid well partly because things like oncall are necessary. It’s priced into the compensation. > There is no situation where it’s ok to contact someone to do work outside of their work hours. It sounds like you should find a company that agrees with this opinion. I don’t agree.
- deleted 5y ago
  [deleted]
- coldtea 5y ago
  >I’m a software engineer. I’m paid well partly because things like oncall are necessary. It’s priced into the compensation. Or you priced yourself down.
- tragictrash 5y ago
  Haha, That's why you still have to do on call.
- _l4lu 5y ago
  This attitude is very common with software engineers and I find it baffling. Is it some kind of inferiority complex? I doubt Warren Buffett worries that he is overpaid and demands to be woken up in the middle of the night for some self-flagellation. So why do so many software engineers think "I am paid well so I deserve whatever the company throws at me"? You are selling something: your skills, expertise, and time on this planet. These things are limited and valuable. Negotiate the price and terms of the sale!
  - option_greek 5y ago
    I would say its a superiority complex : thinking software engineers are paid well and hence need to be ready to sacrifice more to maintain this superior position. And of course also because the risks of bad or interrupted sleep is not recognised widely and mostly ignored.
thrower123 5y ago
Industrial companies that have an assembly line go down call people in to fix it at time and a half or double pay. Something is a little fucked with tech for doing it for free.
- rubinelli 5y ago
  You should also be compensated for the time you spend on-call (which means you can't take on other commitments you otherwise might.) Anything less and it's outright exploitation. "We pay you well" is a non-sequitur.
  - ploxiln 5y ago
    > "We pay you well" is a non-sequitur. Not at all. The choice might be between $60k salary plus around $20k for overtime in one career, vs another career with $200k+ salary plus crazy benefits (gym memberships, a variety of mental health services, $20k in fertility treatments (no joke), 3+ months paternity leave, over 10 more things I can't even name they make no sense...) I don't even use any of the benefits, just the money (and the parental leave) but calling a few on-call nights per month "abuse" or "exploitation" is just ... without perspective. Nobody that does real work has it this cushy - not doctors, not teachers, not construction workers. Maybe aristocratic political appointees or something, but jeez, no field that is accessible to anyone with an old laptop and sufficient motivation. This is as cushy as it gets, for a real job. Enjoy it while it lasts.
    - deleted 5y ago
      [deleted]
    - credittw2021 5y ago
      Agreed. My brother makes about what I do, but he is an MBA and a plant manager with background in logistics. Sometimes he gets calls and has to go in to work. Hell there has been at least one instance where he had to drop everything and courier a bag of parts on a commercial flight so a line wouldn't stop. That said, I have worked in 'abusive' on call environments. I.e. we had a system that would break at least once a week between 3am and 6am (like, multiple times in that period). But because the oncall labor to remediate was 'free', fixing the problem was never a priority over the 2 years I worked there.
emerged 5y ago
It seems perfectly okay if you were hired at a job where on-call is a stated requirement. Is this a controversial opinion? I’ll admit that I sometimes find myself out of phase with some of the dominant HN views on at-will employment, but it would be useful for me to know what the perceived flaw in my stance is.
- monksy 5y ago
  No. If a job is expecting you to be oncall is the part of your job, they are looking for ways to not pay you for the work. (This is not talking about an occasional once a year expectation. ) This is a rotating or permnant basis of where you are expected to be ready to work and they're not paying you.
  - austhrow743 5y ago
    If someone offers you money while at the same time providing a list of things you need to do to get that money, they are paying you to do those things.
    - ipaddr 5y ago
      You are agreeing for them not to pay you standard pay for 24 hour availability. The company is getting out of paying for that availability.
      - joshuamorton 5y ago
        So if you insist on not doing oncall and they reduce your salary, were you getting paid for oncall or not?
        monksy 5y ago
        If you are expected to be on call you should calculate that as part as your hourly salary. You can quickly be a person that is working below minimum wage.
ebiester 5y ago
So, can this mythical 24x7 support the code you wrote? Did you write documentation that explains every case that could go wrong? If your team writes code that doesn’t need to be supported by an engineer, then your team won’t get called.
- redisman 5y ago
  Hah hilarious. Let me just go tell management our bar is now “never breaks” rather than “ship it”.
- pylua 5y ago
  The truth is that other services break , the network breaks , and someone will believe it’s your software and it will be your burden to prove it at 2am in the morning .
- deleted 5y ago
  [deleted]
deleted 5y ago
[deleted]
nouveaux 5y ago
Doctors are on call. Firemen are on call. There are plenty of jobs that are not on call. Lots of enterprise positions do not require you to be on call. If you work on a site that requires 24/7 uptime, why is it so hard to accept that engineers have to be on call?
- ipaddr 5y ago
  Most Doctors are not on call. Police are not on call. Firemen being on call sounds unbelievable from a union perspective but many volunteer fire departments exist. On call for a week every 8 weeks is a lot different to a once a year event like a manager getting a call that a break-in happened at a store.
  - blacksmith_tb 5y ago
    Firefighters in the US typically work 24hr shifts, then have 48hr off, then repeat[1]. You could say they are 'on call' during that 24hr shift, of course they don't normally fight fires every minute of it (luckily), much of that time is spent doing things in the firehouse like maintenance, eating, etc. 1: https://firefighternow.com/firefighter-shift-schedules-and-working-hours-explained/ https://firefighternow.com/firefighter-shift-schedules-and-w...
  - deleted 5y ago
    [deleted]
giantg2 5y ago
I don't think it's abusive. If you rotate on a team of 8, that should mean you go on-call once every 8 weeks (assuming weekly on-call). This really isn't bad in my opinion. Especially if the systems are decently architected and maintained. The real problem is that management doesn't like to pay for updates/upgrades/fixes to make the system more stable (tying back to the management/engineering paragraph in the article). Disclosure: I'm currently on call.
- throwaway55421 5y ago
  Er, what? On call for an entire week at a time? Does this actually exist? It's like an... anti-holiday. Insane.
  - sys_64738 5y ago
    Yes it does exist. I've done plenty of on-call shifts but for paid a minimum of three hours whether I solved the problem in 5 mins or 50 mins. It's great when you're young, starting out, as it's a big pile of money. When you're over thirty, not so much.
  - valzam 5y ago
    I mean it really depends on what the oncall looks like. If your system breaks every 5 minutes or you have other teams escalating non-urgent tasks constantly then 1 week on call seems crazy. But there are many companies (mine included) where on-call basically means: If services go down you have to figure out why. No bug fixing etc. Oncall purely from a infrastructure perspective.
  - magicalhippo 5y ago
    Our company does that. Each person in the support team has one week on call. The following week they get the Friday off, and they get some extra compensation in addition just for being on call. Overtime if they have to work of course. Only customers who pay a lot for it have true 24/7, most who pay extra have extended support from IIRC 06-22. If they can't sort it out, support will try as best they can to work around issues as to not disturb us devs until next morning. I've been woken at most once a year. Most support folks do say it can be a bit stressful at times, on the other hand they also really enjoy the extended weekend so nobody's been keen to change it. Part of why it's so useful to do a full week at a time, rather than rotate intra-week, is that whoever's on call needs to have a small purse full of token generators for 2-factor logins, and we typically only get one to share. If they don't have the token they can't help our customer.
franciscop 5y ago
The way I've heard it works in the US seems abusive, but it doesn't need to be. I thought it was regulated in Spain (to give an alternative point of view), but apparently it's only lightly regulated and the pay is left to either the unions to negotiate or to the contract. However, it's regulated that: - If you do have to work, that's "extraordinary" time paid at 1.75x the normal hourly rate (yes, including for full time employees). - While you don't need breaks (it's kinda considered a break on itself), you do need at least 12h of rest between shifts, so that on itself makes it basically impossible to be on call the whole time between two work days. It makes it possible to be on call for the weekends though, and between two work days for 4h.
- hhh 5y ago
  The way my team handles this in the US and EU is ‘Best Effort,’ where you are provided a company phone and the expectation is that if you’re able to reasonably do so, answer the call. From there, you can triage (is the problem real, how critical is this, is there an alternate method to achieve your goal.) Each team member has services and/or sites they are responsible for, but our team is flexible enough to work at any site as they’re heavily standardized. If you’re not able or willing to help (or go in if things are REALLY bad), as long as you can get ahold of someone else capable of helping or a supervisor and hand off to continue. If you need to go in, 2x pay that day and vacation time. It’s more if it’s disruptive (2 am friday vs 11 am on a Sunday.) Out of 47 sites, this has happened one time in the past year where something went horrifically wrong with some Cisco gear configurations. We’re moving even further to get rid of having a company phone, and moving to a tablet that has our comms tools and enough to RDP somewhere. This moves from ‘answer the phone if you’re able to please’ to ‘if you get an urgent Teams notification check it out please if possible.’ Strict on-call is hard, but in our environment most people are willing to help one another out if something is up. This could mean immediate sudden travel, but only if you are willing and capable of doing so. I personally always accept them, because they’re mad fun road trips and you get time off, extra pay, and I like helping my team members when they’re in a pinch. People have lives, I just have a more flexible one :).
andrewfong 5y ago
> If an organization thinks their systems should be available 24x7 they should staff people 24x7. This is hard to do well. Ideally, you need an eng culture where things are well-documented and product development is regularly handed off between multiple time zones. Otherwise, what happens is that you just end up with product development happening in one time zone and an SRE or Ops team getting screwed over by changes they don't understand.
sershe 5y ago
I really hate on-call, sadly in any serious distributed systems used by many customers/running large scale online services on-call is table stakes :| It's not like SWEs are paid by hour. If you are on call for a week every 2-3 months (for example), you could just consider the extra 2 work weeks (assuming 8 hour days for a normal week) of part-time work to be part of your paycheck. Staffing competent people 24x7 is completely impractical in a sufficiently complex system... what you can have, and I have experienced, is some clueless 24x7 first-tier support. These guys don't work on the product so they usually cannot solve any problem that is not completely basic. And if they can't solve a problem, they'd call a component owner (or a random person) at any time, which is worse than someone being on-call. If you have a rotation, you can safely disappear when you are not on, because there's someone competent responsible for dealing with live site.
shados 5y ago
Staffing folks who can fix these systems 24/7 for the 2-3 times a year something needs to happen doesn't seem like a great idea to me. If the systems are failing left and right continually, or if on-call is used to handle things that can be done from a simple runbook? Sure. But at some point you need an escalation path when its more than that, and sooner or later that escalation path goes to whoever built the thing. These are usually salaried jobs, so "outside of work hour" is kind of subjective, since you're paid to get the job done. So then it comes to expectations, and $$$$. Software engineer salaries are not just a result of the supply vs demand (well, everything is supply vs demand, of course, but there's subtleties). It wasn't that long ago (late 90s, early 2000s) that a 9-5 code monkey job was a low paying job. It makes sense: it's not particularly difficult. But then organizations realized that they could get more bangs for their bucks if the same folks could handle uncertainties, if they would train themselves, if they could architect systems on their own, and yes, if they were responsible for keeping the systems running. That, of course, came at a cost. In addition to growing demands raising wages, you also need to pay a premium to get someone to agree to all that crap. And so they did. But now as so many people entered the field, there's an expectation that the salaries are the baseline, and that the other responsibilities beyond code monkeying are a premium. Supply and demand will kick in sooner or later, when folks realize that 6 figure salaries for a 9-5 desk job is a really good deal (we already see it happening, where entry level roles are getting hard to come by). But in the end, the additional duties, like oncall, really have been baked in the salaries. I really don't think its unreasonable to have someone woken up a handful of times when taking salaries that can go as high as competing with a doctor's... (who's also on call). Now, that's within reason. My partner worked at AWS for a while, and they'd get paged 5-6 times a night on a bad week. F* that. No amount of money is worth screwing over your health.
blitzar 5y ago
> they should staff people 24x7 Perhaps, if the workload for the overnight team is low, instead of having them sit in an office from 1am to 6am in a dark room staring at a flickering screen you work on alternate solution for staffing. How about, as a solutution to the problem, we let them work from home and provided they have all the tools at their disposal, upon recieving some kind of notification, attend to the situation.

echelon 5y ago

> There is no situation where it’s ok to contact someone to do work outside of their work hours. This lacks perspective. Imagine that your system runs banking, healthcare, telecom, etc. Of course someone needs to be on call 24/7. The best people to hold the pager are the engineers that work on the system.

shoo 5y ago
that's a pretty selective quote. you skipped over: > If an organization thinks their systems should be available 24x7 they should staff people 24x7. One way to achieve 24x7 coverage without anyone being structurally on-call outside of standard office hours could be "follow the sun" support -- have offices in multiple timezones around the world and run 3 x 8 hour shifts each day of support working their local office hours to provide 24/7 coverage. Still need enough staff at each site to handle rotation for covering weekends and for people to go on holiday and get sick or quit for new jobs etc. I can understand why many companies would prefer not to pay for that if they can get away with cheaper alternatives, and why they might prefer to understaff and push the burden onto employees, but that's the company's problem, not the problem of the individual worker who trades their life by the hour in exchange for money.
- echelon 5y ago
  If you're not a global empire with offices all over the world yet, the simpler thing is to pay your engineers well and tell them being oncall is one of the responsibilities.
  - ipaddr 5y ago
    If you are not a global company being down at 3am doesn't matter either.
    - valzam 5y ago
      Because of course all companies exist in the same time zone as all their clients... Have you ever worked at a SaaS company? Having customers from around the world is literally a thing from day 1, especially if you are not US or (to a lesser extent already) EU based. Try starting a SaaS company in Australia and see how far "doesn't matter if it goes down at 3am" gets you.
      - ipaddr 5y ago
        Starting a company in Australia and targeting a completely different timezone and not staff support at 3am when traffic is heaviest seems like gambling.
    - shoo 5y ago
      Arguably the importance of availability and time to restore service is less about if the company is global and more about the nature of the services they provide to their customers. Global ecommerce company letting people buy junk they don't need 24/7 at low low prices? An outage may frustrate customers and lose some sales but they might get most of the customers back in the long run, even if it takes them days to get the site to come back up and stay up. Bits of the local electricity grid get damaged? Communications for ambulance dispatch go down and stay down? Those outages may directly cause fatalities or greatly increase the risk of fatalities until problems are safely isolated and service can be restored.
whateveracct 5y ago
And when they hold the pager, they should get to do nothing else during the day - no feature dev etc. I don't care if there's a deadline..you should have accounted for on-call when planning. The only thing they should be doing besides responding to incidents is work to stabilize the system to reduce the need to page people. That's a fair way to make it work imo.

hirundo 5y ago

"There is no situation where it’s ok to contact someone to do work outside of their work hours." Here's one. I get paged for urgent fixes maybe three times per year. To staff all three shifts with senior engineers to cover for that would be enough to make the company unprofitable and lose many jobs. I don't much like being woken in the middle of the night for such things, but it's clearly necessary and not abusive.

deleted 5y ago
[deleted]
whycombinatore 5y ago
On call typically means you gotta be online and respond within 15 minutes if paged. That's abusive. If you want a person to do that, then pay them for that. As an example, on call in Amazon works the way I described, and yes it is abusive.
- sokoloff 5y ago
  One argument is that you are being paid for it; it’s just not broken out into two lines, but the dollars are there. If you’re making $200K/yr, maybe $150K of that is for your job and $50K is for being on-call. That seems like a more than fair price (or at least one that’s in the ballpark) for periodic on-call service.
  - kelnos 5y ago
    In my experience, often people paid more end up not having on-call duties. I know several people who have negotiated high salaries from the start of their employment along with a stipulation that they will never be on the on-call rotation. In my own situation, I removed myself from on-call duties during a leave of absence, and never went back on after I returned to work. Since then, I've still gotten the same raises I got before.
    - lumost 5y ago
      An unfortunate reality of on call is that companies care about projects delivered, not disasters averted. Unless your stepping in to right the ship when a company is on fire operationally, and can point to specific action/results you delivered to right the ship - getting pages 8x per week does nothing for your career.

deleted 5y ago

[deleted]

exsmelliarmus 5y ago

> If an organization thinks their systems should be available 24x7 they should staff people 24x7. I’m a software engineer. I’m paid well partly because things like oncall are necessary. It’s priced into the compensation. > There is no situation where it’s ok to contact someone to do work outside of their work hours. It sounds like you should find a company that agrees with this opinion. I don’t agree.

deleted 5y ago
[deleted]
coldtea 5y ago
>I’m a software engineer. I’m paid well partly because things like oncall are necessary. It’s priced into the compensation. Or you priced yourself down.
tragictrash 5y ago
Haha, That's why you still have to do on call.
_l4lu 5y ago
This attitude is very common with software engineers and I find it baffling. Is it some kind of inferiority complex? I doubt Warren Buffett worries that he is overpaid and demands to be woken up in the middle of the night for some self-flagellation. So why do so many software engineers think "I am paid well so I deserve whatever the company throws at me"? You are selling something: your skills, expertise, and time on this planet. These things are limited and valuable. Negotiate the price and terms of the sale!
- option_greek 5y ago
  I would say its a superiority complex : thinking software engineers are paid well and hence need to be ready to sacrifice more to maintain this superior position. And of course also because the risks of bad or interrupted sleep is not recognised widely and mostly ignored.

thrower123 5y ago

Industrial companies that have an assembly line go down call people in to fix it at time and a half or double pay. Something is a little fucked with tech for doing it for free.

rubinelli 5y ago
You should also be compensated for the time you spend on-call (which means you can't take on other commitments you otherwise might.) Anything less and it's outright exploitation. "We pay you well" is a non-sequitur.
- ploxiln 5y ago
  > "We pay you well" is a non-sequitur. Not at all. The choice might be between $60k salary plus around $20k for overtime in one career, vs another career with $200k+ salary plus crazy benefits (gym memberships, a variety of mental health services, $20k in fertility treatments (no joke), 3+ months paternity leave, over 10 more things I can't even name they make no sense...) I don't even use any of the benefits, just the money (and the parental leave) but calling a few on-call nights per month "abuse" or "exploitation" is just ... without perspective. Nobody that does real work has it this cushy - not doctors, not teachers, not construction workers. Maybe aristocratic political appointees or something, but jeez, no field that is accessible to anyone with an old laptop and sufficient motivation. This is as cushy as it gets, for a real job. Enjoy it while it lasts.
  - deleted 5y ago
    [deleted]
  - credittw2021 5y ago
    Agreed. My brother makes about what I do, but he is an MBA and a plant manager with background in logistics. Sometimes he gets calls and has to go in to work. Hell there has been at least one instance where he had to drop everything and courier a bag of parts on a commercial flight so a line wouldn't stop. That said, I have worked in 'abusive' on call environments. I.e. we had a system that would break at least once a week between 3am and 6am (like, multiple times in that period). But because the oncall labor to remediate was 'free', fixing the problem was never a priority over the 2 years I worked there.

emerged 5y ago

It seems perfectly okay if you were hired at a job where on-call is a stated requirement. Is this a controversial opinion? I’ll admit that I sometimes find myself out of phase with some of the dominant HN views on at-will employment, but it would be useful for me to know what the perceived flaw in my stance is.

monksy 5y ago
No. If a job is expecting you to be oncall is the part of your job, they are looking for ways to not pay you for the work. (This is not talking about an occasional once a year expectation. ) This is a rotating or permnant basis of where you are expected to be ready to work and they're not paying you.
- austhrow743 5y ago
  If someone offers you money while at the same time providing a list of things you need to do to get that money, they are paying you to do those things.
  - ipaddr 5y ago
    You are agreeing for them not to pay you standard pay for 24 hour availability. The company is getting out of paying for that availability.
    - joshuamorton 5y ago
      So if you insist on not doing oncall and they reduce your salary, were you getting paid for oncall or not?
      - monksy 5y ago
        If you are expected to be on call you should calculate that as part as your hourly salary. You can quickly be a person that is working below minimum wage.

ebiester 5y ago

So, can this mythical 24x7 support the code you wrote? Did you write documentation that explains every case that could go wrong? If your team writes code that doesn’t need to be supported by an engineer, then your team won’t get called.

redisman 5y ago
Hah hilarious. Let me just go tell management our bar is now “never breaks” rather than “ship it”.
pylua 5y ago
The truth is that other services break , the network breaks , and someone will believe it’s your software and it will be your burden to prove it at 2am in the morning .
deleted 5y ago
[deleted]

deleted 5y ago

[deleted]

nouveaux 5y ago

Doctors are on call. Firemen are on call. There are plenty of jobs that are not on call. Lots of enterprise positions do not require you to be on call. If you work on a site that requires 24/7 uptime, why is it so hard to accept that engineers have to be on call?

ipaddr 5y ago
Most Doctors are not on call. Police are not on call. Firemen being on call sounds unbelievable from a union perspective but many volunteer fire departments exist. On call for a week every 8 weeks is a lot different to a once a year event like a manager getting a call that a break-in happened at a store.
- blacksmith_tb 5y ago
  Firefighters in the US typically work 24hr shifts, then have 48hr off, then repeat[1]. You could say they are 'on call' during that 24hr shift, of course they don't normally fight fires every minute of it (luckily), much of that time is spent doing things in the firehouse like maintenance, eating, etc. 1: https://firefighternow.com/firefighter-shift-schedules-and-working-hours-explained/ https://firefighternow.com/firefighter-shift-schedules-and-w...
- deleted 5y ago
  [deleted]

giantg2 5y ago

I don't think it's abusive. If you rotate on a team of 8, that should mean you go on-call once every 8 weeks (assuming weekly on-call). This really isn't bad in my opinion. Especially if the systems are decently architected and maintained. The real problem is that management doesn't like to pay for updates/upgrades/fixes to make the system more stable (tying back to the management/engineering paragraph in the article). Disclosure: I'm currently on call.

throwaway55421 5y ago
Er, what? On call for an entire week at a time? Does this actually exist? It's like an... anti-holiday. Insane.
- sys_64738 5y ago
  Yes it does exist. I've done plenty of on-call shifts but for paid a minimum of three hours whether I solved the problem in 5 mins or 50 mins. It's great when you're young, starting out, as it's a big pile of money. When you're over thirty, not so much.
- valzam 5y ago
  I mean it really depends on what the oncall looks like. If your system breaks every 5 minutes or you have other teams escalating non-urgent tasks constantly then 1 week on call seems crazy. But there are many companies (mine included) where on-call basically means: If services go down you have to figure out why. No bug fixing etc. Oncall purely from a infrastructure perspective.
- magicalhippo 5y ago
  Our company does that. Each person in the support team has one week on call. The following week they get the Friday off, and they get some extra compensation in addition just for being on call. Overtime if they have to work of course. Only customers who pay a lot for it have true 24/7, most who pay extra have extended support from IIRC 06-22. If they can't sort it out, support will try as best they can to work around issues as to not disturb us devs until next morning. I've been woken at most once a year. Most support folks do say it can be a bit stressful at times, on the other hand they also really enjoy the extended weekend so nobody's been keen to change it. Part of why it's so useful to do a full week at a time, rather than rotate intra-week, is that whoever's on call needs to have a small purse full of token generators for 2-factor logins, and we typically only get one to share. If they don't have the token they can't help our customer.

franciscop 5y ago

The way I've heard it works in the US seems abusive, but it doesn't need to be. I thought it was regulated in Spain (to give an alternative point of view), but apparently it's only lightly regulated and the pay is left to either the unions to negotiate or to the contract. However, it's regulated that: - If you do have to work, that's "extraordinary" time paid at 1.75x the normal hourly rate (yes, including for full time employees). - While you don't need breaks (it's kinda considered a break on itself), you do need at least 12h of rest between shifts, so that on itself makes it basically impossible to be on call the whole time between two work days. It makes it possible to be on call for the weekends though, and between two work days for 4h.

hhh 5y ago
The way my team handles this in the US and EU is ‘Best Effort,’ where you are provided a company phone and the expectation is that if you’re able to reasonably do so, answer the call. From there, you can triage (is the problem real, how critical is this, is there an alternate method to achieve your goal.) Each team member has services and/or sites they are responsible for, but our team is flexible enough to work at any site as they’re heavily standardized. If you’re not able or willing to help (or go in if things are REALLY bad), as long as you can get ahold of someone else capable of helping or a supervisor and hand off to continue. If you need to go in, 2x pay that day and vacation time. It’s more if it’s disruptive (2 am friday vs 11 am on a Sunday.) Out of 47 sites, this has happened one time in the past year where something went horrifically wrong with some Cisco gear configurations. We’re moving even further to get rid of having a company phone, and moving to a tablet that has our comms tools and enough to RDP somewhere. This moves from ‘answer the phone if you’re able to please’ to ‘if you get an urgent Teams notification check it out please if possible.’ Strict on-call is hard, but in our environment most people are willing to help one another out if something is up. This could mean immediate sudden travel, but only if you are willing and capable of doing so. I personally always accept them, because they’re mad fun road trips and you get time off, extra pay, and I like helping my team members when they’re in a pinch. People have lives, I just have a more flexible one :).

andrewfong 5y ago

> If an organization thinks their systems should be available 24x7 they should staff people 24x7. This is hard to do well. Ideally, you need an eng culture where things are well-documented and product development is regularly handed off between multiple time zones. Otherwise, what happens is that you just end up with product development happening in one time zone and an SRE or Ops team getting screwed over by changes they don't understand.

sershe 5y ago

I really hate on-call, sadly in any serious distributed systems used by many customers/running large scale online services on-call is table stakes :| It's not like SWEs are paid by hour. If you are on call for a week every 2-3 months (for example), you could just consider the extra 2 work weeks (assuming 8 hour days for a normal week) of part-time work to be part of your paycheck. Staffing competent people 24x7 is completely impractical in a sufficiently complex system... what you can have, and I have experienced, is some clueless 24x7 first-tier support. These guys don't work on the product so they usually cannot solve any problem that is not completely basic. And if they can't solve a problem, they'd call a component owner (or a random person) at any time, which is worse than someone being on-call. If you have a rotation, you can safely disappear when you are not on, because there's someone competent responsible for dealing with live site.

shados 5y ago

Staffing folks who can fix these systems 24/7 for the 2-3 times a year something needs to happen doesn't seem like a great idea to me. If the systems are failing left and right continually, or if on-call is used to handle things that can be done from a simple runbook? Sure. But at some point you need an escalation path when its more than that, and sooner or later that escalation path goes to whoever built the thing. These are usually salaried jobs, so "outside of work hour" is kind of subjective, since you're paid to get the job done. So then it comes to expectations, and $$$$. Software engineer salaries are not just a result of the supply vs demand (well, everything is supply vs demand, of course, but there's subtleties). It wasn't that long ago (late 90s, early 2000s) that a 9-5 code monkey job was a low paying job. It makes sense: it's not particularly difficult. But then organizations realized that they could get more bangs for their bucks if the same folks could handle uncertainties, if they would train themselves, if they could architect systems on their own, and yes, if they were responsible for keeping the systems running. That, of course, came at a cost. In addition to growing demands raising wages, you also need to pay a premium to get someone to agree to all that crap. And so they did. But now as so many people entered the field, there's an expectation that the salaries are the baseline, and that the other responsibilities beyond code monkeying are a premium. Supply and demand will kick in sooner or later, when folks realize that 6 figure salaries for a 9-5 desk job is a really good deal (we already see it happening, where entry level roles are getting hard to come by). But in the end, the additional duties, like oncall, really have been baked in the salaries. I really don't think its unreasonable to have someone woken up a handful of times when taking salaries that can go as high as competing with a doctor's... (who's also on call). Now, that's within reason. My partner worked at AWS for a while, and they'd get paged 5-6 times a night on a bad week. F* that. No amount of money is worth screwing over your health.

blitzar 5y ago

> they should staff people 24x7 Perhaps, if the workload for the overnight team is low, instead of having them sit in an office from 1am to 6am in a dark room staring at a flickering screen you work on alternate solution for staffing. How about, as a solutution to the problem, we let them work from home and provided they have all the tools at their disposal, upon recieving some kind of notification, attend to the situation.