5 ms·

Software engineers having "on call" schedules at all is crazy to me. You shouldn't be writing code at 3am to fix a bug after working all day just to turn around

by pc86 2y ago

Software engineers having "on call" schedules at all is crazy to me. You shouldn't be writing code at 3am to fix a bug after working all day just to turn around and work the next day as well.

totally 2y ago
What would be more sane? Allowing things to remain broken at night? Having someone who is not a software engineer fix it?
- tobyjsullivan 2y ago
  If a 7-Eleven is open 24 hours a day, they usually hire three 8-hour shifts (roughly speaking).
- bpfrh 2y ago
  Either pay another full time night shift or yes, accept that things are broken for the time. If your product is really so important that it can't be down, hire more engineers and pass the markup to your customers. I'm glad I live in the country where you have to have 11 hours between end of work and start of work(except for special cases afaik).
  - slaymaker1907 2y ago
    There are legitimate reasons to pull in someone after hours, but it really has to be catastrophic. I'd 100% want to be called in if I deployed something knocking out 911 service for a whole state and I was the only one with the knowledge to actually fix it in a timely manner. However, most problems are not like that and are either able to be delayed until an actual business day or can be solved by someone else.
    - pc86 2y ago
      Let's be real, we're talking about line of business apps and ecommerce stores making $5,000/day total revenue. "Critical infrastructure" has an entirely different failure model.
- dakiol 2y ago
  If the product that breaks in the night is SO important for the company, well, why is not the company paying for dedicated people (not the engineers who create the product) to take care of it when it's broken? As said above, while on-call you don't write code, you just turn off feature flags, reboot machines, etc. If the company cannot afford that, then the product is not that important and can remain broken until the morning. Even 24h fast food places hire 3 people (each working 8h)!
- antisthenes 2y ago
  If you want things to not break, have redundancy in hardware and failover modes that let you function in reduced capacity. Manual fixes should never be done in a hurry, and if your system is that fragile, I really wonder about the competency of your senior employees and leadership.
rawgabbit 2y ago
It is a self fulfilling prophecy. Unrealistic schedules results in crappy code which results in pagers/alert going off at all hours which results in unrealistic schedules. Agile's answer is that we reduce the scope of things delivered. You might as well spit into the wind. Deadlines are set by the business, no matter what the Agile evangelist said.
vizzier 2y ago
My experience being "on call" as an engineer has mostly not been that you need to write code at 3am. It usually comes down to restarting a machine, deploying a new machine or copy, or informing the rest of the company that some 3rd party API that you rely on is currently down.
- MadnessASAP 2y ago
  But that's not engineering work, that's technician or operator work. The engineer comes in later to discuss what went wrong and how to prevent it next time. Speaking as a technician whose seen 3 AM at work many a time.
  - geoelectric 2y ago
    I think the devops-inspired idea of engineers owning what they deploy has become fairly popular, for better or for worse.
    - vizzier 2y ago
      For my own part, this wasn't a huge team. We had the knowledge of if the issue was application/software based but would pass back to ops if it was hardware/OS related. One possible bonus, being on call operating your own software also gives you a solid incentive to not wake yourself up in the morning by writing bad code, and fixing those issues that do arise quickly.
      - Viliam1234 2y ago
        > being on call operating your own software also gives you a solid incentive to not wake yourself up in the morning by writing bad code Unfortunately, my software interacts over network with software written by other people; if something goes wrong at 3 AM the users don't know which part caused the problem, so they wake up a random person.
        fragmede 2y ago
crystal_revenge 2y ago
Seriously! Working weekends in retail when I was young, one of the hallmarks of a "real, professional job" was not having to work nights/weekends when your routine schedule is during the day. It was a major motivator to get through school and get skilled. Now I see young engineers from top-tier school working "on call" without complaint. I've found ways to avoid such roles, but it always seemed ridiculous and completely unnecessary in a world where there are software engineers around the globe that could easily work full time support positions.
- silverquiet 2y ago
  I found it to be rather the opposite; when I was off from a wage-slave job, I was actually off. If the boss called, you could just ignore it and say you missed it because you were studying or sleeping or with friends or whatever and they couldn't really say anything because they knew they didn't pay you enough to care. When you're making real money, they own your ass.
  - bravetraveler 2y ago
    Indeed, those who are exempt or whatever with salaries... consider it purchase/lease with at-will employment in most states. Both salary and hourly gigs have income hanging by a thread with plenty of work, yet only one can get overtime. Be given responsibility/salary for something (aka hired) by a particularly needy manager/org and be 'undependable'. Read: not at their call. See how it turns out. The worst/eventual outcome: bye-bye money. Hopefully one has a more reasonable environment. Workers have little on their side. As someone who does SRE (not AWS, elsewhere)... I would absolutely prefer pay as an hourly rate over salary. I don't like putting in more hours/making less money because Developer Kelly had a bad launch... but I have to, The 9s (and bills) Must Flow. Fortunately, my current place takes this into account. I don't actually need bonuses or structure change... but the larger trends remain. The employer is buying you, salary opens the time box.
chipdart 2y ago
> Software engineers having "on call" schedules at all is crazy to me. You shouldn't be writing code at 3am to fix a bug after working all day just to turn around and work the next day as well. Oncall is only crazy to anyone who also believes it's totally acceptable to have whole services down for hours throughout the night. To those who understand what it takes to have anything available 24/7, you understand damn well that you need someone to jump on a laptop as soon as an alarm bell rings.
- 9dev 2y ago
  Well, that someone better be someone else than me, because I'm not going to do unpaid night shifts. If you want something running 24/7, it's surely important enough to warrant hiring someone else to take care of it while I'm asleep, no? Keep your fancy valley salary (with the ridiculous rent prices attached), and I'll keep my European workers right's protection—including undisturbed sleep after my 8 hours workday.
  - spdionis 2y ago
    Is it an US thing that on-call shifts are unpaid? That is not common in Europe. Generous compensation and additional time off is quite typical for engineers handling on-call burdens.
    - rawgabbit 2y ago
      Yes it is. At one company, I was technically on call 24 hours a day 7 days a week for over ten years. Did I get called that often? No. Did I get called at the worst possible moments? Yes.
      - chipdart 2y ago
        That's not a problem with the concept of being oncall. That's an entirely different problem that's not technical nor operational not industry-specific.
devbent 2y ago
It works well of the company empowers engineers to write software that doesn't break all the time. At HBO Max every incident had a full writeup and then real solutions were put in place to make the service more stable. My team had around 3 incidents in 2 years. If the cultural expectation is that the on call buzzer will never go off, and that it going off is a Bad Thing, then on call itself isn't a problem. Or as I was fond of saying "my number one design criteria (for software) is that everyone gets to sleep through the night." The customers win (stable service) and the engineers win (sleep).
- wheelinsupial 2y ago
  What was the time to implementation for the real solutions? Was there any cost considerations associated with prioritizing the implementation or even limiting the scope of the solution?
  - devbent 2y ago
    The tooling there was amazing so a barebones services could get deployed into prod in a couple days if needs be. That typically didn't happen because engineering reviews had to occur first. A single command created a new repo, setup ingress/egress configs in AWS, and setup all the boilerplate to handle secrets management, environment configs, and the like. All that was left to do was business logic.
    - wheelinsupial 2y ago
      It sounds like if you have production issues, there is a priority to put out an actual fix immediately.
      - devbent 2y ago
        It depends on the severity of the issue. If the issue impacts tens of millions of customers, then yes, get it fixed right now. Extended outages can be front page news. Too many in a row and people leave the service. Ideally monitoring catches outages when they first get started and run books have steps to quickly restore service even if a full fix cannot be put into place immediately.

by pc86 2y ago

Software engineers having "on call" schedules at all is crazy to me. You shouldn't be writing code at 3am to fix a bug after working all day just to turn around and work the next day as well.

totally 2y ago

What would be more sane? Allowing things to remain broken at night? Having someone who is not a software engineer fix it?

tobyjsullivan 2y ago
If a 7-Eleven is open 24 hours a day, they usually hire three 8-hour shifts (roughly speaking).
bpfrh 2y ago
Either pay another full time night shift or yes, accept that things are broken for the time. If your product is really so important that it can't be down, hire more engineers and pass the markup to your customers. I'm glad I live in the country where you have to have 11 hours between end of work and start of work(except for special cases afaik).
- slaymaker1907 2y ago
  There are legitimate reasons to pull in someone after hours, but it really has to be catastrophic. I'd 100% want to be called in if I deployed something knocking out 911 service for a whole state and I was the only one with the knowledge to actually fix it in a timely manner. However, most problems are not like that and are either able to be delayed until an actual business day or can be solved by someone else.
  - pc86 2y ago
    Let's be real, we're talking about line of business apps and ecommerce stores making $5,000/day total revenue. "Critical infrastructure" has an entirely different failure model.
dakiol 2y ago
If the product that breaks in the night is SO important for the company, well, why is not the company paying for dedicated people (not the engineers who create the product) to take care of it when it's broken? As said above, while on-call you don't write code, you just turn off feature flags, reboot machines, etc. If the company cannot afford that, then the product is not that important and can remain broken until the morning. Even 24h fast food places hire 3 people (each working 8h)!
antisthenes 2y ago
If you want things to not break, have redundancy in hardware and failover modes that let you function in reduced capacity. Manual fixes should never be done in a hurry, and if your system is that fragile, I really wonder about the competency of your senior employees and leadership.

rawgabbit 2y ago

It is a self fulfilling prophecy. Unrealistic schedules results in crappy code which results in pagers/alert going off at all hours which results in unrealistic schedules. Agile's answer is that we reduce the scope of things delivered. You might as well spit into the wind. Deadlines are set by the business, no matter what the Agile evangelist said.

vizzier 2y ago

My experience being "on call" as an engineer has mostly not been that you need to write code at 3am. It usually comes down to restarting a machine, deploying a new machine or copy, or informing the rest of the company that some 3rd party API that you rely on is currently down.

MadnessASAP 2y ago
But that's not engineering work, that's technician or operator work. The engineer comes in later to discuss what went wrong and how to prevent it next time. Speaking as a technician whose seen 3 AM at work many a time.
- geoelectric 2y ago
  I think the devops-inspired idea of engineers owning what they deploy has become fairly popular, for better or for worse.
  - vizzier 2y ago
    For my own part, this wasn't a huge team. We had the knowledge of if the issue was application/software based but would pass back to ops if it was hardware/OS related. One possible bonus, being on call operating your own software also gives you a solid incentive to not wake yourself up in the morning by writing bad code, and fixing those issues that do arise quickly.
    - Viliam1234 2y ago
      > being on call operating your own software also gives you a solid incentive to not wake yourself up in the morning by writing bad code Unfortunately, my software interacts over network with software written by other people; if something goes wrong at 3 AM the users don't know which part caused the problem, so they wake up a random person.
      - fragmede 2y ago

crystal_revenge 2y ago

Seriously! Working weekends in retail when I was young, one of the hallmarks of a "real, professional job" was not having to work nights/weekends when your routine schedule is during the day. It was a major motivator to get through school and get skilled. Now I see young engineers from top-tier school working "on call" without complaint. I've found ways to avoid such roles, but it always seemed ridiculous and completely unnecessary in a world where there are software engineers around the globe that could easily work full time support positions.

silverquiet 2y ago
I found it to be rather the opposite; when I was off from a wage-slave job, I was actually off. If the boss called, you could just ignore it and say you missed it because you were studying or sleeping or with friends or whatever and they couldn't really say anything because they knew they didn't pay you enough to care. When you're making real money, they own your ass.
- bravetraveler 2y ago
  Indeed, those who are exempt or whatever with salaries... consider it purchase/lease with at-will employment in most states. Both salary and hourly gigs have income hanging by a thread with plenty of work, yet only one can get overtime. Be given responsibility/salary for something (aka hired) by a particularly needy manager/org and be 'undependable'. Read: not at their call. See how it turns out. The worst/eventual outcome: bye-bye money. Hopefully one has a more reasonable environment. Workers have little on their side. As someone who does SRE (not AWS, elsewhere)... I would absolutely prefer pay as an hourly rate over salary. I don't like putting in more hours/making less money because Developer Kelly had a bad launch... but I have to, The 9s (and bills) Must Flow. Fortunately, my current place takes this into account. I don't actually need bonuses or structure change... but the larger trends remain. The employer is buying you, salary opens the time box.

chipdart 2y ago

> Software engineers having "on call" schedules at all is crazy to me. You shouldn't be writing code at 3am to fix a bug after working all day just to turn around and work the next day as well. Oncall is only crazy to anyone who also believes it's totally acceptable to have whole services down for hours throughout the night. To those who understand what it takes to have anything available 24/7, you understand damn well that you need someone to jump on a laptop as soon as an alarm bell rings.

9dev 2y ago
Well, that someone better be someone else than me, because I'm not going to do unpaid night shifts. If you want something running 24/7, it's surely important enough to warrant hiring someone else to take care of it while I'm asleep, no? Keep your fancy valley salary (with the ridiculous rent prices attached), and I'll keep my European workers right's protection—including undisturbed sleep after my 8 hours workday.
- spdionis 2y ago
  Is it an US thing that on-call shifts are unpaid? That is not common in Europe. Generous compensation and additional time off is quite typical for engineers handling on-call burdens.
  - rawgabbit 2y ago
    Yes it is. At one company, I was technically on call 24 hours a day 7 days a week for over ten years. Did I get called that often? No. Did I get called at the worst possible moments? Yes.
    - chipdart 2y ago
      That's not a problem with the concept of being oncall. That's an entirely different problem that's not technical nor operational not industry-specific.

devbent 2y ago

It works well of the company empowers engineers to write software that doesn't break all the time. At HBO Max every incident had a full writeup and then real solutions were put in place to make the service more stable. My team had around 3 incidents in 2 years. If the cultural expectation is that the on call buzzer will never go off, and that it going off is a Bad Thing, then on call itself isn't a problem. Or as I was fond of saying "my number one design criteria (for software) is that everyone gets to sleep through the night." The customers win (stable service) and the engineers win (sleep).

wheelinsupial 2y ago
What was the time to implementation for the real solutions? Was there any cost considerations associated with prioritizing the implementation or even limiting the scope of the solution?
- devbent 2y ago
  The tooling there was amazing so a barebones services could get deployed into prod in a couple days if needs be. That typically didn't happen because engineering reviews had to occur first. A single command created a new repo, setup ingress/egress configs in AWS, and setup all the boilerplate to handle secrets management, environment configs, and the like. All that was left to do was business logic.
  - wheelinsupial 2y ago
    It sounds like if you have production issues, there is a priority to put out an actual fix immediately.
    - devbent 2y ago
      It depends on the severity of the issue. If the issue impacts tens of millions of customers, then yes, get it fixed right now. Extended outages can be front page news. Too many in a row and people leave the service. Ideally monitoring catches outages when they first get started and run books have steps to quickly restore service even if a full fix cannot be put into place immediately.