3 ms·

The problem of being on-call is not only the potential issues that can appear at 3am in the morning. For me, at least, the worst part is the fact that I cannot

The problem of being on-call is not only the potential issues that can appear at 3am in the morning. For me, at least, the worst part is the fact that I cannot do anything outside home like going out for a run, or to to the cinema, or for a walk with my friends. I cannot even go for a 30 min walk without taking my laptop!

gregjor 4y ago
I get calls, messages, emails on my phone. I don't tell customers I will sit in front of my screen 24/7. I tell them one hour or less. If they need faster response than that they will have to plan for it and pay for it. I'm guessing you have a f/t job with on-call required. Do you really need to have your laptop on-hand every minute? I have had jobs in the past with managers who told me I would have to do on-call duty. I told them no, my kids and personal life mean more to me. I agreed to carry a phone or pager, and get to problems as fast as I could, but not let on-call intrude so much I was effectively working extra hours for free. I never got penalized or fired for setting some limits. I never knew a manager or executive who was on call 24/7 or carried their laptop around at the golf course.
- tkiolp4 4y ago
  I’m not on call, but when doing interviews I explicitly ask about the on-call setup (if any). The usual setup (for backend engineers) is: - 1 week in-call every 5 to 6 weeks (depending on how many engineers are available for on call) - if you get paged, you need to acknowledge within 10-15 min. Sometimes it’s nothing and you can go back to sleep, but sometimes you may need your laptop to fix things. The point is: you don’t know before hand - it’s paid. In my opinion the money is not worth it - it’s mandatory. But even if it’s not, if all your teammates are on call and you’re the only one who refuses it, well, that usually creates friction
  - gregjor 4y ago
    As I wrote already, if an app needs this level of babysitting, fix the code and/or the infrastructure. If having people on-call 24/7 seems normal to management, they need to step up and address the bugs, failures, downtime, and the lack of redundancy, monitoring, and automation. I would strive to get the support burden down to a very occasional alert on my phone that I can respond to from my phone. When I see this kind of thing with my customers it comes from a combination of unrealistic expectations ("We need 99.9999% uptime for our marketing sites") and over-engineered and fragile infrastructure, code, and development/deployment processes ("Our K8S/microservices architecture can fail in many ways that no one understands.") I did some work for a company a few years ago that had a cron job that rebooted the web server every day, because that was the solution they had come up with to "fix" the server crashing (Apache + Rails in that case). The actual problem was running MySQL and Apache on the same server, leading to MySQL eventually starving Apache for memory. Once I fixed that Apache + Rails would run for years without needed a reboot. I had another customer with a server that ran out of disk space every couple of months, requiring frantic resizing of an attached disk. Moved them to S3, problem solved, no more calls.
    - lknfrnpt 4y ago
      "If having people on-call 24/7 seems normal to management, they need to step up and address the bugs, failures, downtime, and the lack of redundancy, monitoring, and automation." "I would strive to get the support burden down to a very occasional alert on my phone that I can respond to from my phone." It's possible we're working on different kinds (or scales) of software or differently sized organizations – there are no human customers actually "calling," nor can you typically resolve true production incidents (let alone root cause them or estimate impact) from a phone.

gregjor 4y ago

I get calls, messages, emails on my phone. I don't tell customers I will sit in front of my screen 24/7. I tell them one hour or less. If they need faster response than that they will have to plan for it and pay for it. I'm guessing you have a f/t job with on-call required. Do you really need to have your laptop on-hand every minute? I have had jobs in the past with managers who told me I would have to do on-call duty. I told them no, my kids and personal life mean more to me. I agreed to carry a phone or pager, and get to problems as fast as I could, but not let on-call intrude so much I was effectively working extra hours for free. I never got penalized or fired for setting some limits. I never knew a manager or executive who was on call 24/7 or carried their laptop around at the golf course.

tkiolp4 4y ago
I’m not on call, but when doing interviews I explicitly ask about the on-call setup (if any). The usual setup (for backend engineers) is: - 1 week in-call every 5 to 6 weeks (depending on how many engineers are available for on call) - if you get paged, you need to acknowledge within 10-15 min. Sometimes it’s nothing and you can go back to sleep, but sometimes you may need your laptop to fix things. The point is: you don’t know before hand - it’s paid. In my opinion the money is not worth it - it’s mandatory. But even if it’s not, if all your teammates are on call and you’re the only one who refuses it, well, that usually creates friction
- gregjor 4y ago
  As I wrote already, if an app needs this level of babysitting, fix the code and/or the infrastructure. If having people on-call 24/7 seems normal to management, they need to step up and address the bugs, failures, downtime, and the lack of redundancy, monitoring, and automation. I would strive to get the support burden down to a very occasional alert on my phone that I can respond to from my phone. When I see this kind of thing with my customers it comes from a combination of unrealistic expectations ("We need 99.9999% uptime for our marketing sites") and over-engineered and fragile infrastructure, code, and development/deployment processes ("Our K8S/microservices architecture can fail in many ways that no one understands.") I did some work for a company a few years ago that had a cron job that rebooted the web server every day, because that was the solution they had come up with to "fix" the server crashing (Apache + Rails in that case). The actual problem was running MySQL and Apache on the same server, leading to MySQL eventually starving Apache for memory. Once I fixed that Apache + Rails would run for years without needed a reboot. I had another customer with a server that ran out of disk space every couple of months, requiring frantic resizing of an attached disk. Moved them to S3, problem solved, no more calls.
  - lknfrnpt 4y ago
    "If having people on-call 24/7 seems normal to management, they need to step up and address the bugs, failures, downtime, and the lack of redundancy, monitoring, and automation." "I would strive to get the support burden down to a very occasional alert on my phone that I can respond to from my phone." It's possible we're working on different kinds (or scales) of software or differently sized organizations – there are no human customers actually "calling," nor can you typically resolve true production incidents (let alone root cause them or estimate impact) from a phone.