3 ms·

I’m not on call, but when doing interviews I explicitly ask about the on-call setup (if any). The usual setup (for backend engineers) is: - 1 week in-call ever

I’m not on call, but when doing interviews I explicitly ask about the on-call setup (if any). The usual setup (for backend engineers) is: - 1 week in-call every 5 to 6 weeks (depending on how many engineers are available for on call) - if you get paged, you need to acknowledge within 10-15 min. Sometimes it’s nothing and you can go back to sleep, but sometimes you may need your laptop to fix things. The point is: you don’t know before hand - it’s paid. In my opinion the money is not worth it - it’s mandatory. But even if it’s not, if all your teammates are on call and you’re the only one who refuses it, well, that usually creates friction

gregjor 4y ago
As I wrote already, if an app needs this level of babysitting, fix the code and/or the infrastructure. If having people on-call 24/7 seems normal to management, they need to step up and address the bugs, failures, downtime, and the lack of redundancy, monitoring, and automation. I would strive to get the support burden down to a very occasional alert on my phone that I can respond to from my phone. When I see this kind of thing with my customers it comes from a combination of unrealistic expectations ("We need 99.9999% uptime for our marketing sites") and over-engineered and fragile infrastructure, code, and development/deployment processes ("Our K8S/microservices architecture can fail in many ways that no one understands.") I did some work for a company a few years ago that had a cron job that rebooted the web server every day, because that was the solution they had come up with to "fix" the server crashing (Apache + Rails in that case). The actual problem was running MySQL and Apache on the same server, leading to MySQL eventually starving Apache for memory. Once I fixed that Apache + Rails would run for years without needed a reboot. I had another customer with a server that ran out of disk space every couple of months, requiring frantic resizing of an attached disk. Moved them to S3, problem solved, no more calls.
- lknfrnpt 4y ago
  "If having people on-call 24/7 seems normal to management, they need to step up and address the bugs, failures, downtime, and the lack of redundancy, monitoring, and automation." "I would strive to get the support burden down to a very occasional alert on my phone that I can respond to from my phone." It's possible we're working on different kinds (or scales) of software or differently sized organizations – there are no human customers actually "calling," nor can you typically resolve true production incidents (let alone root cause them or estimate impact) from a phone.

gregjor 4y ago

As I wrote already, if an app needs this level of babysitting, fix the code and/or the infrastructure. If having people on-call 24/7 seems normal to management, they need to step up and address the bugs, failures, downtime, and the lack of redundancy, monitoring, and automation. I would strive to get the support burden down to a very occasional alert on my phone that I can respond to from my phone. When I see this kind of thing with my customers it comes from a combination of unrealistic expectations ("We need 99.9999% uptime for our marketing sites") and over-engineered and fragile infrastructure, code, and development/deployment processes ("Our K8S/microservices architecture can fail in many ways that no one understands.") I did some work for a company a few years ago that had a cron job that rebooted the web server every day, because that was the solution they had come up with to "fix" the server crashing (Apache + Rails in that case). The actual problem was running MySQL and Apache on the same server, leading to MySQL eventually starving Apache for memory. Once I fixed that Apache + Rails would run for years without needed a reboot. I had another customer with a server that ran out of disk space every couple of months, requiring frantic resizing of an attached disk. Moved them to S3, problem solved, no more calls.

lknfrnpt 4y ago
"If having people on-call 24/7 seems normal to management, they need to step up and address the bugs, failures, downtime, and the lack of redundancy, monitoring, and automation." "I would strive to get the support burden down to a very occasional alert on my phone that I can respond to from my phone." It's possible we're working on different kinds (or scales) of software or differently sized organizations – there are no human customers actually "calling," nor can you typically resolve true production incidents (let alone root cause them or estimate impact) from a phone.