3 ms·
As I wrote already, if an app needs this level of babysitting, fix the code and/or the infrastructure. If having people on-call 24/7 seems normal to management,
by gregjor 4y ago
As I wrote already, if an app needs this level of babysitting, fix the code and/or the infrastructure. If having people on-call 24/7 seems normal to management, they need to step up and address the bugs, failures, downtime, and the lack of redundancy, monitoring, and automation. I would strive to get the support burden down to a very occasional alert on my phone that I can respond to from my phone.
When I see this kind of thing with my customers it comes from a combination of unrealistic expectations ("We need 99.9999% uptime for our marketing sites") and over-engineered and fragile infrastructure, code, and development/deployment processes ("Our K8S/microservices architecture can fail in many ways that no one understands.")
I did some work for a company a few years ago that had a cron job that rebooted the web server every day, because that was the solution they had come up with to "fix" the server crashing (Apache + Rails in that case). The actual problem was running MySQL and Apache on the same server, leading to MySQL eventually starving Apache for memory. Once I fixed that Apache + Rails would run for years without needed a reboot. I had another customer with a server that ran out of disk space every couple of months, requiring frantic resizing of an attached disk. Moved them to S3, problem solved, no more calls.
- lknfrnpt 4y ago"If having people on-call 24/7 seems normal to management, they need to step up and address the bugs, failures, downtime, and the lack of redundancy, monitoring, and automation." "I would strive to get the support burden down to a very occasional alert on my phone that I can respond to from my phone." It's possible we're working on different kinds (or scales) of software or differently sized organizations – there are no human customers actually "calling," nor can you typically resolve true production incidents (let alone root cause them or estimate impact) from a phone.