3 ms·
I and another person are taking over an old batch processing system at work. The system itself is fairly stable, but interacts with some other systems which kee
by l0b0 5y ago
I and another person are taking over an old batch processing system at work. The system itself is fairly stable, but interacts with some other systems which keep falling over or changing things without adequate notification. Currently there are (mostly false) alarms almost every day, usually for stupid reasons like the network falling over for a few minutes or disk use going above a fairly arbitrary level. So when the subject of weekend on-call came up we just went hard "no", and we've been discussing solutions to that ever since (adjusting cron scheduling, making alarms more forgiving, adjusting customer expectations, improving automation, etc).
Importantly, we can do this because we live somewhere with employee protection, and our contracts do not mention being on call or working weekends. If you don't have that you're pretty much SOL.
- physicles 5y agoYou probably know this already, but getting rid of those false alarms is absolutely critical. Instead of alerting on disk use, alert on user-facing metrics when possible (unless it’s disk space going below a certain threshold, since that’s one of the worst things that can happen).
- l0b0 5y agoProblem is, the only user facing metric is basically "Did you get the last week's updates?", which currently is shipped on Friday and the user usually notices on Monday.