12 ms·
> This is just as true without systemd. Auto-restarting services without monitoring will always hide problems, regardless of what’s doing the restarting. And
by outworlder 3y ago
> This is just as true without systemd. Auto-restarting services without monitoring will always hide problems, regardless of what’s doing the restarting.
And we should be thankful. We don't have to be aware of every single hiccup a service experiences. Even if (and specially if) you are the one responsible for keeping it up. Modern distributed systems are far too complex for us to care about every minutia.
If something failed over, and nobody noticed, is it a problem? The answer is _maybe_. How often does that happen? Is there a pattern? Is it getting more frequent? Is that happening more often than predicted?
At work, we blow up entire VMs if they fail their health checks. They can fail for many reasons, mostly uninteresting ones. And the customers don't even notice and SRE doesn't usually care. They only become relevant once there are anomalies. When you are managing thousands of instances, self-healing is required.
Error rate may not even be impacted in a significant way when those systems restart, it is often a tiny increase in a deluge of requests. So you also need observability on self-healing events to catch trends. When self-healing fails, it tends to do so pretty catastrophically.
> and alerting on set thresholds that seem aberrant.
I wish we would remove 'thresholds' from our alerting vocabulary. Very often we end up setting simple and completely arbitrary thresholds that don't actually mean much - unless it's based on SLOs and SLAs. Generally we don't know what those thresholds are supposed to be and just guess, unless it's capacity. But even for capacity: 0% disk space free is obviously bad; what people will do is set some threshold like ("alert when 80% disk is used"). Then you get page once that threshold is crossed. Is that an emergency? I don't know, maybe it took 5 years to get there. However, if you set an alert that says "Alert me if the disk will fill up in <X> days at the current rate", you can then tell if it is an emergency or not.
I suspect that you would agree with this given the use of the word "aberrant", which to me implies anomalies. But in many contexts, people think that "alert every single time CPU utilization crosses 90%" is what we are talking about.
- nemothekid 3y ago>However, if you set an alert that says "Alert me if the disk will fill up in <X> days at the current rate", you can then tell if it is an emergency or not. Somehow I've never come across this before, and it's really unlocked something for me with how I will now approach alerts
- hnbear 3y agoUsing derivative functions in alerts to look at speed or acceleration is a game changer. You really care about time to fill (speed), and whether your system has gone nuts and is logging faster and faster than normal (acceleration). Disk usage is X says do little. Disk utilization rate is Y therefore will fill by YYYYMMDD is great. Disk utilization rate change has gone off the chart so a process crapped itself and will fill the disk in 5 mins is life saving.
- hnbear 3y agoYes. I mean anomaly detection based on identifying what’s unusual behavior not just typical. We have computers to use their CPUs, RAM, and disk. Using it isn’t weird. Detecting actual unwanted or dangerous behavior is bad. For bonus points auto-remediate it (eg take the node offline so the workload moves to another node) and alert when your overall capacity of your service is potentially going to be impacted or is near a limit, for some version of near. Being human I much prefer “you have hours to fix this” than “I’m broken fix me”. RED, Golden Signals, etc. there are plenty of nice frameworks for how to think about utilization and performance of your service that you can use to manage capacity and health. CPU at 90 is not one of them. As soon as someone suggests it I always ask for how long? On which processes? on which boxes? What is the “perfect” CPU utilization? Then make them think about why we wouldn’t want to use the resources we paid for.
- eternityforest 3y agoI like to investigate every random nuisance error and restart, if I have nothing else to do. Otherwise, any system simple enough to be 100% but free without millions of dollars of work and avionics level formal proofs, might be a system that could probably be replaced with a yellow legal pad and a phone call and might not even need to exist.