3 ms·

This is just as true without systemd. Auto-restarting services without monitoring will always hide problems, regardless of what’s doing the restarting. This is

by hnbear 3y ago

This is just as true without systemd. Auto-restarting services without monitoring will always hide problems, regardless of what’s doing the restarting. This is just another reason to track and alert on error rates. It also depends how sensitive you are to failures, and required availability. But in general this sounds more like: “you need observability on error rate”, more than anything else, and that includes notification and alerting on set thresholds that seem aberrant.

yxre 3y ago
Observability is so key for successful infrastructure. A poor infrastructure with good observability is going to be more successful with than good infrastructure and no observability
throwaway073123 3y ago
Auto restarting is actually intended to hide problems! ...from the user. The user, who probably isn't you if you're developing or administrating a service. Any supervisory process, systemd, supervisord, kubernetes, etc should absolutely be making those restarts visible to the administrator so that they can resolve it. The restart is just in hopes to keep the service available (symptoms in check) until the problem is actually fixed (disease is cured).
outworlder 3y ago
> This is just as true without systemd. Auto-restarting services without monitoring will always hide problems, regardless of what’s doing the restarting. And we should be thankful. We don't have to be aware of every single hiccup a service experiences. Even if (and specially if) you are the one responsible for keeping it up. Modern distributed systems are far too complex for us to care about every minutia. If something failed over, and nobody noticed, is it a problem? The answer is _maybe_. How often does that happen? Is there a pattern? Is it getting more frequent? Is that happening more often than predicted? At work, we blow up entire VMs if they fail their health checks. They can fail for many reasons, mostly uninteresting ones. And the customers don't even notice and SRE doesn't usually care. They only become relevant once there are anomalies. When you are managing thousands of instances, self-healing is required. Error rate may not even be impacted in a significant way when those systems restart, it is often a tiny increase in a deluge of requests. So you also need observability on self-healing events to catch trends. When self-healing fails, it tends to do so pretty catastrophically. > and alerting on set thresholds that seem aberrant. I wish we would remove 'thresholds' from our alerting vocabulary. Very often we end up setting simple and completely arbitrary thresholds that don't actually mean much - unless it's based on SLOs and SLAs. Generally we don't know what those thresholds are supposed to be and just guess, unless it's capacity. But even for capacity: 0% disk space free is obviously bad; what people will do is set some threshold like ("alert when 80% disk is used"). Then you get page once that threshold is crossed. Is that an emergency? I don't know, maybe it took 5 years to get there. However, if you set an alert that says "Alert me if the disk will fill up in <X> days at the current rate", you can then tell if it is an emergency or not. I suspect that you would agree with this given the use of the word "aberrant", which to me implies anomalies. But in many contexts, people think that "alert every single time CPU utilization crosses 90%" is what we are talking about.
- nemothekid 3y ago
  >However, if you set an alert that says "Alert me if the disk will fill up in <X> days at the current rate", you can then tell if it is an emergency or not. Somehow I've never come across this before, and it's really unlocked something for me with how I will now approach alerts
  - hnbear 3y ago
    Using derivative functions in alerts to look at speed or acceleration is a game changer. You really care about time to fill (speed), and whether your system has gone nuts and is logging faster and faster than normal (acceleration). Disk usage is X says do little. Disk utilization rate is Y therefore will fill by YYYYMMDD is great. Disk utilization rate change has gone off the chart so a process crapped itself and will fill the disk in 5 mins is life saving.
- hnbear 3y ago
  Yes. I mean anomaly detection based on identifying what’s unusual behavior not just typical. We have computers to use their CPUs, RAM, and disk. Using it isn’t weird. Detecting actual unwanted or dangerous behavior is bad. For bonus points auto-remediate it (eg take the node offline so the workload moves to another node) and alert when your overall capacity of your service is potentially going to be impacted or is near a limit, for some version of near. Being human I much prefer “you have hours to fix this” than “I’m broken fix me”. RED, Golden Signals, etc. there are plenty of nice frameworks for how to think about utilization and performance of your service that you can use to manage capacity and health. CPU at 90 is not one of them. As soon as someone suggests it I always ask for how long? On which processes? on which boxes? What is the “perfect” CPU utilization? Then make them think about why we wouldn’t want to use the resources we paid for.
- eternityforest 3y ago
  I like to investigate every random nuisance error and restart, if I have nothing else to do. Otherwise, any system simple enough to be 100% but free without millions of dollars of work and avionics level formal proofs, might be a system that could probably be replaced with a yellow legal pad and a phone call and might not even need to exist.
paulddraper 3y ago
Exactly. Do you want one memory allocation error to take everything down? Of course not. Log it, and restart.
m463 3y ago
What I hate is that everything with systemd requires software it access (instead of plain text log files). It would be nice if there was a way to configure log file type/format, like "ICanHazPlainLogs=yespls"

by hnbear 3y ago

yxre 3y ago

Observability is so key for successful infrastructure. A poor infrastructure with good observability is going to be more successful with than good infrastructure and no observability

throwaway073123 3y ago

Auto restarting is actually intended to hide problems! ...from the user. The user, who probably isn't you if you're developing or administrating a service. Any supervisory process, systemd, supervisord, kubernetes, etc should absolutely be making those restarts visible to the administrator so that they can resolve it. The restart is just in hopes to keep the service available (symptoms in check) until the problem is actually fixed (disease is cured).

outworlder 3y ago

> This is just as true without systemd. Auto-restarting services without monitoring will always hide problems, regardless of what’s doing the restarting. And we should be thankful. We don't have to be aware of every single hiccup a service experiences. Even if (and specially if) you are the one responsible for keeping it up. Modern distributed systems are far too complex for us to care about every minutia. If something failed over, and nobody noticed, is it a problem? The answer is _maybe_. How often does that happen? Is there a pattern? Is it getting more frequent? Is that happening more often than predicted? At work, we blow up entire VMs if they fail their health checks. They can fail for many reasons, mostly uninteresting ones. And the customers don't even notice and SRE doesn't usually care. They only become relevant once there are anomalies. When you are managing thousands of instances, self-healing is required. Error rate may not even be impacted in a significant way when those systems restart, it is often a tiny increase in a deluge of requests. So you also need observability on self-healing events to catch trends. When self-healing fails, it tends to do so pretty catastrophically. > and alerting on set thresholds that seem aberrant. I wish we would remove 'thresholds' from our alerting vocabulary. Very often we end up setting simple and completely arbitrary thresholds that don't actually mean much - unless it's based on SLOs and SLAs. Generally we don't know what those thresholds are supposed to be and just guess, unless it's capacity. But even for capacity: 0% disk space free is obviously bad; what people will do is set some threshold like ("alert when 80% disk is used"). Then you get page once that threshold is crossed. Is that an emergency? I don't know, maybe it took 5 years to get there. However, if you set an alert that says "Alert me if the disk will fill up in <X> days at the current rate", you can then tell if it is an emergency or not. I suspect that you would agree with this given the use of the word "aberrant", which to me implies anomalies. But in many contexts, people think that "alert every single time CPU utilization crosses 90%" is what we are talking about.

nemothekid 3y ago
>However, if you set an alert that says "Alert me if the disk will fill up in <X> days at the current rate", you can then tell if it is an emergency or not. Somehow I've never come across this before, and it's really unlocked something for me with how I will now approach alerts
- hnbear 3y ago
  Using derivative functions in alerts to look at speed or acceleration is a game changer. You really care about time to fill (speed), and whether your system has gone nuts and is logging faster and faster than normal (acceleration). Disk usage is X says do little. Disk utilization rate is Y therefore will fill by YYYYMMDD is great. Disk utilization rate change has gone off the chart so a process crapped itself and will fill the disk in 5 mins is life saving.
hnbear 3y ago
Yes. I mean anomaly detection based on identifying what’s unusual behavior not just typical. We have computers to use their CPUs, RAM, and disk. Using it isn’t weird. Detecting actual unwanted or dangerous behavior is bad. For bonus points auto-remediate it (eg take the node offline so the workload moves to another node) and alert when your overall capacity of your service is potentially going to be impacted or is near a limit, for some version of near. Being human I much prefer “you have hours to fix this” than “I’m broken fix me”. RED, Golden Signals, etc. there are plenty of nice frameworks for how to think about utilization and performance of your service that you can use to manage capacity and health. CPU at 90 is not one of them. As soon as someone suggests it I always ask for how long? On which processes? on which boxes? What is the “perfect” CPU utilization? Then make them think about why we wouldn’t want to use the resources we paid for.
eternityforest 3y ago
I like to investigate every random nuisance error and restart, if I have nothing else to do. Otherwise, any system simple enough to be 100% but free without millions of dollars of work and avionics level formal proofs, might be a system that could probably be replaced with a yellow legal pad and a phone call and might not even need to exist.

paulddraper 3y ago

Exactly. Do you want one memory allocation error to take everything down? Of course not. Log it, and restart.

m463 3y ago

What I hate is that everything with systemd requires software it access (instead of plain text log files). It would be nice if there was a way to configure log file type/format, like "ICanHazPlainLogs=yespls"