4 ms·
> Whether you want to monitor for this sort of thing (and how) is an open question. It's certainly possible that this is one of the times where your monitoring
by evil-olive 3y ago
> Whether you want to monitor for this sort of thing (and how) is an open question. It's certainly possible that this is one of the times where your monitoring isn't going to be comprehensive, because it's infrequent enough, low impact enough, and hard enough to craft a specific alert.
my preferred way to alert on this is to use the process start time that's included by default in most process-level Prometheus metrics (and is trivial to implement yourself, if you need to):
> curl -s localhost:9100/metrics | grep process_start_time_seconds
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.68957669109e+09
on a stable system, this metric will be very close to static. you can feed it through the PromQL changes() function to get a count of how many restarts have happened in a given time window.
in my experience, for anything with an "alert if it's down for X minutes" rule, you probably also want an "alert if it restarts N times in Y minutes" rule.
- captn3m0 3y agoA relevant alternative is the systemd_exporter, which will export metrics for various systemd services, including restart count: https://github.com/prometheus-community/systemd_exporter#metrics https://github.com/prometheus-community/systemd_exporter#met...
- tetha 3y agoMh, good points. I'll have to take a look to include a restart count in our zabbix systemd template. Should be a cheap metric to track with deduplication but valuable to create tickets upon if it changes too much in a short timeframe.
- vidarh 3y agoA whole lot of alerting is made better by generally considering whether to alert on large changes in in almost any kind of event, and sometimes even by measuring the changes in the rate of change, because it has a better chance of surviving "alert fatigue". "That process that restarts spuriously once in a while has restarted again" stops being noteworthy very quickly if it's deemed low priority, and often even might lead to alerting thresholds being change because it becomes a nuisance. "That process that restarted every now and again now restarts 5 time as often as it used to" on the other hand is a lot more likely to get attention.