3 ms·
> Of course, at some point, [...] the local service gets restarted by the ops team (because it can't self-heal, naturally) Maybe off-topic, but what are some g
by dgritsko 7y ago
> Of course, at some point, [...] the local service gets restarted by the ops team (because it can't self-heal, naturally)
Maybe off-topic, but what are some good strategies for the kind of "self-healing" being talked about here? If a service needs to be restarted, how could you automate the detection and restart process?
- perlgeek 7y agoIn the simplest case, the service could shut itself down, and the supervising daemon / scheduler would restart it. Supervisors like systemd also have a watchdog that will force-restart a service that hasn't checked in for some time. For a service that manages its own network connection, implementing auto-reconnect can be a form of self-healing (and surprisingly hard to get right in all edge cases). The key is, as Rachel wrote in the OP, to get a good signal. You need to be able to distinguish a working from a non-working service to implement reliable self-healing.
- dgritsko 7y ago> You need to be able to distinguish a working from a non-working service to implement reliable self-healing. I think this is the crux of what I was trying to get at. Curious to read how others have approached this problem.