3 ms·

Nowadays this watchdog is init/PID1, on most distributions - systemd. If init can't be relied on to manage services, what guarantees do you have for the system

Nowadays this watchdog is init/PID1, on most distributions - systemd. If init can't be relied on to manage services, what guarantees do you have for the system to provide them? Sure, one could reinvent this in scripts, but we've moved past this. I mention systemd a lot, but that's not to cast favor - there are alternatives. Most services don't make appropriate use of the environment they exist in. I assume they expect some site customization, ie: declaring your web server needs these mounts. A commonly overlooked directive is 'PartOf='. You can tie restarts of one service/resource to another. Heck, more simply, I think NetworkManager offers a way to customize the wifi/portal checking. You may not have to go completely heavy-handed

smashed 3y ago
Just a fyi if you're going down that path, the hardware watchdog is disabled by default in most distros. Enable it using RuntimeWatchdogSec. Second, if you are running an important service, see if it can support systemd notification socket and even better the software watchdog protocol with systemd. The service just has to send a heart-beat every X seconds otherwise systemd will restart it. This ties in the hw watchdog to systemd and systemd watches your service, ensuring both the hardware and the software is running, otherwise the system will restart. Lots of details here: http://0pointer.de/blog/projects/watchdog.html http://0pointer.de/blog/projects/watchdog.html None of this is raspberry pi specific, it works on every system with a supported hardware watchdog, which includes raspberry pis.
hn8305823 3y ago
> Nowadays this watchdog is init/PID1, on most distributions - systemd. If init can't be relied on to manage services, what guarantees do you have for the system to provide them? Does that monitor actual (not just apparent link up) network connectivity or the ability to access remote networks? Not all failure modes are local to OS processes and sometimes rebooting fixes things like this. process-specific monitoring can be quite useful but so is a generic "I can't reach my server I just want it to reboot now" with minimal complexity/dependencies. I can't imagine all of the failure modes where a simple bash watchdog script would help - and that's the point! Now, if you have OOB access to the server (like in a data center) then you might not need or even want a watchdog script. But for remote and/or difficult to access servers (I refer to these as "mars probes"), they can be a life saver.

smashed 3y ago

Just a fyi if you're going down that path, the hardware watchdog is disabled by default in most distros. Enable it using RuntimeWatchdogSec. Second, if you are running an important service, see if it can support systemd notification socket and even better the software watchdog protocol with systemd. The service just has to send a heart-beat every X seconds otherwise systemd will restart it. This ties in the hw watchdog to systemd and systemd watches your service, ensuring both the hardware and the software is running, otherwise the system will restart. Lots of details here: http://0pointer.de/blog/projects/watchdog.html http://0pointer.de/blog/projects/watchdog.html None of this is raspberry pi specific, it works on every system with a supported hardware watchdog, which includes raspberry pis.

hn8305823 3y ago

> Nowadays this watchdog is init/PID1, on most distributions - systemd. If init can't be relied on to manage services, what guarantees do you have for the system to provide them? Does that monitor actual (not just apparent link up) network connectivity or the ability to access remote networks? Not all failure modes are local to OS processes and sometimes rebooting fixes things like this. process-specific monitoring can be quite useful but so is a generic "I can't reach my server I just want it to reboot now" with minimal complexity/dependencies. I can't imagine all of the failure modes where a simple bash watchdog script would help - and that's the point! Now, if you have OOB access to the server (like in a data center) then you might not need or even want a watchdog script. But for remote and/or difficult to access servers (I refer to these as "mars probes"), they can be a life saver.