3 ms·

> This feels like a hack, but based on hours of reading online discussions, most people seem to settle on a script that periodically checks whether the WiFi con

> This feels like a hack, but based on hours of reading online discussions, most people seem to settle on a script that periodically checks whether the WiFi connection is good, and restarts the WiFi interface or the whole Pi if it’s not. It's not a hack, it's best practice! Just like important servers in a data center should have some kind of out of band connectivity (IPMI, remote controllable RPDU outlets, etc), Important servers in remote difficult to reach locations should have some kind of watchdog script. The script should of course be tuned to the specific use case, considering the impact of a reboot vs downtime until reboot. At the very least it could log adverse events for later investigation. A simple bash watchdog script was the very first thing I did when I deployed a remote RPI. Not just for wi-fi issues but for any of the dozens of things that could break and be fixed with a reboot.

vb6sp6 3y ago
[dead]
slicktux 3y ago
Totally agree! Watchdog timers are essential for microcontroller and even computers running software “forever”. Things do happen that even perfect code and design can’t prevent and a watchdog timer will break out of an infinite loop and reset. Things such as cosmic rays flipping a bit or even brownouts…in raspberry pi you also have to worry about SD card corruption… Edit: raspberry pi’s have built-in hardware watchdog timers I believe. I know Arduino's do!
bravetraveler 3y ago
Nowadays this watchdog is init/PID1, on most distributions - systemd. If init can't be relied on to manage services, what guarantees do you have for the system to provide them? Sure, one could reinvent this in scripts, but we've moved past this. I mention systemd a lot, but that's not to cast favor - there are alternatives. Most services don't make appropriate use of the environment they exist in. I assume they expect some site customization, ie: declaring your web server needs these mounts. A commonly overlooked directive is 'PartOf='. You can tie restarts of one service/resource to another. Heck, more simply, I think NetworkManager offers a way to customize the wifi/portal checking. You may not have to go completely heavy-handed
- smashed 3y ago
  Just a fyi if you're going down that path, the hardware watchdog is disabled by default in most distros. Enable it using RuntimeWatchdogSec. Second, if you are running an important service, see if it can support systemd notification socket and even better the software watchdog protocol with systemd. The service just has to send a heart-beat every X seconds otherwise systemd will restart it. This ties in the hw watchdog to systemd and systemd watches your service, ensuring both the hardware and the software is running, otherwise the system will restart. Lots of details here: http://0pointer.de/blog/projects/watchdog.html http://0pointer.de/blog/projects/watchdog.html None of this is raspberry pi specific, it works on every system with a supported hardware watchdog, which includes raspberry pis.
- hn8305823 3y ago
  > Nowadays this watchdog is init/PID1, on most distributions - systemd. If init can't be relied on to manage services, what guarantees do you have for the system to provide them? Does that monitor actual (not just apparent link up) network connectivity or the ability to access remote networks? Not all failure modes are local to OS processes and sometimes rebooting fixes things like this. process-specific monitoring can be quite useful but so is a generic "I can't reach my server I just want it to reboot now" with minimal complexity/dependencies. I can't imagine all of the failure modes where a simple bash watchdog script would help - and that's the point! Now, if you have OOB access to the server (like in a data center) then you might not need or even want a watchdog script. But for remote and/or difficult to access servers (I refer to these as "mars probes"), they can be a life saver.
IshKebab 3y ago
It's best practice and a hack. It shouldn't be required, but bugs exist so it is.
myself248 3y ago
Similarly, I use an ESP8266 to supervise my wifi router and cable modem. Any trouble, reboot! For the router, it just tries to connect to the appropriate SSID, and then tries to ping the router, and if either of those fails, it swaps to the other router. I have two identical routers with identical configs, with their power connected to the NO and NC contacts of an SPDT relay. If one fails, it just toggles the relay state to switch to the other. If the router is up, the watchdog tries to load the cable modem's status page, and tries to ping any of three different IPs I've identified within my ISP's network which seem to be either the CMTS or closely associated hardware, and should indicate aliveness of the HFC plant -- I don't want to bother rebooting if the failure isn't something that a reboot could solve. Sadly I haven't figured out how to have two cable modems with the same MAC such that I could swap between them too, and the ISP won't let me have two modems on the same account, so my only resort if the CM fails is to reboot it and hope for the best. This, plus a rack of batteries that'll keep the router and modem running for 30-plus hours in a utility outage, has kept me online nearly-continuously since May of 2020 when I built it. The code is an absolute horror-show (I'm much better with a soldering iron than an arduino), but in practice it's been rock solid.

vb6sp6 3y ago

[dead]

slicktux 3y ago

Totally agree! Watchdog timers are essential for microcontroller and even computers running software “forever”. Things do happen that even perfect code and design can’t prevent and a watchdog timer will break out of an infinite loop and reset. Things such as cosmic rays flipping a bit or even brownouts…in raspberry pi you also have to worry about SD card corruption… Edit: raspberry pi’s have built-in hardware watchdog timers I believe. I know Arduino's do!

bravetraveler 3y ago

Nowadays this watchdog is init/PID1, on most distributions - systemd. If init can't be relied on to manage services, what guarantees do you have for the system to provide them? Sure, one could reinvent this in scripts, but we've moved past this. I mention systemd a lot, but that's not to cast favor - there are alternatives. Most services don't make appropriate use of the environment they exist in. I assume they expect some site customization, ie: declaring your web server needs these mounts. A commonly overlooked directive is 'PartOf='. You can tie restarts of one service/resource to another. Heck, more simply, I think NetworkManager offers a way to customize the wifi/portal checking. You may not have to go completely heavy-handed

smashed 3y ago
Just a fyi if you're going down that path, the hardware watchdog is disabled by default in most distros. Enable it using RuntimeWatchdogSec. Second, if you are running an important service, see if it can support systemd notification socket and even better the software watchdog protocol with systemd. The service just has to send a heart-beat every X seconds otherwise systemd will restart it. This ties in the hw watchdog to systemd and systemd watches your service, ensuring both the hardware and the software is running, otherwise the system will restart. Lots of details here: http://0pointer.de/blog/projects/watchdog.html http://0pointer.de/blog/projects/watchdog.html None of this is raspberry pi specific, it works on every system with a supported hardware watchdog, which includes raspberry pis.
hn8305823 3y ago
> Nowadays this watchdog is init/PID1, on most distributions - systemd. If init can't be relied on to manage services, what guarantees do you have for the system to provide them? Does that monitor actual (not just apparent link up) network connectivity or the ability to access remote networks? Not all failure modes are local to OS processes and sometimes rebooting fixes things like this. process-specific monitoring can be quite useful but so is a generic "I can't reach my server I just want it to reboot now" with minimal complexity/dependencies. I can't imagine all of the failure modes where a simple bash watchdog script would help - and that's the point! Now, if you have OOB access to the server (like in a data center) then you might not need or even want a watchdog script. But for remote and/or difficult to access servers (I refer to these as "mars probes"), they can be a life saver.

IshKebab 3y ago

It's best practice and a hack. It shouldn't be required, but bugs exist so it is.

myself248 3y ago

Similarly, I use an ESP8266 to supervise my wifi router and cable modem. Any trouble, reboot! For the router, it just tries to connect to the appropriate SSID, and then tries to ping the router, and if either of those fails, it swaps to the other router. I have two identical routers with identical configs, with their power connected to the NO and NC contacts of an SPDT relay. If one fails, it just toggles the relay state to switch to the other. If the router is up, the watchdog tries to load the cable modem's status page, and tries to ping any of three different IPs I've identified within my ISP's network which seem to be either the CMTS or closely associated hardware, and should indicate aliveness of the HFC plant -- I don't want to bother rebooting if the failure isn't something that a reboot could solve. Sadly I haven't figured out how to have two cable modems with the same MAC such that I could swap between them too, and the ISP won't let me have two modems on the same account, so my only resort if the CM fails is to reboot it and hope for the best. This, plus a rack of batteries that'll keep the router and modem running for 30-plus hours in a utility outage, has kept me online nearly-continuously since May of 2020 when I built it. The code is an absolute horror-show (I'm much better with a soldering iron than an arduino), but in practice it's been rock solid.