7 ms·

Using a hardware watchdog is an alternative and it covers the additional case where the system hangs without panicking.

by oakwhiz 4y ago

jerrysievert 4y ago
Yes thank you. As someone who built these for a living I can definitely say that this is a really good solution.
- foobiekr 4y ago
  Just gotta be careful about how you implement it. The real problem to solve is to detect when the application is. It making forward progress. I’ve seen this implemented in awful, stupid ways (like spinning off a thread to stroke the watchdog - borderline pointless).
  - ilyt 4y ago
    That's generally pretty hard problem, to find a point where it triggers where app is unresponsible but not when say it is running something CPU intensive and just lags for few seconds
    - kqr 4y ago
      Isn't that in its most general form exactly the halting problem?
      - foobiekr 4y ago
        The halting problem is for the general case. If you design the application to facilitate it you can demonstrate forward progress in a pretty robust way. Is it perfect? No. But you can do much, much better than “is the kernel up.”
  - londons_explore 4y ago
    For network connected things (which aren't designed to work offline), I like to implement a network watchdog poker... Ie. every 30 seconds, contact an update server server and, if successful, poke the hardware watchdog. That means every device in your network will either be rebooting, or correctly talking to your server with the latest version of your software. There are no other stuck/error states to consider. I also implement reboot-with-no-screen-flicker, so that I can display a basic company logo so things look respectable while there is an outage and everything is bootlooping. Some brands of screen let you upload an image to display if there is no valid signal, which is an easy way to implement this. Also worth doing a load test of your server to verify that it can withstand all your devices rebooting simultaneously and you won't suffer the thundering herd problem.
  - throwaway9870 4y ago
    I think you are conflating hardware watchdogs and software timers. The hardware watchdog should be to catch cases where the system is no longer running your software (or the kernel or the watchdog daemon). Timeouts for software catch things like slow procedures, etc.
    - foobiekr 4y ago
      Not at all. In most embedded devices, the point of the software on the device is some specific function which is something more than “the kernel is running” or “the kernel is scheduling the zero-io watchdog process.” You want to actually pick up the case where the kernel is up but, for example, your process is all but dead because your storage driver has wedged. The goal is to prove forward progress and the best way to do that is to come as close to proving that your userplane SW is actually working and not dead, or worse, half dead.
noodlesUK 4y ago
Could you elaborate a little about how these are implemented?
- jesse_cureton 4y ago
  Preface: my knowledge here is on ARM, particularly baremetal, but also embedded Linux. No idea about Windows or x86. Generally there’s a hardware watchdog implemented as a counter/timer in the processor. It can have a predefined or configurable period. It counts down, and if it times out then it initiates a hardware reset of the processor. You can ensure your software/OS is always at least executing code by having a task (in-kernel on Linux, or an RTOS task, or just in your main event loop on baremetal) that resets that timer. Then, if your code stops resetting that timer, it expires and resets the processor.
  - noodlesUK 4y ago
    Is there some way of accessing this from user space on Linux?
    - amluto 4y ago
      https://www.kernel.org/doc/html/latest/watchdog/watchdog-api.html https://www.kernel.org/doc/html/latest/watchdog/watchdog-api... Many x86 systems have a built in hardware watchdog.
    - jesse_cureton 4y ago
      Yes! There’s an ioctl interface for managing the watchdog, and a character device at /dev/watchdog. The kernel docs[1] are a decent jumping off point to learn more. Upon reading these I did realize on Linux it’s implemented as a kernel device, but it’s usually a userspace task that has to notify the kernel watchdog interface to actually kick the timer. This makes sense, since userspace being functional is probably what you really care about. [1] https://www.kernel.org/doc/html/latest/watchdog/watchdog-api.html https://www.kernel.org/doc/html/latest/watchdog/watchdog-api...
c7DJTLrn 4y ago
Great point. If the hardware is available I'd use both.
dtgriscom 4y ago
I have a network-controllable outlet switch with a timeout feature. You can configure each outlet to ping a specific IP address; if the address doesn't respond after X seconds then the outlet will be power-cycled.
- mananaysiempre 4y ago
  Did you get that as fabulously expensive datacenter gear, or were you able to squeeze it out of a consumer “smart outlet”? I’ve been looking for a thing like that some weeks ago (for reasons that are now moot), and the easiest option seemed to be to buy a handful of the vendor-app-locked whitelabel crap and see which can be reflashed with open source firmware. Which is not as easy as I’d wish.
  - yencabulator 4y ago
    Shelly relays are known to be easy to control via Wifi. Off the shelf, you'd need to run the actual ping+decide logic elsewhere, though you could re-flash the firmware with custom logic too... https://shelly-api-docs.shelly.cloud/gen2/General/RPCProtocol/ https://shelly-api-docs.shelly.cloud/gen2/General/RPCProtoco... https://www.amazon.com/stores/page/4E39C18F-DCA3-4726-A8A7-6FB65C0DCBCA https://www.amazon.com/stores/page/4E39C18F-DCA3-4726-A8A7-6... Plus, they're also easy to reflash with other firmwares (e.g. Tasmota, ESPHome).
    - oakwhiz 4y ago
      I want one of these in "plug" form factor that uses a normally-closed relay rather than normally-open. The reason being that the internal power supply and microcontroller could fail by shutting down, preventing things from working at all. There's nothing that can be done about the microcontroller failing while holding the circuit open, though, but power supply failure has been a reoccurring but infrequent thing for me lately and it would be nice to cover that case well. Edit: For small do-it-yourself computer clusters you probably want it to fail normally-open, this would be for non-redundant stuff
- metadat 4y ago
throwaway9870 4y ago
Exactly. Every Intel machine has a hardware watchdog built into the ICH chipset. Make sure it is turned on.
- metadat 4y ago
  I tried it on my supermicro server and recall it causing problems, so I have to turn it off.
- oakwhiz 4y ago
  Make sure you turn on the watchdog poker before you turn on the watchdog. Otherwise it can be quite annoying to turn it back off.

Using a hardware watchdog is an alternative and it covers the additional case where the system hangs without panicking.

jerrysievert 4y ago

Yes thank you. As someone who built these for a living I can definitely say that this is a really good solution.

foobiekr 4y ago
Just gotta be careful about how you implement it. The real problem to solve is to detect when the application is. It making forward progress. I’ve seen this implemented in awful, stupid ways (like spinning off a thread to stroke the watchdog - borderline pointless).
- ilyt 4y ago
  That's generally pretty hard problem, to find a point where it triggers where app is unresponsible but not when say it is running something CPU intensive and just lags for few seconds
  - kqr 4y ago
    Isn't that in its most general form exactly the halting problem?
    - foobiekr 4y ago
      The halting problem is for the general case. If you design the application to facilitate it you can demonstrate forward progress in a pretty robust way. Is it perfect? No. But you can do much, much better than “is the kernel up.”
- londons_explore 4y ago
  For network connected things (which aren't designed to work offline), I like to implement a network watchdog poker... Ie. every 30 seconds, contact an update server server and, if successful, poke the hardware watchdog. That means every device in your network will either be rebooting, or correctly talking to your server with the latest version of your software. There are no other stuck/error states to consider. I also implement reboot-with-no-screen-flicker, so that I can display a basic company logo so things look respectable while there is an outage and everything is bootlooping. Some brands of screen let you upload an image to display if there is no valid signal, which is an easy way to implement this. Also worth doing a load test of your server to verify that it can withstand all your devices rebooting simultaneously and you won't suffer the thundering herd problem.
- throwaway9870 4y ago
  I think you are conflating hardware watchdogs and software timers. The hardware watchdog should be to catch cases where the system is no longer running your software (or the kernel or the watchdog daemon). Timeouts for software catch things like slow procedures, etc.
  - foobiekr 4y ago
    Not at all. In most embedded devices, the point of the software on the device is some specific function which is something more than “the kernel is running” or “the kernel is scheduling the zero-io watchdog process.” You want to actually pick up the case where the kernel is up but, for example, your process is all but dead because your storage driver has wedged. The goal is to prove forward progress and the best way to do that is to come as close to proving that your userplane SW is actually working and not dead, or worse, half dead.

noodlesUK 4y ago

Could you elaborate a little about how these are implemented?

jesse_cureton 4y ago
Preface: my knowledge here is on ARM, particularly baremetal, but also embedded Linux. No idea about Windows or x86. Generally there’s a hardware watchdog implemented as a counter/timer in the processor. It can have a predefined or configurable period. It counts down, and if it times out then it initiates a hardware reset of the processor. You can ensure your software/OS is always at least executing code by having a task (in-kernel on Linux, or an RTOS task, or just in your main event loop on baremetal) that resets that timer. Then, if your code stops resetting that timer, it expires and resets the processor.
- noodlesUK 4y ago
  Is there some way of accessing this from user space on Linux?
  - amluto 4y ago
    https://www.kernel.org/doc/html/latest/watchdog/watchdog-api.html https://www.kernel.org/doc/html/latest/watchdog/watchdog-api... Many x86 systems have a built in hardware watchdog.
  - jesse_cureton 4y ago
    Yes! There’s an ioctl interface for managing the watchdog, and a character device at /dev/watchdog. The kernel docs[1] are a decent jumping off point to learn more. Upon reading these I did realize on Linux it’s implemented as a kernel device, but it’s usually a userspace task that has to notify the kernel watchdog interface to actually kick the timer. This makes sense, since userspace being functional is probably what you really care about. [1] https://www.kernel.org/doc/html/latest/watchdog/watchdog-api.html https://www.kernel.org/doc/html/latest/watchdog/watchdog-api...

c7DJTLrn 4y ago

Great point. If the hardware is available I'd use both.

dtgriscom 4y ago

I have a network-controllable outlet switch with a timeout feature. You can configure each outlet to ping a specific IP address; if the address doesn't respond after X seconds then the outlet will be power-cycled.

mananaysiempre 4y ago
Did you get that as fabulously expensive datacenter gear, or were you able to squeeze it out of a consumer “smart outlet”? I’ve been looking for a thing like that some weeks ago (for reasons that are now moot), and the easiest option seemed to be to buy a handful of the vendor-app-locked whitelabel crap and see which can be reflashed with open source firmware. Which is not as easy as I’d wish.
- yencabulator 4y ago
  Shelly relays are known to be easy to control via Wifi. Off the shelf, you'd need to run the actual ping+decide logic elsewhere, though you could re-flash the firmware with custom logic too... https://shelly-api-docs.shelly.cloud/gen2/General/RPCProtocol/ https://shelly-api-docs.shelly.cloud/gen2/General/RPCProtoco... https://www.amazon.com/stores/page/4E39C18F-DCA3-4726-A8A7-6FB65C0DCBCA https://www.amazon.com/stores/page/4E39C18F-DCA3-4726-A8A7-6... Plus, they're also easy to reflash with other firmwares (e.g. Tasmota, ESPHome).
  - oakwhiz 4y ago
    I want one of these in "plug" form factor that uses a normally-closed relay rather than normally-open. The reason being that the internal power supply and microcontroller could fail by shutting down, preventing things from working at all. There's nothing that can be done about the microcontroller failing while holding the circuit open, though, but power supply failure has been a reoccurring but infrequent thing for me lately and it would be nice to cover that case well. Edit: For small do-it-yourself computer clusters you probably want it to fail normally-open, this would be for non-redundant stuff
metadat 4y ago

throwaway9870 4y ago

Exactly. Every Intel machine has a hardware watchdog built into the ICH chipset. Make sure it is turned on.

metadat 4y ago
I tried it on my supermicro server and recall it causing problems, so I have to turn it off.
oakwhiz 4y ago
Make sure you turn on the watchdog poker before you turn on the watchdog. Otherwise it can be quite annoying to turn it back off.