5 ms·

Could you elaborate a little about how these are implemented?

jesse_cureton 4y ago
Preface: my knowledge here is on ARM, particularly baremetal, but also embedded Linux. No idea about Windows or x86. Generally there’s a hardware watchdog implemented as a counter/timer in the processor. It can have a predefined or configurable period. It counts down, and if it times out then it initiates a hardware reset of the processor. You can ensure your software/OS is always at least executing code by having a task (in-kernel on Linux, or an RTOS task, or just in your main event loop on baremetal) that resets that timer. Then, if your code stops resetting that timer, it expires and resets the processor.
- noodlesUK 4y ago
  Is there some way of accessing this from user space on Linux?
  - amluto 4y ago
    https://www.kernel.org/doc/html/latest/watchdog/watchdog-api.html https://www.kernel.org/doc/html/latest/watchdog/watchdog-api... Many x86 systems have a built in hardware watchdog.
  - jesse_cureton 4y ago
    Yes! There’s an ioctl interface for managing the watchdog, and a character device at /dev/watchdog. The kernel docs[1] are a decent jumping off point to learn more. Upon reading these I did realize on Linux it’s implemented as a kernel device, but it’s usually a userspace task that has to notify the kernel watchdog interface to actually kick the timer. This makes sense, since userspace being functional is probably what you really care about. [1] https://www.kernel.org/doc/html/latest/watchdog/watchdog-api.html https://www.kernel.org/doc/html/latest/watchdog/watchdog-api...
    - touisteur 4y ago
      One of the preternal problems of such hardware watchdogs was the inability to discriminate whether a sudden reboot was due a reset-button, hw security (e.g. temperature), ECC problem, or (micro) loss of power, or HW watchdog. On most IPMI-capable BIOS/firmware there's now (been for 10 years but I'm old) an option to log 'system' events (ipmi failures like fan speeds if you've set threshold, but also reboot reasons). It's call the System Event Log. Very useful. And on IPMI-plugged watchdogs, you can also see the state of the HW watchdog (is it running, how many seconds are left). Very useful too.
  - AnssiH 4y ago
    In addition to those already mentioned, one way is to enable it in systemd: # /etc/systemd/system.conf.d/foobar.conf [Manager] RuntimeWatchdogSec=60 When used in this manner, if systemd fails to ping the watchdog for 60 seconds, the system resets. https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html https://www.freedesktop.org/software/systemd/man/systemd-sys... Somewhat related, nowadays by default systemd enables a 10-minute watchdog just before a regular reboot (i.e. after everything has been shut down) to ensure the reboot happens even if there is a hang for some kernel/HW reason.
- ilyt 4y ago
  There is a caveat here, it won't stop your app from crashing before the watchdog activation. Some CPUs have fuse that can enable watchdog before any code starts running but the ARMs I played with (STM32) don't appear to have that option.
  - shabble 4y ago
    At least some STM32s do, see page 89 of the STM32F4xx reference manual[1], the option bits 5:7 at 0x1fffc000 let you activate the hardware watchdog immediately following reset if you wish. [1] https://www.st.com/resource/en/reference_manual/rm0090-stm32f405415-stm32f407417-stm32f427437-and-stm32f429439-advanced-armbased-32bit-mcus-stmicroelectronics.pdf https://www.st.com/resource/en/reference_manual/rm0090-stm32...
- shabble 4y ago
  A more specialised variant that's also quite common is the "window watchdog" peripheral, which is similar to the timer version, but will also trigger a reset if the keep-alive signal arrives too early, as well as too late. It can be useful where you've got a mainloop doing some very predictably timed activities, and allows detection of faults which cause your watchdog servicing to occur too frequently. I think it's quite common in DSP and things like motor control, where you often have hard realtime requirements and things happening too soon is just as bad as too late.
zh3 4y ago
A true hardware watchdog is separate electronics. For example, it's really easy to make an electronic circuit that operates a relay once every 5 minutes, say, unless a 'restart timer' button is pressed. Connect the relay contacts across the PC reset button, and run a program on the PC that 'presses' (electronically) the 'restart timer' button once a minute, say. Then if the PC fails to boot, it gets reset once every 5 minutes until it does (fairly obviously, another set of relay contacts can be used to trigger an alerting device, e.g. wailing sirens). We use this sort of approach with diskless systems in particular. If there's a power cut, the first boot attempt after power restoration might not work (because the network isn't back up yet). So the diskless systems just sit there, continually attempting a network boot until successfu (at which point the software on the PC hits the 'restart timer' button periodicaly. This is closely related to the concept of a "deadman's handle", for example train drivers who must keep a lever pressed down during operation - if it's released, the train stops automatically.
- deleted 4y ago
  [deleted]
dtgriscom 4y ago
Embedded micros (Systems On a Chip) often include dedicated watchdog hardware. This is a timer which is reset ("fed") by writing values to a specific register. Crucially, it often isn't just one value; you have to alternate between two values. That way, you can write one value at one point in your event loop and the other value at another point, making it less likely that something will break but keep feeding the watchdog. If the watchdog hasn't been fed for X milliseconds, then it resets the system.

Preface: my knowledge here is on ARM, particularly baremetal, but also embedded Linux. No idea about Windows or x86. Generally there’s a hardware watchdog implemented as a counter/timer in the processor. It can have a predefined or configurable period. It counts down, and if it times out then it initiates a hardware reset of the processor. You can ensure your software/OS is always at least executing code by having a task (in-kernel on Linux, or an RTOS task, or just in your main event loop on baremetal) that resets that timer. Then, if your code stops resetting that timer, it expires and resets the processor.

noodlesUK 4y ago
Is there some way of accessing this from user space on Linux?
- amluto 4y ago
  https://www.kernel.org/doc/html/latest/watchdog/watchdog-api.html https://www.kernel.org/doc/html/latest/watchdog/watchdog-api... Many x86 systems have a built in hardware watchdog.
- jesse_cureton 4y ago
  Yes! There’s an ioctl interface for managing the watchdog, and a character device at /dev/watchdog. The kernel docs[1] are a decent jumping off point to learn more. Upon reading these I did realize on Linux it’s implemented as a kernel device, but it’s usually a userspace task that has to notify the kernel watchdog interface to actually kick the timer. This makes sense, since userspace being functional is probably what you really care about. [1] https://www.kernel.org/doc/html/latest/watchdog/watchdog-api.html https://www.kernel.org/doc/html/latest/watchdog/watchdog-api...
  - touisteur 4y ago
    One of the preternal problems of such hardware watchdogs was the inability to discriminate whether a sudden reboot was due a reset-button, hw security (e.g. temperature), ECC problem, or (micro) loss of power, or HW watchdog. On most IPMI-capable BIOS/firmware there's now (been for 10 years but I'm old) an option to log 'system' events (ipmi failures like fan speeds if you've set threshold, but also reboot reasons). It's call the System Event Log. Very useful. And on IPMI-plugged watchdogs, you can also see the state of the HW watchdog (is it running, how many seconds are left). Very useful too.
- AnssiH 4y ago
  In addition to those already mentioned, one way is to enable it in systemd: # /etc/systemd/system.conf.d/foobar.conf [Manager] RuntimeWatchdogSec=60 When used in this manner, if systemd fails to ping the watchdog for 60 seconds, the system resets. https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html https://www.freedesktop.org/software/systemd/man/systemd-sys... Somewhat related, nowadays by default systemd enables a 10-minute watchdog just before a regular reboot (i.e. after everything has been shut down) to ensure the reboot happens even if there is a hang for some kernel/HW reason.
ilyt 4y ago
There is a caveat here, it won't stop your app from crashing before the watchdog activation. Some CPUs have fuse that can enable watchdog before any code starts running but the ARMs I played with (STM32) don't appear to have that option.
- shabble 4y ago
  At least some STM32s do, see page 89 of the STM32F4xx reference manual[1], the option bits 5:7 at 0x1fffc000 let you activate the hardware watchdog immediately following reset if you wish. [1] https://www.st.com/resource/en/reference_manual/rm0090-stm32f405415-stm32f407417-stm32f427437-and-stm32f429439-advanced-armbased-32bit-mcus-stmicroelectronics.pdf https://www.st.com/resource/en/reference_manual/rm0090-stm32...
shabble 4y ago
A more specialised variant that's also quite common is the "window watchdog" peripheral, which is similar to the timer version, but will also trigger a reset if the keep-alive signal arrives too early, as well as too late. It can be useful where you've got a mainloop doing some very predictably timed activities, and allows detection of faults which cause your watchdog servicing to occur too frequently. I think it's quite common in DSP and things like motor control, where you often have hard realtime requirements and things happening too soon is just as bad as too late.

zh3 4y ago

A true hardware watchdog is separate electronics. For example, it's really easy to make an electronic circuit that operates a relay once every 5 minutes, say, unless a 'restart timer' button is pressed. Connect the relay contacts across the PC reset button, and run a program on the PC that 'presses' (electronically) the 'restart timer' button once a minute, say. Then if the PC fails to boot, it gets reset once every 5 minutes until it does (fairly obviously, another set of relay contacts can be used to trigger an alerting device, e.g. wailing sirens). We use this sort of approach with diskless systems in particular. If there's a power cut, the first boot attempt after power restoration might not work (because the network isn't back up yet). So the diskless systems just sit there, continually attempting a network boot until successfu (at which point the software on the PC hits the 'restart timer' button periodicaly. This is closely related to the concept of a "deadman's handle", for example train drivers who must keep a lever pressed down during operation - if it's released, the train stops automatically.

deleted 4y ago
[deleted]

dtgriscom 4y ago

Embedded micros (Systems On a Chip) often include dedicated watchdog hardware. This is a timer which is reset ("fed") by writing values to a specific register. Crucially, it often isn't just one value; you have to alternate between two values. That way, you can write one value at one point in your event loop and the other value at another point, making it less likely that something will break but keep feeding the watchdog. If the watchdog hasn't been fed for X milliseconds, then it resets the system.