6 ms·

A Practical Guide to Watchdogs for Embedded Systems

104 points by fra 7y ago

fra 7y ago
Watchdogs are one of the more frustrating types of issues to debug. Chris's overview of how to implement them properly, and investigate resets is an amazing resource I wish I had earlier in my career.
- RealityVoid 7y ago
  Heh, I work in embedded and, IMO, they were one of the most fun issues to debug in the begging of my career. Mostly, if I had such an issue, I enabled the ISR triggering and just before the context got trashed, I looked where the program was. Things got pretty obvious then. Once you get the hang of it, it becomes pretty straightforward. And it's great training on how the HW looks like and functions! Granted, on more lower powered systems, with fewer, more primitive debugging options (looking at you, PIC) things miiight look a bit more painful. Thankfully, such systems are (much) smaller. IMO, the _most_ painful issues to debug are memory corruption issues happening on large systems without MPU enabled.
  - fra 7y ago
    You’re right, if it reproduces at your desk you’ve got some good tools at your disposal. Collecting enough state to fix them from a customer report is tricky, I would say.
  - raxxorrax 7y ago
    On the other side even the smallest PICs and Atmels have sooo much EEPROM and flash memory available. I am still amazed how much better these chips got in the last decade. You have all the I/O you could want, nearly endless memory and the chip costs less than a bubblegum. In smaller systems for memory checks I always have a complete self-check routine at boot. Although I cannot even remember when I had the last error for internal memory. The quality seems to be quite high, even after many read/write cycles. Never implemented that for a system with virtual memory. What I mostly miss in embedded systems isn't debugging, it is having real unit tests without having the target board and all the periphery installed in a laboratory, which is the only location I can feasibly run any tests. And those are mostly restricted to integration tests.
fwsgonzo 7y ago
This was an awesome read. Thanks for writing this.
joezydeco 7y ago
One scenario missed in the list of causes is a corrupted runtime image. An advanced topic is enabling support for the watchdog in your bootloader and having a defined recovery path when the system fails to load or, worse, the application falls into a boot loop. If you have the space, you can fall back to a recovery image or duplicate of the application. If you don’t have the space, falling into a DFU mode is a good plan.
monocasa 7y ago
One neat thing I've seen that doesn't get called out enough, is a high priority timer that has a slightly smaller period than your watchdog. When you let the watchdog, you pet this timer too. Then in the timer ISR, write out the trap frame to brain dead non volatile memory (we had battery backed SRAM, and then MRAM on newer boards). Then when the board reboots, and checks in, you can pull down what it was doing when the watchdog triggered.
- tyhoff 7y ago
  It's mind-boggling to me that many projects I've heard about and worked on treat (hardware) watchdogs issues as impossible to debug, for the exact reason you mentioned...they didn't set up a software watchdog. Doing some basic groundwork and implementing the timer/software watchdog, writing out to logs when a watchdog is about to trigger, and not disabling the watchdog during development are all basic things teams can do to more easily catch these issues before shipping firmware. JSYK: I do believe this was mentioned in the post as well, under the section 'Adding a “Software” Watchdog', but when a little bit further and implemented a per-task watchdog.
- unwind 7y ago
  Yes! I implemented something similar on a particularly annoying 8/16-bit controller just a few weeks ago. Extra fun since it had no instruction to read the program counter (and no general purpose register wide enough to hold it). I hope ARM adds that (or IC vendors like ST if possible) to the core itself, just mirror the PC to a register on reset. Should be trivial in hardware.
  - elcritch 7y ago
    Hmm, maybe you could possibly instrument the functions[1] to write the function address to a register, presuming you have enough program space. Presuming you're in C and your compiler supports that. 1: https://mcuoneclipse.com/2015/04/04/poor-mans-trace-free-of-charge-function-entryexit-trace-with-gnu-tools/ https://mcuoneclipse.com/2015/04/04/poor-mans-trace-free-of-...
- alxlaz 7y ago
NelsonMinar 7y ago
I was surprised at the poor state of watchdogs in PC-class Linux systems. I needed one recently and was bummed at the state of the old watchdog daemon / softdog kernel module. It works, but it is not nearly as easy to get going (on Ubuntu) as I expected. systemd also has its own watchdog and I can't figure it out. Anyway turns out I really needed a full PC hardware watchdog. I ended up buying some $8 anonymous piece of Chinese hardware that's USB powered. It hits the motherboard reset switch if the motherboard hard drive activity light hasn't flashed in awhile. Dumb thing, but it seems to work.
- kjs3 7y ago
  I found the same thing. Last time I needed a reliable PC watchdog it was an ISA (really PC/104) bus card that triggered a relay (causing reset) if a timer expired and a set of conditions hadn't been met.
- agapon 7y ago
  Almost all consumer motherboards have a chipset with embedded hardware watchdog that's well supported by respective drivers in FreeBSD / Linux / etc. Some vendors like Intel may lock down that functionality, but most (Asus, Gigabyte, etc) don't.
  - NelsonMinar 7y ago
    Do you have any more info on that chipset watchdog and how I'd use it in Linux? Is it this, the TCO Watchdog? http://www.madore.org/~david/linux/iTCO-wdt-test.html http://www.madore.org/~david/linux/iTCO-wdt-test.html
    - agapon 7y ago
      Yes, iTCO_wdt is for Intel chipsets; sp5100_tco is for AMD chipsets (including FCH). The names are a bit weird to my taste. FreeBSD ones sound a little bit better (but maybe I am just more used to them): ichwd (ICH WD) and amdsbwd (AMD SouthBridge WD).
senderista 7y ago
I would like to find a reliable software watchdog that kills a process when a timer expires (for preventing zombie processes from violating lease timeouts).
- foota 7y ago
  I mean, you probably shouldn't rely on the local system time for determining lease timeouts for correctness, if it matters.
  - senderista 7y ago
    I mean something like CLOCK_MONOTONIC.
    - foota 7y ago
      Don't you still need to worry about the clock not going up in pace with others? Like if a system has a lease until "100" but the clock doesn't tick (or more realistically ticks slowly), then another system could think it had the lease if it observed a local system time of 101? Maybe I'm misunderstanding how you're using the leases though.
      - senderista 7y ago
        Yes you do, and normally an upper bound epsilon for clock rate skew is explicitly assumed. That's obviously fragile but there are many highly available systems that have managed to get away with it, partly by keeping leases short.
- titzer 7y ago
  Why can't you do this which plain old C and the POSIX APIs for timers and process control? i.e. roll your own?
Paul_S 7y ago
Never in my career have I ever heard of any other term used than "kicking" the watchdog. Are the other terms popular in America?
- alxlaz 7y ago
  Why would you kick the poor thing? :( Not in America but terms I've heard include feeding the watchdog, keeping it alive, petting, greeting, barking at it, calling it and, my favourite, shushing it.
retSava 7y ago
The watchdog is a nice feature to have a borked system reboot, lifesaver in the field if feces hits the fan. What's less fun is if there is too little protection against electrostatic fields/EMI on the JTAG clock pin. On the small cortex m-class devices we work with, some of them can't shut off the JTAG part of the chip, meaning that when operating, if there are enough (I think 8) logic flips on the TCK pin in _any_ amount of time, the JTAG part wakes up, sets the HALT ON BOOT flag. Next time the device reboots (due to firmware update, or watchdog, ...), it will stop and stay in JTAG debug mode. Not nice. You need to manually power cycle the thing. We detect this by periodically checking the JTAG power domain, and if it is on, tell the server this so that we avoid rebooting it (eg automatically after firmware update). This way we've found poor hw implementations and tough EMI environments by proxy of JTAG power domain :D.
- scoutt 7y ago
  I work with Cortex-M since they launched and I've never seen on heard about this problem. Of course, I don't know the kind of environment your devices work, but many of my projects run in automotive. TCK has usually a pull-up/termination (but I've also seen pull-downs with/without caps). You see this issue even with the pull-up/down?
  - retSava 7y ago
    This was primarily in a lighting solution, and it wasn't the only problem with that early hw. We noticed lights not coming back online after a firmware update without a power cycle. Since it was very early hw, IIRC it didn't have any shielding, and also IIRC no pull on TCK. We now see this primarily on 1st/early iteration hw and on prototypes, but checking this proactively have saved us from a lot of headaches of the type "why doesn't it come back online - hw, or something in the update, or what...".
cmroanirgo 7y ago
A neat enough article, but surprised it didn't talk about an electronic watchdog: basically pulsing a gpio pin to trigger a recharge of a cap which holds a transistor active for a second or so, and that transistor drives eg a relay. An alternate method uses a gpio to reset a 555 timer. This will allow machinery to cut off when the embedded circuit stops looping. That is, any attached machinery would have a guaranteed NO (normally open) circuit and can only be engaged when all the watchdogs are working properly. Some mcu pins also go into an unknown state (neither guaranteed high nor low), so resetting a cpu can have bad consequences if it's driving big machinery, if not designed correctly. One project I had a pc sending software watchdog pings to several independent devices and each of those had an actual hardware watchdog (as opposed to the cpu resetting one in the article). I used the watchdog to physically control the power to contactors: no watchdog = no power = nothing activates. The system controlled firing of gas burners and fans etc, but the design was very safe, heaps of redundancy and was guaranteed to fail into a safe mode at any instant.

104 points by fra 7y ago

fra 7y ago

Watchdogs are one of the more frustrating types of issues to debug. Chris's overview of how to implement them properly, and investigate resets is an amazing resource I wish I had earlier in my career.

RealityVoid 7y ago
Heh, I work in embedded and, IMO, they were one of the most fun issues to debug in the begging of my career. Mostly, if I had such an issue, I enabled the ISR triggering and just before the context got trashed, I looked where the program was. Things got pretty obvious then. Once you get the hang of it, it becomes pretty straightforward. And it's great training on how the HW looks like and functions! Granted, on more lower powered systems, with fewer, more primitive debugging options (looking at you, PIC) things miiight look a bit more painful. Thankfully, such systems are (much) smaller. IMO, the _most_ painful issues to debug are memory corruption issues happening on large systems without MPU enabled.
- fra 7y ago
  You’re right, if it reproduces at your desk you’ve got some good tools at your disposal. Collecting enough state to fix them from a customer report is tricky, I would say.
- raxxorrax 7y ago
  On the other side even the smallest PICs and Atmels have sooo much EEPROM and flash memory available. I am still amazed how much better these chips got in the last decade. You have all the I/O you could want, nearly endless memory and the chip costs less than a bubblegum. In smaller systems for memory checks I always have a complete self-check routine at boot. Although I cannot even remember when I had the last error for internal memory. The quality seems to be quite high, even after many read/write cycles. Never implemented that for a system with virtual memory. What I mostly miss in embedded systems isn't debugging, it is having real unit tests without having the target board and all the periphery installed in a laboratory, which is the only location I can feasibly run any tests. And those are mostly restricted to integration tests.

fwsgonzo 7y ago

This was an awesome read. Thanks for writing this.

joezydeco 7y ago

One scenario missed in the list of causes is a corrupted runtime image. An advanced topic is enabling support for the watchdog in your bootloader and having a defined recovery path when the system fails to load or, worse, the application falls into a boot loop. If you have the space, you can fall back to a recovery image or duplicate of the application. If you don’t have the space, falling into a DFU mode is a good plan.

monocasa 7y ago

One neat thing I've seen that doesn't get called out enough, is a high priority timer that has a slightly smaller period than your watchdog. When you let the watchdog, you pet this timer too. Then in the timer ISR, write out the trap frame to brain dead non volatile memory (we had battery backed SRAM, and then MRAM on newer boards). Then when the board reboots, and checks in, you can pull down what it was doing when the watchdog triggered.

tyhoff 7y ago
It's mind-boggling to me that many projects I've heard about and worked on treat (hardware) watchdogs issues as impossible to debug, for the exact reason you mentioned...they didn't set up a software watchdog. Doing some basic groundwork and implementing the timer/software watchdog, writing out to logs when a watchdog is about to trigger, and not disabling the watchdog during development are all basic things teams can do to more easily catch these issues before shipping firmware. JSYK: I do believe this was mentioned in the post as well, under the section 'Adding a “Software” Watchdog', but when a little bit further and implemented a per-task watchdog.
unwind 7y ago
Yes! I implemented something similar on a particularly annoying 8/16-bit controller just a few weeks ago. Extra fun since it had no instruction to read the program counter (and no general purpose register wide enough to hold it). I hope ARM adds that (or IC vendors like ST if possible) to the core itself, just mirror the PC to a register on reset. Should be trivial in hardware.
- elcritch 7y ago
  Hmm, maybe you could possibly instrument the functions[1] to write the function address to a register, presuming you have enough program space. Presuming you're in C and your compiler supports that. 1: https://mcuoneclipse.com/2015/04/04/poor-mans-trace-free-of-charge-function-entryexit-trace-with-gnu-tools/ https://mcuoneclipse.com/2015/04/04/poor-mans-trace-free-of-...
alxlaz 7y ago

NelsonMinar 7y ago

I was surprised at the poor state of watchdogs in PC-class Linux systems. I needed one recently and was bummed at the state of the old watchdog daemon / softdog kernel module. It works, but it is not nearly as easy to get going (on Ubuntu) as I expected. systemd also has its own watchdog and I can't figure it out. Anyway turns out I really needed a full PC hardware watchdog. I ended up buying some $8 anonymous piece of Chinese hardware that's USB powered. It hits the motherboard reset switch if the motherboard hard drive activity light hasn't flashed in awhile. Dumb thing, but it seems to work.

kjs3 7y ago
I found the same thing. Last time I needed a reliable PC watchdog it was an ISA (really PC/104) bus card that triggered a relay (causing reset) if a timer expired and a set of conditions hadn't been met.
agapon 7y ago
Almost all consumer motherboards have a chipset with embedded hardware watchdog that's well supported by respective drivers in FreeBSD / Linux / etc. Some vendors like Intel may lock down that functionality, but most (Asus, Gigabyte, etc) don't.
- NelsonMinar 7y ago
  Do you have any more info on that chipset watchdog and how I'd use it in Linux? Is it this, the TCO Watchdog? http://www.madore.org/~david/linux/iTCO-wdt-test.html http://www.madore.org/~david/linux/iTCO-wdt-test.html
  - agapon 7y ago
    Yes, iTCO_wdt is for Intel chipsets; sp5100_tco is for AMD chipsets (including FCH). The names are a bit weird to my taste. FreeBSD ones sound a little bit better (but maybe I am just more used to them): ichwd (ICH WD) and amdsbwd (AMD SouthBridge WD).

senderista 7y ago

I would like to find a reliable software watchdog that kills a process when a timer expires (for preventing zombie processes from violating lease timeouts).

foota 7y ago
I mean, you probably shouldn't rely on the local system time for determining lease timeouts for correctness, if it matters.
- senderista 7y ago
  I mean something like CLOCK_MONOTONIC.
  - foota 7y ago
    Don't you still need to worry about the clock not going up in pace with others? Like if a system has a lease until "100" but the clock doesn't tick (or more realistically ticks slowly), then another system could think it had the lease if it observed a local system time of 101? Maybe I'm misunderstanding how you're using the leases though.
    - senderista 7y ago
      Yes you do, and normally an upper bound epsilon for clock rate skew is explicitly assumed. That's obviously fragile but there are many highly available systems that have managed to get away with it, partly by keeping leases short.
titzer 7y ago
Why can't you do this which plain old C and the POSIX APIs for timers and process control? i.e. roll your own?

Paul_S 7y ago

Never in my career have I ever heard of any other term used than "kicking" the watchdog. Are the other terms popular in America?

alxlaz 7y ago
Why would you kick the poor thing? :( Not in America but terms I've heard include feeding the watchdog, keeping it alive, petting, greeting, barking at it, calling it and, my favourite, shushing it.

retSava 7y ago

The watchdog is a nice feature to have a borked system reboot, lifesaver in the field if feces hits the fan. What's less fun is if there is too little protection against electrostatic fields/EMI on the JTAG clock pin. On the small cortex m-class devices we work with, some of them can't shut off the JTAG part of the chip, meaning that when operating, if there are enough (I think 8) logic flips on the TCK pin in _any_ amount of time, the JTAG part wakes up, sets the HALT ON BOOT flag. Next time the device reboots (due to firmware update, or watchdog, ...), it will stop and stay in JTAG debug mode. Not nice. You need to manually power cycle the thing. We detect this by periodically checking the JTAG power domain, and if it is on, tell the server this so that we avoid rebooting it (eg automatically after firmware update). This way we've found poor hw implementations and tough EMI environments by proxy of JTAG power domain :D.

scoutt 7y ago
I work with Cortex-M since they launched and I've never seen on heard about this problem. Of course, I don't know the kind of environment your devices work, but many of my projects run in automotive. TCK has usually a pull-up/termination (but I've also seen pull-downs with/without caps). You see this issue even with the pull-up/down?
- retSava 7y ago
  This was primarily in a lighting solution, and it wasn't the only problem with that early hw. We noticed lights not coming back online after a firmware update without a power cycle. Since it was very early hw, IIRC it didn't have any shielding, and also IIRC no pull on TCK. We now see this primarily on 1st/early iteration hw and on prototypes, but checking this proactively have saved us from a lot of headaches of the type "why doesn't it come back online - hw, or something in the update, or what...".

cmroanirgo 7y ago

A neat enough article, but surprised it didn't talk about an electronic watchdog: basically pulsing a gpio pin to trigger a recharge of a cap which holds a transistor active for a second or so, and that transistor drives eg a relay. An alternate method uses a gpio to reset a 555 timer. This will allow machinery to cut off when the embedded circuit stops looping. That is, any attached machinery would have a guaranteed NO (normally open) circuit and can only be engaged when all the watchdogs are working properly. Some mcu pins also go into an unknown state (neither guaranteed high nor low), so resetting a cpu can have bad consequences if it's driving big machinery, if not designed correctly. One project I had a pc sending software watchdog pings to several independent devices and each of those had an actual hardware watchdog (as opposed to the cpu resetting one in the article). I used the watchdog to physically control the power to contactors: no watchdog = no power = nothing activates. The system controlled firing of gas burners and fans etc, but the design was very safe, heaps of redundancy and was guaranteed to fail into a safe mode at any instant.