---------------- One should be able to design a watchdog system to handle the all kind of HW failures. ---------------- The KISS design principle is : "If I

---------------- One should be able to design a watchdog system to handle the all kind of HW failures. ---------------- The KISS design principle is : "If I am a working SW/HW, I always assume the Master has failed and ready to take over the overall control unless I got Absolutely Positive confirmation that the Master system is working perfectly within the in N milliseconds windows." 10 years ago, I designed the system HA software for such system. It was only with two completely identical redundant system. Each system had 19 Xilinx FPGAs. It worked very well. One can trigger any type of failures on any of one the system. Within 250 milliseconds, the other system automatically can / did take over the control whenever for whatever failure that was detected. The system software design was very simple and worked very well in the Comcast and other cable network deployments. The input was 10GBits MPEG2 streams. The output 48 channels of Analog NTSC TV pictures. If you blink your eyes, you won't notices the complete System HW / Data Flow had been switched over. It also made very cool demo. Unplugged power, network, boards on any one system, the other one automatically took over the complete control within 250 ms. If you watch very carefully, you only see slight flicker on the output TV signals for all 48 channels. One other benefit, once the system was in place, it was trivial to add in-service SW Upgrade to the system. One can upgrade any SW (uboot, kernel, FPGA code, HW, etc) with only 250 ms impact on the end customer's service. Once you got the idea on how it worked, it is actually not hard to expand for system with N - redundant HW/SW and let the "master working" HW to own / generate the correct solution within set N-millseconds time limit.