I'll add another rule: don't ship behavior you can't test. This includes the behavior of a stateful system after an extended period of uptime. I once spent wee

I'll add another rule: don't ship behavior you can't test. This includes the behavior of a stateful system after an extended period of uptime. I once spent weekends and late nights fixing a bug which, if it had shipped, would have eventually bricked an embedded product. It caused the software to hang in a way which evaded every watchdog and reset timer. This bug was only barely caught by the edge of a multi-week stress test. As it turned out, the bug itself wasn't in any of the code I had written, but in an open-source dependency. There are several ways to approach this problem. Besides simply testing your product over an extended period of time, you can build a test harness which simulates time at an accelerated rate. Another method is to periodically reboot your system from scratch after the longest period of uptime you're able or willing to test. It also helps to use software and hardware watchdogs which automatically reset your system, in whole or in part, if it becomes unresponsive (fails to "feed" the watchdog). However, watchdogs by themselves aren't foolproof - in my case, the program's main I/O loop kept running, but a critical part of the stack no longer functioned correctly.