The telecom industry ran into similar issues and developed Erlang. I think a big takeaway from it is that designing systems which are failure-free is a fool's

The telecom industry ran into similar issues and developed Erlang. I think a big takeaway from it is that designing systems which are failure-free is a fool's errant - no matter how hard you try, you can never get rid of 100% of the bugs. Instead, make it failure-tolerant: sooner or later every part of the system will break, so it should be constructed in such a way that it can gracefully recover from failures, and even operate with some parts of it unavailable. Crashes are expected, so the system is designed to handle them properly.