Going back to your comment, I feel like I missed this part, or at least did not address it: >Write better tests, clean-up the interfaces, you can even try to u

Going back to your comment, I feel like I missed this part, or at least did not address it: >Write better tests, clean-up the interfaces, you can even try to use formal methods to ensure correctness etc. I definitely put effort into all these things, but my experience shows that no matter how much work goes into that, things will find a way to fail in a way I could not predict.