Watching this kind of complex distributed system in the throws of random failure every day at work, I wish we actually did this instead of just reacting to one

Watching this kind of complex distributed system in the throws of random failure every day at work, I wish we actually did this instead of just reacting to one nightmare after another.