The issue here is one of scale, not necessarily intentional complexity - though admittedly one often leads to the other. In a world of single servers - or even

The issue here is one of scale, not necessarily intentional complexity - though admittedly one often leads to the other. In a world of single servers - or even multiple servers load balanced over a single database - you only have a single point of failure. And while that is often touted as an terrible thing, once it goes down it's game over... in some ways, our scaled up distributed systems have a worse problem. The issue is we envision our distributed systems optimistically - as isolated little bubbles that receive and emit clean, efficient, fault tolerant packets. When one goes down, only a few others get affected, and the broken module can be easily re-deployed with very little fuss. In reality, the people with the vision quickly become horrified with what is produced. The authentication system touches everything. The stock management system gets hooked directly into the homepage. Suddenly, every single part of your system is hinged on at least 5 servers, and if any of them go down, it's game over until one of them gets brought back up, and by that point the job management queue is so backed up with critical tasks (that should probably have never been entrusted to a background jobs queue) that it takes hours for the organism to re-balance itself. And so, these systems go down more often, because instead of one single point of failure, you have 5 - while telling yourself you have none. The question is, how can we do this better? I've seen big teams and small teams similarly struggle with this. I guess Netflix with their chaos monkey strategy wins on this, but even that is not infallible - as you say, the 1000 layers of abstraction (and goodness knows how many unique combinations of interactions and potential simultaneous failures) something is bound to be missed from time to time. So I don't think the problem is we're moving faster or breaking more things. It's just that we don't know how broken our systems are until they break.