I worked at a place with a few thousand servers. Because the business had been around for a while, there was two separate ways to deploy and manage apps: old a

I worked at a place with a few thousand servers. Because the business had been around for a while, there was two separate ways to deploy and manage apps: old and new. We reconfigured how code was deployed within and between racks for increase resistance to racks or even whole colos dying. This change was made in the new deployment system. And all was well for most of a year. Software and services gradually migrated from old to new as people had the time and inclination. The new deployment also packed services more tightly, so most computers kept copies of most binaries on them. Then we crossed one of those tipping points. On a code update, several thousand servers across the world demanded a small set of binaries from a distributed data node via the newer deployment method. Which would be fine, except we had degraded service from a couple TOR routers due to a separate bug. And so the distributed data nodes hosting this particular set of binaries were less "distributed" and more "just one poor about to be overloaded machine and network." So a few thousand machines demanded a couple hundred megs nearly simultaneously. And when they didn't get it in a timely fashion, all triggered a rollback. Simultaneously. Which demanded another couple hundred mb of code. Simultaneously. And then the freakouts started. And because other deploys were in flight, there was more than one service deploy ongoing. And they all started fighting each other for bandwidth.