>Eventually you get burned. For me, it was a routine should-have-been-minor web server update where one of the packages I relied on suddenly became unsupported

>Eventually you get burned. For me, it was a routine should-have-been-minor web server update where one of the packages I relied on suddenly became unsupported and every single hosted site stopped working. Since there's no way to roll back server upgrades, I had a marathon night involving building a new server stack and migrating all hosted sites there by 8 a.m. The traditional way to handle this in a cluster is with testing. The idea being that you have the upstream repo, the testing repo, and the production repo. Note, in most cases 'repo' means "directory tree served via http" and "sync" means "Copy, you know, with rsync or something" - This is not complicated. 10% of your servers point at the testing repo, the rest at the production repo. every X days, you set a test box against the upstream repo, update, reboot, and run your tests, you know, to catch the obvious stuff. If that works, you sync your testing repo with the upstream repo, and so 10% of your production now runs your new stuff. This is where you are gonna catch most of your problems, in my experience, but if something chokes, you only lose 10% of capacity. after Y days of the test stuff being on 10% of your boxes, you sync the test repo to the production repo. Of course, if you are like me, and when 10% of your boxes are down 10% of your customers are down, you want to spend a lot more effort on the 'test before you get customers on it' step. Also, sometimes there are 'don't sleep until you roll it out everywhere' updates, like this one (oh god, I am so happy that srn is on that now and I didn't have to deal with it) or like shellshock. In that case, well, sometimes you sync the upstream repo straight to production.