One of the first companies I worked with had a gigantic, replicated filesystem across multiple Solaris boxes, in multiple geographies. It was actually faster to

One of the first companies I worked with had a gigantic, replicated filesystem across multiple Solaris boxes, in multiple geographies. It was actually faster to replicate things with rsync across geographies than do anything else, simply because of how slow compilation could be (12 hours on a tricked out e450). A simple "rm -rf" turfed a directory full of configuration files. This was immediately (well, 2004 immediately) replicated to several countries. Thus several telephony systems went poof. Obviously the same rm -rf took out the CVS repository, and on a Saturday night. Want to make it more fun - the ENTIRE (worldwide) development team was offsite for a team-building week, in Bonn. Two of my friends got on a plane, flew to Ottawa, and started having a go at computers to find a recent repository. Hard drives were ripped out of machines, jacked into others, mounted, and prayers made. This wouldn't have had to happen had the tape robots (yup, plural), with backups not both their arms stuck, and the vendor simply not providing 24x7x365 support because the company was too small. On the bright side, everything was up relatively quickly all things considered. Way better than the time Mike BBQed 20000 chickens accidentally.