Ugh, I hate situations where multiple screwups turn into massive data loss. I'm glad this one worked out. I remember the time when a sysadmin gave two weeks no

Ugh, I hate situations where multiple screwups turn into massive data loss. I'm glad this one worked out. I remember the time when a sysadmin gave two weeks notice and left just before the Christmas holidays, before getting new backup system fully working. During the holidays, when almost everybody was out of town, one of the disks in our main server's RAID array failed. Then when the RAID controller tried to rebuild the array using the hot spare, a second disk failed. Then, just because the situation wasn't already horrible enough, the ancient proprietary controller (the one which knew how all the data was actually spread across all the disks) decided that this would be a great time to give up for good. We ended up paying about $7K to DriveSavers, who managed to salvage nearly all the data from the remnants of the RAID array. And while we were interviewing sysadmins, we replaced the hardware and set up a real backup system. Then we built an offsite mirror of the backup system, and we configured paranoid monitoring for all our RAID controllers. It's easy to get 100% uptime and no data loss for several years, if you're lucky. But if you build sloppy systems, sooner or later everything will go wrong, and when it does, the chaos can be impressive.