I cannot say who the client is for this, but it's a government department. I built a cluster which used a disk array for storage. When I was contracted to buil

I cannot say who the client is for this, but it's a government department. I built a cluster which used a disk array for storage. When I was contracted to build this I was assured that they were building several other clusters for use as test, QA and prod and that all I had to do was build the dev system and document so thoroughly that the other systems could be built from the documentation. At the time I wasn't yet security cleared so would not be allowed to build the other environments. Some time later (6-8 months) I was called back to help solve a performance related problem. When I arrived I discovered that the dev machine was now production, and that no other machines were built. I would have to do the work on production. Worse though... their backup and failover mechanism wasn't what I'd documented, but involved an identical piece of hardware nearby which was powered off. They hoped to simply turn it on when production failed. Their daily process was to approach the disk array and remove the 1st hot swappable disk from the live machine, and swap it with the 1st disk in the powered off identical system. When the array had rebuilt the first disk, then move the 2nd disk. I am seldom speechless. It was only a critical part of a £2bn project. Beyond all of the obvious WTFs, I still wonder whether the 2nd machine could be powered on at all given that all of the disks were pulled at different points in time.