The worst bug I encountered was when physically relocating a multi rack storage array for a mobile provider. The array had never been powered down(!) so we anti

The worst bug I encountered was when physically relocating a multi rack storage array for a mobile provider. The array had never been powered down(!) so we anticipated that a good number of the spindles would fail to come up on restart. So we added an extra mirror to protect each existing raid set. Problem is a bug in the firmware meant the mere existence of this extra mirror caused the entire arrays volume layout to become corrupted at reboot time. Fortunately a field engineer managed to reconstruct the layout, but not before a lot of hair had been whitened.