Many backup systems copy the data automatically each night, but if the data is damaged on the main system and nobody notices right away it can overwrite the bac

Many backup systems copy the data automatically each night, but if the data is damaged on the main system and nobody notices right away it can overwrite the backup. Or, in the stress of late-night recovery sessions, someone can copy things in the wrong direction and overwrite the backup. Sometimes the backups are encrypted with a highly secure key, and the key is lost with the original failure, or stored within the encrypted backup like keys locked inside a car. Because SANs allow multiple machines to write to the same disk, they have byzantine failure modes where data can be overwritten on disk, but cached for long periods of time by the machines that need it so there's no visible problem until after a power failure. When data is migrated to new database hardware, sometimes the backups haven't actually been tested. Sometimes during the cutover, someone who didn't fully understand the architecture backs up the new (empty) machine on top of the old machine's backups, so there is no backup for a while. All these mistakes can happen pretty easily when sysadmins are woken up in the night to fix a service outage while people are yelling at them. I have made some of these mistakes myself, though so far I've been lucky. The computer industry has never learned to admit that there is always some level of risk. You won't hear an oil company exec say "Inconceivable!" when an iceberg crashes into a North Sea oil rig, or a refinery catches on fire. Systems fail, and all you can do is try your best to avoid it and move forward after it does happen.