4 ms·

I don't think it's fair to rule out the possibility of a simple screw-up. I've been responsible for important web services and I took a lot of precautions, but

by tlb 17y ago

I don't think it's fair to rule out the possibility of a simple screw-up. I've been responsible for important web services and I took a lot of precautions, but I can't say it was inconceivable that something could have gone wrong causing major permanent data loss. There are so many ways in which failures can cascade in a complex system there can be no absolute guarantees. Can anyone quantify how much address book data there was? As an outside guess, 1,000,000 customers times 1000 addresses times 300 bytes each is only 300 GB. They could have kept an emergency backup on rsync.net for $300 / month. I keep secondary, well-encrypted backups of things I particularly care about (not nearly 300 GB worth) there, as well as on a USB disk under my bed, in addition to the regular complete backups.

pyre 17y ago
> There are so many ways in which failures can cascade in a complex system there can be no absolute guarantees. What about off-site backup? How many failures can cascade in a way to wipe out off-site backups?
- domodomo 17y ago
  Totally, there was a line in the article about a "time bomb erasing everything including the tapes". Well...you leave all the tapes in you library, nothing gets offsited? Or not even offsited, it's not like the building burned down. Just tapes not left in the library. The theory that they might have the data but just don't know how to restore things because all the Danger people are gone, now that actually seems somewhat more credible to me.
- tlb 17y ago
  Many backup systems copy the data automatically each night, but if the data is damaged on the main system and nobody notices right away it can overwrite the backup. Or, in the stress of late-night recovery sessions, someone can copy things in the wrong direction and overwrite the backup. Sometimes the backups are encrypted with a highly secure key, and the key is lost with the original failure, or stored within the encrypted backup like keys locked inside a car. Because SANs allow multiple machines to write to the same disk, they have byzantine failure modes where data can be overwritten on disk, but cached for long periods of time by the machines that need it so there's no visible problem until after a power failure. When data is migrated to new database hardware, sometimes the backups haven't actually been tested. Sometimes during the cutover, someone who didn't fully understand the architecture backs up the new (empty) machine on top of the old machine's backups, so there is no backup for a while. All these mistakes can happen pretty easily when sysadmins are woken up in the night to fix a service outage while people are yelling at them. I have made some of these mistakes myself, though so far I've been lucky. The computer industry has never learned to admit that there is always some level of risk. You won't hear an oil company exec say "Inconceivable!" when an iceberg crashes into a North Sea oil rig, or a refinery catches on fire. Systems fail, and all you can do is try your best to avoid it and move forward after it does happen.
  - pyre 17y ago
    A lot of those possibilities are preventable though. If you backup through an automated rsync, why not used versioning (ala rdiff)? Why not periodically swap out the disks on the backup machine so that if the disk gets hosed you have some data backed up, even if it is old data. Some of the things like 'locking the keys in the car' can be mitigated by making sure that you test you backup system. Obviously you can't prevent everything, but to say that some of these issues are 'unavoidable risk' is like saying that an oil company exec would say, "shit happens," to reports that one of their oil tanker captains crashed a ship while drunk. The real problem is that backup is usually an after-though.

by tlb 17y ago

pyre 17y ago

> There are so many ways in which failures can cascade in a complex system there can be no absolute guarantees. What about off-site backup? How many failures can cascade in a way to wipe out off-site backups?

domodomo 17y ago
Totally, there was a line in the article about a "time bomb erasing everything including the tapes". Well...you leave all the tapes in you library, nothing gets offsited? Or not even offsited, it's not like the building burned down. Just tapes not left in the library. The theory that they might have the data but just don't know how to restore things because all the Danger people are gone, now that actually seems somewhat more credible to me.
tlb 17y ago
Many backup systems copy the data automatically each night, but if the data is damaged on the main system and nobody notices right away it can overwrite the backup. Or, in the stress of late-night recovery sessions, someone can copy things in the wrong direction and overwrite the backup. Sometimes the backups are encrypted with a highly secure key, and the key is lost with the original failure, or stored within the encrypted backup like keys locked inside a car. Because SANs allow multiple machines to write to the same disk, they have byzantine failure modes where data can be overwritten on disk, but cached for long periods of time by the machines that need it so there's no visible problem until after a power failure. When data is migrated to new database hardware, sometimes the backups haven't actually been tested. Sometimes during the cutover, someone who didn't fully understand the architecture backs up the new (empty) machine on top of the old machine's backups, so there is no backup for a while. All these mistakes can happen pretty easily when sysadmins are woken up in the night to fix a service outage while people are yelling at them. I have made some of these mistakes myself, though so far I've been lucky. The computer industry has never learned to admit that there is always some level of risk. You won't hear an oil company exec say "Inconceivable!" when an iceberg crashes into a North Sea oil rig, or a refinery catches on fire. Systems fail, and all you can do is try your best to avoid it and move forward after it does happen.
- pyre 17y ago
  A lot of those possibilities are preventable though. If you backup through an automated rsync, why not used versioning (ala rdiff)? Why not periodically swap out the disks on the backup machine so that if the disk gets hosed you have some data backed up, even if it is old data. Some of the things like 'locking the keys in the car' can be mitigated by making sure that you test you backup system. Obviously you can't prevent everything, but to say that some of these issues are 'unavoidable risk' is like saying that an oil company exec would say, "shit happens," to reports that one of their oil tanker captains crashed a ship while drunk. The real problem is that backup is usually an after-though.