4 ms·

If you don't test your backups, they're not backups. But I guess in this case if you don't have backups at all, you also don't have backups... 1 is 0, 2 is 1, e

If you don't test your backups, they're not backups. But I guess in this case if you don't have backups at all, you also don't have backups... 1 is 0, 2 is 1, etc...

LinuxBender 7y ago
I would take that a step further and say, "If anything, including automation, can tamper with your backups, they are not backups" Several companies have gone out of business this year due to getting their servers wiped by an attacker. Their backups were on live servers as well.
mdip 7y ago
I used to manage a set of tape backup robots (huge square tapes, I want to say 25 glorious GB a piece, and some Sony DAT devices @50-75GB a piece) that backed up sister data centers over a few DS-3s back in the early 2000s. The hardware/servers involved were around a 3/4 million. I only provide that context because I've been as far away from that experience as I can possibly put myself and I can only hope it's gotten a lot better. Prior to me and during the time that I managed backups, the statement "if you don't have backups at all, you don't have backups" was true for the entire backup set probably about once a week[0]. Those miserable robotic devices would crush tapes, the drives would eat tapes or otherwise fail, the backup servers would be over-whelmed (despite being nearly the largest hosts we had), the software would crash (Backup Exec), the server would just be ... skipped? And the tapes had a lifetime that was 1/10th what was advertised (and 1/2 what we had planned on[1]). Our process included a monthly restore test of a few critical pieces of infrastructure and a rotating list of other hosts. I think the one time that we passed might have been worthy of opening a bottle of wine. Most of the time we could get enough data back that the restore wouldn't result in more than a rounding-error financial loss, but for the most part our server team spent a lot of time praying. That system gave me nightmares for years after I left that position. [0] It was so bad that our initial plan was to backup DC A using DC B and vice versa, but we opted to backup both from both data-centers (one of our requirements was off-site backups). [1] I think I read the headline to an article a month ago about why "tape is still king" in backups. I couldn't bring myself to click-in. I assume things have gotten better, but if they're still selling this product the way they did back then, I'd want real-world numbers on MTBF and the like.
- tracker1 7y ago
  Worked in a helpdesk for a while, with a similar system, (this was after working support for iomega). I have no desire to work with anything like this ever again. Systems push to drop location. Backup services pull from drop location(s). Redundant storage, etc. I don't trust "disposable" media at all anymore, and limited trust for hard storage (hdd/ssd/etc). To me, if it's not on at least 3 devices at at least 2 locations, it's not properly backed up.
snarf21 7y ago
This reminds me of the Gitlab outage when they realized their backup images existed but were size 0. If you aren't regularly restoring backups to a machine, you are setting up for failure.
decasia 7y ago
Question: OK, so I know how to test — manually — that a few randomly chosen files are correctly backed up in my backup systems. But what if there are larger classes of systemic error in the process I haven't thought of testing? What if some particular file type or directory tree has vanished from the backups, but since it isn't in my manual testing process, I never catch it? Are there best practices for validating that whole directory trees are correctly backed up across the board? Or any form of automated testing for backups (but, presumably, separate from the software that does the backup process)?
- rhinoceraptor 7y ago
  I would just use ZFS snapshots and replication. They're read-only, and they only use as much space as the difference between the current state and the snapshot. Plus, you'll have solid data integrity instead of relying on buggy firmware in your RAID controller.
  - tracker1 7y ago
    And there's no way to compress/eliminate snapshots? Also, redundancy and snapshots are not a backup.
- magashna 7y ago
  Doing some DR testing is the only way to truly know
- benjohnson 7y ago
  For key servers, we've been known to clone the entire drive and put the clone in service. This helped us with several problems: Backup system didn't record the permissions. Backup didn't get 'resource forks' or 'alternate data streams' Backup wasn't complete enough to satisfy copy protection for software that was no longer supported. Backup process didn't see deep into the folder because there was a hardlink/symlink loop that caused it to stop Backup didn't get a coherent copy of a bunch of files that was acting like a database Drive had a CRC error that let it keep running but stopped the backup system from reading the file Windows program stored vale in the case of the file name while the Unix backup command assumed it could squish the Windows files to lowercase.
irrational 7y ago
Every night we rebuild our dev and qa servers and databases from backups of production which were taken that same evening. If there is something wrong with the backups, the dev team will definitely know about it the next day when they try to start working and find that their development and qa systems are not working.
- at-fates-hands 7y ago
  The insane thing is it's 2019 and this should be a baseline when dealing with your backups. It's shocking to know many people don't even do this.
AdmiralAsshat 7y ago
Reminds me of a client's system I was on a few years ago. They had a backup script on their server that was written some time in the 90's by the looks of it. Used ye old `cpio` command to write to a tape drive; someone's job was to manually go and eject the tape every morning and replace it. For verification, the script simply checked the exit status of the cpio command, and if it returned 0, it wrote "BACKUP STATUS: SUCCESSFUL"-- or 'BACKUP STATUS: BAD" if non-zero--to a log and sent an email. At some point we had to try to restore something, and spent several hours trying to figure out why their backup tape was empty. Some quick digging into the script discovered that the command would indeed write some kind of tarball to the backup tape, it was just zero bytes. IIRC, I eventually traced it to some problem with the cpio command erroring out if it tried to copy a file larger than 4GB. Suffice to say, the client was a little shaken that they had gone god-knows-how-many-months without an actual backup.

If you don't test your backups, they're not backups. But I guess in this case if you don't have backups at all, you also don't have backups... 1 is 0, 2 is 1, etc...

LinuxBender 7y ago

I would take that a step further and say, "If anything, including automation, can tamper with your backups, they are not backups" Several companies have gone out of business this year due to getting their servers wiped by an attacker. Their backups were on live servers as well.

mdip 7y ago

I used to manage a set of tape backup robots (huge square tapes, I want to say 25 glorious GB a piece, and some Sony DAT devices @50-75GB a piece) that backed up sister data centers over a few DS-3s back in the early 2000s. The hardware/servers involved were around a 3/4 million. I only provide that context because I've been as far away from that experience as I can possibly put myself and I can only hope it's gotten a lot better. Prior to me and during the time that I managed backups, the statement "if you don't have backups at all, you don't have backups" was true for the entire backup set probably about once a week[0]. Those miserable robotic devices would crush tapes, the drives would eat tapes or otherwise fail, the backup servers would be over-whelmed (despite being nearly the largest hosts we had), the software would crash (Backup Exec), the server would just be ... skipped? And the tapes had a lifetime that was 1/10th what was advertised (and 1/2 what we had planned on[1]). Our process included a monthly restore test of a few critical pieces of infrastructure and a rotating list of other hosts. I think the one time that we passed might have been worthy of opening a bottle of wine. Most of the time we could get enough data back that the restore wouldn't result in more than a rounding-error financial loss, but for the most part our server team spent a lot of time praying. That system gave me nightmares for years after I left that position. [0] It was so bad that our initial plan was to backup DC A using DC B and vice versa, but we opted to backup both from both data-centers (one of our requirements was off-site backups). [1] I think I read the headline to an article a month ago about why "tape is still king" in backups. I couldn't bring myself to click-in. I assume things have gotten better, but if they're still selling this product the way they did back then, I'd want real-world numbers on MTBF and the like.

tracker1 7y ago
Worked in a helpdesk for a while, with a similar system, (this was after working support for iomega). I have no desire to work with anything like this ever again. Systems push to drop location. Backup services pull from drop location(s). Redundant storage, etc. I don't trust "disposable" media at all anymore, and limited trust for hard storage (hdd/ssd/etc). To me, if it's not on at least 3 devices at at least 2 locations, it's not properly backed up.

snarf21 7y ago

This reminds me of the Gitlab outage when they realized their backup images existed but were size 0. If you aren't regularly restoring backups to a machine, you are setting up for failure.

decasia 7y ago

Question: OK, so I know how to test — manually — that a few randomly chosen files are correctly backed up in my backup systems. But what if there are larger classes of systemic error in the process I haven't thought of testing? What if some particular file type or directory tree has vanished from the backups, but since it isn't in my manual testing process, I never catch it? Are there best practices for validating that whole directory trees are correctly backed up across the board? Or any form of automated testing for backups (but, presumably, separate from the software that does the backup process)?

rhinoceraptor 7y ago
I would just use ZFS snapshots and replication. They're read-only, and they only use as much space as the difference between the current state and the snapshot. Plus, you'll have solid data integrity instead of relying on buggy firmware in your RAID controller.
- tracker1 7y ago
  And there's no way to compress/eliminate snapshots? Also, redundancy and snapshots are not a backup.
magashna 7y ago
Doing some DR testing is the only way to truly know
benjohnson 7y ago
For key servers, we've been known to clone the entire drive and put the clone in service. This helped us with several problems: Backup system didn't record the permissions. Backup didn't get 'resource forks' or 'alternate data streams' Backup wasn't complete enough to satisfy copy protection for software that was no longer supported. Backup process didn't see deep into the folder because there was a hardlink/symlink loop that caused it to stop Backup didn't get a coherent copy of a bunch of files that was acting like a database Drive had a CRC error that let it keep running but stopped the backup system from reading the file Windows program stored vale in the case of the file name while the Unix backup command assumed it could squish the Windows files to lowercase.

irrational 7y ago

Every night we rebuild our dev and qa servers and databases from backups of production which were taken that same evening. If there is something wrong with the backups, the dev team will definitely know about it the next day when they try to start working and find that their development and qa systems are not working.

at-fates-hands 7y ago
The insane thing is it's 2019 and this should be a baseline when dealing with your backups. It's shocking to know many people don't even do this.

AdmiralAsshat 7y ago

Reminds me of a client's system I was on a few years ago. They had a backup script on their server that was written some time in the 90's by the looks of it. Used ye old `cpio` command to write to a tape drive; someone's job was to manually go and eject the tape every morning and replace it. For verification, the script simply checked the exit status of the cpio command, and if it returned 0, it wrote "BACKUP STATUS: SUCCESSFUL"-- or 'BACKUP STATUS: BAD" if non-zero--to a log and sent an email. At some point we had to try to restore something, and spent several hours trying to figure out why their backup tape was empty. Some quick digging into the script discovered that the command would indeed write some kind of tarball to the backup tape, it was just zero bytes. IIRC, I eventually traced it to some problem with the cpio command erroring out if it tried to copy a file larger than 4GB. Suffice to say, the client was a little shaken that they had gone god-knows-how-many-months without an actual backup.