3 ms·

A gentle reminder to actually TEST your restore workflow. WBD: since 2011! https://hn.algolia.com/?q=worldbackupday https://hn.algolia.com/?q=worldbackupday

dewey 3y ago
People always say that but what do they actually mean? Restore one file, check if the backup is bootable in case it should?
- 369548684892826 3y ago
  For static files like photos I hash check drives against each other to check for bit rot. But yeah, for tape based cloud storage I can't think what else to do except restore one file to check I still have access. It's too expensive to restore the whole archive.
  - lloydatkinson 3y ago
    Do you have any automation or scripts for that, or is it usually adhoc?
    - dmd 3y ago
      You absolutely should NOT automate this. If you do, you then need to manually check that automation with the same frequency- so you haven’t gained anything!
      - 369548684892826 3y ago
        I do have this automated, but the checksum output is posted to a kind of watchdog service. Every day I get an email that says everything is as expected, or not.
      - lloydatkinson 3y ago
        I need to google every time to find a good process for hashing files and comparing across disks? Not only that, but remember to do it frequently enough?
    - 369548684892826 3y ago
      Yeah, I use rsync with `-cavin`, if the output includes `>` or `<` then there are differences in checksums. #!/bin/bash hostname=$(hostname) checksumFile=/tmp/checksum-file.txt rsync -cavin --info=name2 --no-perms --no-owner --no-group /srv/data/photos/ user@remote:/media/data/rsnapshot/daily.0/srv/data/photos/ > $checksumFile minimumSize=5000 actualSize=$(wc -l <"$checksumFile") if [ $actualSize -ge $minimumSize ]; then fileDiffs=$(cat $checksumFile | grep -e '<' -e '>' || echo 'all checksums match') else fileDiffs=$(echo $checksumFile is too small) fi echo $fileDiffs
- rzzzt 3y ago
  You can also restore the entire backup to /dev/null (this device has lots of free space) or pipe the output into a checksum generator. Borg also has a consistency checking command.
- tomatocracy 3y ago
  I'm sure there will be plenty of advice for testing the backups themselves, but for online backups (either to server or to a service), one thing which is easy to forget to do is to check you can still get access from "scratch" - ie if you have lost all local data. Make sure you don't need eg passwords or keys for the online backup service which you have no other copy of (doesn't help if they are in the backup itself!) or indeed for email/cloud accounts which you might need to regain access to to get to your backups.
- wjnc 3y ago
  You’d actually want to think a bit more. Where is your paper decryption key? Is it off premises? Test a, some or all of the files in the archive. Boot into an ISO. Try it again. And that for all of your backups. It’s not a backup until it is tested.

dewey 3y ago

People always say that but what do they actually mean? Restore one file, check if the backup is bootable in case it should?

369548684892826 3y ago
For static files like photos I hash check drives against each other to check for bit rot. But yeah, for tape based cloud storage I can't think what else to do except restore one file to check I still have access. It's too expensive to restore the whole archive.
- lloydatkinson 3y ago
  Do you have any automation or scripts for that, or is it usually adhoc?
  - dmd 3y ago
    You absolutely should NOT automate this. If you do, you then need to manually check that automation with the same frequency- so you haven’t gained anything!
    - 369548684892826 3y ago
      I do have this automated, but the checksum output is posted to a kind of watchdog service. Every day I get an email that says everything is as expected, or not.
    - lloydatkinson 3y ago
      I need to google every time to find a good process for hashing files and comparing across disks? Not only that, but remember to do it frequently enough?
  - 369548684892826 3y ago
    Yeah, I use rsync with `-cavin`, if the output includes `>` or `<` then there are differences in checksums. #!/bin/bash hostname=$(hostname) checksumFile=/tmp/checksum-file.txt rsync -cavin --info=name2 --no-perms --no-owner --no-group /srv/data/photos/ user@remote:/media/data/rsnapshot/daily.0/srv/data/photos/ > $checksumFile minimumSize=5000 actualSize=$(wc -l <"$checksumFile") if [ $actualSize -ge $minimumSize ]; then fileDiffs=$(cat $checksumFile | grep -e '<' -e '>' || echo 'all checksums match') else fileDiffs=$(echo $checksumFile is too small) fi echo $fileDiffs
rzzzt 3y ago
You can also restore the entire backup to /dev/null (this device has lots of free space) or pipe the output into a checksum generator. Borg also has a consistency checking command.
tomatocracy 3y ago
I'm sure there will be plenty of advice for testing the backups themselves, but for online backups (either to server or to a service), one thing which is easy to forget to do is to check you can still get access from "scratch" - ie if you have lost all local data. Make sure you don't need eg passwords or keys for the online backup service which you have no other copy of (doesn't help if they are in the backup itself!) or indeed for email/cloud accounts which you might need to regain access to to get to your backups.
wjnc 3y ago
You’d actually want to think a bit more. Where is your paper decryption key? Is it off premises? Test a, some or all of the files in the archive. Boot into an ISO. Try it again. And that for all of your backups. It’s not a backup until it is tested.