4 ms·

restic used to be my tool of choice in the past, but I unfortunately ran into an issue where a single corrupted blob caused many files to be unrecoverable. Than

restic used to be my tool of choice in the past, but I unfortunately ran into an issue where a single corrupted blob caused many files to be unrecoverable. Thankfully, I noticed it when testing the backups so I didn't actually lose any data. I've since switched to using encrypted rclone backups due to the design decision of having each source file map to one target file. For my specific use case, it's more important that corruption of backup files has minimal impact vs. having good deduplication. I also like that the format is simple enough where I feel confident I could write an extraction tool that skips checksum validation or bad blocks if I really need to.

andrewchambers 6y ago
Yeah, this is one downfall of these tools, they amplify the risk of corruption unless you take additional redundancy measures.
StavrosK 6y ago
Did you test the backups with Restic's built in test command, or by restoring?
- chenxiaolong 6y ago
  I tested by attempting to do a full restore to a temp directory.
  - StavrosK 6y ago
    Do you remember if `restic check` said that everything was fine?
    - chenxiaolong 6y ago
      Unfortunately I don't remember. From the (little) notes I kept, the corruption occurred with restic version 0.9.3. I had also tried restoring with the latest code from git master, but I didn't make a note of the exact commit (it didn't make a difference anyway). The commands that triggered the corruption were the following. I tested the restore around the 10th run of this. repo=sftp:chenxiaolong@<ip address>:/home/restic restic -r "${repo}" backup -x /home restic -r "${repo}" backup -x --exclude /stuff/Android/AOSP /stuff restic -r "${repo}" forget --keep-last 3 --prune
aborsy 6y ago
Does this risk exist with un-encrypted Borg back ups also? (Compressed deduped also doesn’t map file to file).
- dimatura 6y ago
  Almost certainly, yes. It's an inherent risk with any backup system that deduplicates at at a block level, encrypted or not -- if the same block appears in many files, then it's stored only once (which is good in terms of space efficiency), but if that block is corrupted, then all the files sharing that block will be affected. Incidentally, this is not just an issue with backup systems; almost any compressed data format will also have similar risks. It's a tradeoff of the safety in redundancy for space efficiency.
  - jbboehr 6y ago
    Would it be possible or desirable for one of these backup system to natively implement parity?

andrewchambers 6y ago

Yeah, this is one downfall of these tools, they amplify the risk of corruption unless you take additional redundancy measures.

StavrosK 6y ago

Did you test the backups with Restic's built in test command, or by restoring?

chenxiaolong 6y ago
I tested by attempting to do a full restore to a temp directory.
- StavrosK 6y ago
  Do you remember if `restic check` said that everything was fine?
  - chenxiaolong 6y ago
    Unfortunately I don't remember. From the (little) notes I kept, the corruption occurred with restic version 0.9.3. I had also tried restoring with the latest code from git master, but I didn't make a note of the exact commit (it didn't make a difference anyway). The commands that triggered the corruption were the following. I tested the restore around the 10th run of this. repo=sftp:chenxiaolong@<ip address>:/home/restic restic -r "${repo}" backup -x /home restic -r "${repo}" backup -x --exclude /stuff/Android/AOSP /stuff restic -r "${repo}" forget --keep-last 3 --prune

aborsy 6y ago

Does this risk exist with un-encrypted Borg back ups also? (Compressed deduped also doesn’t map file to file).

dimatura 6y ago
Almost certainly, yes. It's an inherent risk with any backup system that deduplicates at at a block level, encrypted or not -- if the same block appears in many files, then it's stored only once (which is good in terms of space efficiency), but if that block is corrupted, then all the files sharing that block will be affected. Incidentally, this is not just an issue with backup systems; almost any compressed data format will also have similar risks. It's a tradeoff of the safety in redundancy for space efficiency.
- jbboehr 6y ago
  Would it be possible or desirable for one of these backup system to natively implement parity?