4 ms·

> "borg" has basically solved backups permanently. It's deduplicated, even across snapshots, compressed, and end-to-end encrypted. Deduplication helps minimize

> "borg" has basically solved backups permanently. It's deduplicated, even across snapshots, compressed, and end-to-end encrypted. Deduplication helps minimize space, but isn't it a major liability in backups? I mean, what happens when you try to restore your backups but a lone sector holding a file from way back in the past happens to not be recoverable? Doesn't it mean that no matter how frequent your backups are, your data is lost?

NSUserDefaults 9mo ago
In that case you are supposed to use your /other/ backup. Which you have.
- DaSHacka 9mo ago
  Defeating the point of deduplication...
  - homebrewer 9mo ago
    You don't have backups if you only have one "backup". Look up sysadmin's 3-2-1.
  - NSUserDefaults 9mo ago
    On a single drive, deduplicate all you want. But you need to have two drives, at least, anyway.
haradion 9mo ago
It doesn't really hurt in practice because it's only one part of the full backup procedure. Deduplicate to save space; re-duplicate the (smaller) backups to separate media for redundancy; scrub regularly for bit rot and rotate your media. A proper backup system requires all but the first of those anyway.
stuxnet79 9mo ago
If you are storing your data in a filesystem like ZFS then the risk is lower since you'll know you have a bad sector long before you attempt to deduplicate. But I otherwise share your same concerns and I'm eager to hear from others how to mitigate this. I've lost a volume due to poor practice while encrypting so I'm understandably reluctant to overcomplicate my backup strategies in the name of space compression or privacy.
LeoPanthera 9mo ago
This is bordering on paranoia. If the bad sector contains a critical part of the filesystem, you're going to lose everything anyway. Do modern disks even have physical "sectors" anymore? Isn't it all virtual? Without dedup you're just going to backup less stuff, which is far worse.
- locknitpicker 9mo ago
  > This is bordering on paranoia. What? You actually think that the mere idea of a backup getting corrupted is something that is "bordering on paranoia"? Have you ever heard of redundancy or even RAID? > If the bad sector contains a critical part of the filesystem, you're going to lose everything anyway. Do you honestly failed to understand that the problem is actually the "lose everything" part?
dmd 9mo ago
This is why Proxmox Backup Server has scheduled "verify" tasks which look at the sha256 of every stored hunk to make sure it hasn't rotted on disk. (If it has, it makes sure to re-acquire that data on the next backup - hopefully it's still available!)
DrBurrito 9mo ago
the backups in borg and restic are not classical copies, each snapshot is not independent, it is a collection of hashed blocks. In a sense, time machine and hard links do the same: they reference the same bits across snapshots, saving space. for the bit rot / faulty sector or failed drive, you use redundancy with RAID or similar in your file system. if all else fails and your backup system burns in a fire, then you reach for your other backup. deduplication and redundancy solve different problems.

NSUserDefaults 9mo ago

In that case you are supposed to use your /other/ backup. Which you have.

DaSHacka 9mo ago
Defeating the point of deduplication...
- homebrewer 9mo ago
  You don't have backups if you only have one "backup". Look up sysadmin's 3-2-1.
- NSUserDefaults 9mo ago
  On a single drive, deduplicate all you want. But you need to have two drives, at least, anyway.

haradion 9mo ago

It doesn't really hurt in practice because it's only one part of the full backup procedure. Deduplicate to save space; re-duplicate the (smaller) backups to separate media for redundancy; scrub regularly for bit rot and rotate your media. A proper backup system requires all but the first of those anyway.

stuxnet79 9mo ago

If you are storing your data in a filesystem like ZFS then the risk is lower since you'll know you have a bad sector long before you attempt to deduplicate. But I otherwise share your same concerns and I'm eager to hear from others how to mitigate this. I've lost a volume due to poor practice while encrypting so I'm understandably reluctant to overcomplicate my backup strategies in the name of space compression or privacy.

LeoPanthera 9mo ago

This is bordering on paranoia. If the bad sector contains a critical part of the filesystem, you're going to lose everything anyway. Do modern disks even have physical "sectors" anymore? Isn't it all virtual? Without dedup you're just going to backup less stuff, which is far worse.

locknitpicker 9mo ago
> This is bordering on paranoia. What? You actually think that the mere idea of a backup getting corrupted is something that is "bordering on paranoia"? Have you ever heard of redundancy or even RAID? > If the bad sector contains a critical part of the filesystem, you're going to lose everything anyway. Do you honestly failed to understand that the problem is actually the "lose everything" part?

dmd 9mo ago

This is why Proxmox Backup Server has scheduled "verify" tasks which look at the sha256 of every stored hunk to make sure it hasn't rotted on disk. (If it has, it makes sure to re-acquire that data on the next backup - hopefully it's still available!)

DrBurrito 9mo ago

the backups in borg and restic are not classical copies, each snapshot is not independent, it is a collection of hashed blocks. In a sense, time machine and hard links do the same: they reference the same bits across snapshots, saving space. for the bit rot / faulty sector or failed drive, you use redundancy with RAID or similar in your file system. if all else fails and your backup system burns in a fire, then you reach for your other backup. deduplication and redundancy solve different problems.