4 ms·

I did not see anywhere mention of deduplication - is this implemented in Restic? (I use Borg and the feature is an absolute must for me)

lloeki 5y ago
IIRC restic operates at the file level on the source side and then stores data in blobs, and unchanged blobs are deduplicated (it operates as a content addressed store, kind of like git), which means iterative backups are possible. EDIT: see "blob" https://restic.readthedocs.io/en/latest/100_references.html#design https://restic.readthedocs.io/en/latest/100_references.html#... and "efficient" https://github.com/restic/restic#design-principles https://github.com/restic/restic#design-principles
lucb1e 5y ago
Yes, it deduplicates but does not compress. Since most large files (mp3, jpg, mp4, docx, etc.) are already compressed, this is not a big issue but it's not ideal either. If I remember correctly, another tool either does both or does compression instead of deduplication, this might have been borg or bup. In case that's something you care about.
- l33tman 5y ago
  Typical borg run: Original size Compressed size Deduplicated size This archive: 167.99 GB 136.78 GB 53.85 MB All archives: 2.93 TB 2.38 TB 133.66 GB Sorry about the formatting. But the compression is not completely irrelevant. Dedup of blocks between files and backups is of course the absolutely most crucial part though.
  - lucb1e 5y ago
    Thanks for the real-world stats! I must say it took me a bit to read it though, let me format it for others (hopefully also narrow enough for mobile): compress data | dedup 1 backup: 168G 137G 54M Σ backups: 2.9T 2.4T 134G Please correct me if 'one backup' and 'all backups' is an incorrect interpretation of 'all archives'. I wasn't entirely sure what you mean by that but I think I get the point. So in conclusion, adding compression saves about 17% (a $10 monthly bill would be $8.28 instead, if you pay per GB) for you.
    - lucb1e 5y ago
      Another friend has a stat of 30%, by the way. Even if I'd get the better of the two values, I don't have enough systems that a backup being 2/3rds of the original data size reduces the number of backup disks I need to buy, but it's not insignificant either.
- BrandoElFollito 5y ago
  Compression may help but that would be best case 10-30% (which is worth or not depending on the use case). Deduplication means ~20x smaller backups so this is absolutely key for me.
dividuum 5y ago
Yes. There's deduplication. It's using a rolling hash for chunking and then deduplicates those chunks. So every snapshot it always a full backup.
- lucb1e 5y ago
  To be clear, "every snapshot [is] always a full backup" does not mean that you have to upload your whole terabytes-large drive every time. I guess what you mean is that each backup stands on its own, referring to the chunks it needs. I don't think that's what people mean when they say that each snapshot is a new full backup but rather that this is the definition of it being incremental.
  - dividuum 5y ago
    Correct. Thanks for clarifying.

lloeki 5y ago

IIRC restic operates at the file level on the source side and then stores data in blobs, and unchanged blobs are deduplicated (it operates as a content addressed store, kind of like git), which means iterative backups are possible. EDIT: see "blob" https://restic.readthedocs.io/en/latest/100_references.html#design https://restic.readthedocs.io/en/latest/100_references.html#... and "efficient" https://github.com/restic/restic#design-principles https://github.com/restic/restic#design-principles

lucb1e 5y ago

Yes, it deduplicates but does not compress. Since most large files (mp3, jpg, mp4, docx, etc.) are already compressed, this is not a big issue but it's not ideal either. If I remember correctly, another tool either does both or does compression instead of deduplication, this might have been borg or bup. In case that's something you care about.

l33tman 5y ago
Typical borg run: Original size Compressed size Deduplicated size This archive: 167.99 GB 136.78 GB 53.85 MB All archives: 2.93 TB 2.38 TB 133.66 GB Sorry about the formatting. But the compression is not completely irrelevant. Dedup of blocks between files and backups is of course the absolutely most crucial part though.
- lucb1e 5y ago
  Thanks for the real-world stats! I must say it took me a bit to read it though, let me format it for others (hopefully also narrow enough for mobile): compress data | dedup 1 backup: 168G 137G 54M Σ backups: 2.9T 2.4T 134G Please correct me if 'one backup' and 'all backups' is an incorrect interpretation of 'all archives'. I wasn't entirely sure what you mean by that but I think I get the point. So in conclusion, adding compression saves about 17% (a $10 monthly bill would be $8.28 instead, if you pay per GB) for you.
  - lucb1e 5y ago
    Another friend has a stat of 30%, by the way. Even if I'd get the better of the two values, I don't have enough systems that a backup being 2/3rds of the original data size reduces the number of backup disks I need to buy, but it's not insignificant either.
BrandoElFollito 5y ago
Compression may help but that would be best case 10-30% (which is worth or not depending on the use case). Deduplication means ~20x smaller backups so this is absolutely key for me.

dividuum 5y ago

Yes. There's deduplication. It's using a rolling hash for chunking and then deduplicates those chunks. So every snapshot it always a full backup.

lucb1e 5y ago
To be clear, "every snapshot [is] always a full backup" does not mean that you have to upload your whole terabytes-large drive every time. I guess what you mean is that each backup stands on its own, referring to the chunks it needs. I don't think that's what people mean when they say that each snapshot is a new full backup but rather that this is the definition of it being incremental.
- dividuum 5y ago
  Correct. Thanks for clarifying.