3 ms·

Interesting. There was no deduplication option on the main settings screen (I am using version 5.71.) I could choose compression method as anywhere from Store t

Interesting. There was no deduplication option on the main settings screen (I am using version 5.71.) I could choose compression method as anywhere from Store to Best, and a dictionary size. I tried the "best" otpion and the largest dictionary size in my version which is 1024 MB but it didn't help. All of my rar tests showed the triple size. However, after carefully reading your comment I saw another tab called Options with "Save identical files as references" and if I checked it it did indeed get reduced to 1/3 size. I wonder why it's not checked by default? (Or even appear on the main page.) Why don't all compressors deduplicate? It seems like such an easy win. I am concerned about using a hidden page called "save identical files as references" as I would think by not being on the main General page it is a code path that isn't exercised as frequently. I am concerned about the longevity of my archives and would prefer a well-supported format that happened to deduplicate as well.

jamesfinlayson 4y ago
I don't know much about the different compression formats but they all seem to store some amount of file metadata (timestamps, permissions etc). Maybe some formats store them as part of the compressed data which would result in data loss if files were deduplicated. There are ways around that of course but maybe the .rar format doesn't lend itself well to that.
DemocracyFTW2 4y ago
> I am concerned about the longevity of my archives If you're concerned about archival longevity, and I for one certainly are, then maybe consider not to compress at all. Both compression and encryption add to obscurity and loss of redundancy in the backup. Using a widely understood file system and a very obvious arrangement of the data (for me that means: directories with dates, below a tree of files that mimics their original locations) will be a huge plus should the data have to be recovered at some point in the future. Personally I am using a slightly adapted version of https://github.com/cytopia/linux-timemachine https://github.com/cytopia/linux-timemachine for this task. You do get de-duplication for the file transfer, but each file is written as it was on the target. You'll get a timestamped directory for each time a backup was run. Like MacOS' timemachine, the script uses hard links to de-duplicate identical files across different timestamped directories so the overall space requirement for that incremental backup you did an hour or a day later can be very small. I can certify that this setup, while it does not occupy the least conceivable amount of storage area, is very amenable to be searched and trivial to use for recovery. much better in this regard than any kind of compressed archive format which are always a pain in terms of searchability and so on.
- logicallee 4y ago
  Great tip! Thanks.
  - DemocracyFTW2 4y ago
    Hope it helps!

jamesfinlayson 4y ago

I don't know much about the different compression formats but they all seem to store some amount of file metadata (timestamps, permissions etc). Maybe some formats store them as part of the compressed data which would result in data loss if files were deduplicated. There are ways around that of course but maybe the .rar format doesn't lend itself well to that.

DemocracyFTW2 4y ago

> I am concerned about the longevity of my archives If you're concerned about archival longevity, and I for one certainly are, then maybe consider not to compress at all. Both compression and encryption add to obscurity and loss of redundancy in the backup. Using a widely understood file system and a very obvious arrangement of the data (for me that means: directories with dates, below a tree of files that mimics their original locations) will be a huge plus should the data have to be recovered at some point in the future. Personally I am using a slightly adapted version of https://github.com/cytopia/linux-timemachine https://github.com/cytopia/linux-timemachine for this task. You do get de-duplication for the file transfer, but each file is written as it was on the target. You'll get a timestamped directory for each time a backup was run. Like MacOS' timemachine, the script uses hard links to de-duplicate identical files across different timestamped directories so the overall space requirement for that incremental backup you did an hour or a day later can be very small. I can certify that this setup, while it does not occupy the least conceivable amount of storage area, is very amenable to be searched and trivial to use for recovery. much better in this regard than any kind of compressed archive format which are always a pain in terms of searchability and so on.

logicallee 4y ago
Great tip! Thanks.
- DemocracyFTW2 4y ago
  Hope it helps!