4 ms·
Ask HN: What compression doesn't re-include the same file multiple times?
As a test, I copied a 91 megabyte creative commons mp3 file to a test folder's /1, /2 and /3 subfolder.
I then tried compressing the three directories into one compressed file. I tried various utilities. I tested every setting of winrar (including maximum dictionary size) and most settings of 7zip, as well as tar -czf from the git bash prompt.
Except for 7zip when outputing 7zip with a very large dictionary size (3840 MB) and when it output .wim which is some kind of windows packaging, all tries by all programs failed to get a file around 91 megabytes (every try resulted in the same file being included 3 times, pushing the resulting zip or .tar.gz file to 275 MB, even on maximum settings.)
Other than 7zip on such large settings, which compression formats would handle this case? Does any open compression format support it?
- pestatije 4y agoYou could consider ln as a compression tool/format...
- DemocracyFTW2 4y agothe keywords to search for include "content-addressed storage" and "rolling hash". The latter is used by rsync to avoid having to send repeating byte sequences
- elmerfud 4y agoCompression and deduplication are different things although there is some overlap. The theory behind each is quite different. The only compressor that I know that has deduplication option in it is rar 5 and higher.
- logicallee 4y agoInteresting. There was no deduplication option on the main settings screen (I am using version 5.71.) I could choose compression method as anywhere from Store to Best, and a dictionary size. I tried the "best" otpion and the largest dictionary size in my version which is 1024 MB but it didn't help. All of my rar tests showed the triple size. However, after carefully reading your comment I saw another tab called Options with "Save identical files as references" and if I checked it it did indeed get reduced to 1/3 size. I wonder why it's not checked by default? (Or even appear on the main page.) Why don't all compressors deduplicate? It seems like such an easy win. I am concerned about using a hidden page called "save identical files as references" as I would think by not being on the main General page it is a code path that isn't exercised as frequently. I am concerned about the longevity of my archives and would prefer a well-supported format that happened to deduplicate as well.
- jamesfinlayson 4y agoI don't know much about the different compression formats but they all seem to store some amount of file metadata (timestamps, permissions etc). Maybe some formats store them as part of the compressed data which would result in data loss if files were deduplicated. There are ways around that of course but maybe the .rar format doesn't lend itself well to that.
- DemocracyFTW2 4y ago> I am concerned about the longevity of my archives If you're concerned about archival longevity, and I for one certainly are, then maybe consider not to compress at all. Both compression and encryption add to obscurity and loss of redundancy in the backup. Using a widely understood file system and a very obvious arrangement of the data (for me that means: directories with dates, below a tree of files that mimics their original locations) will be a huge plus should the data have to be recovered at some point in the future. Personally I am using a slightly adapted version of https://github.com/cytopia/linux-timemachine https://github.com/cytopia/linux-timemachine for this task. You do get de-duplication for the file transfer, but each file is written as it was on the target. You'll get a timestamped directory for each time a backup was run. Like MacOS' timemachine, the script uses hard links to de-duplicate identical files across different timestamped directories so the overall space requirement for that incremental backup you did an hour or a day later can be very small. I can certify that this setup, while it does not occupy the least conceivable amount of storage area, is very amenable to be searched and trivial to use for recovery. much better in this regard than any kind of compressed archive format which are always a pain in terms of searchability and so on.
- logicallee 4y agoGreat tip! Thanks.
- DemocracyFTW2 4y agoHope it helps!
- defrost 4y agoYou could try (depending on file system) running a {hard|soft} link dedup program (that replaces duplicate files with hardlinks or symlinks to a single data file, and then archive your tree with tar | zip using a --symlinks flag. WARNING: - I've read about this .. never used it despite storing and archiving for backup and data rention for decades. I prefer to keep file trees 'clean' (no extranous dupes) and fully store them as they are.
- mharig 4y agoI am working on a Python progrämmchen (little program) that uses a block based approach to store files in a SQLite DB. I guess I can push an alpha version to github during the next 2 weeks.