3 ms·
Are these files not compressed on disk? I'd imagine duplicate data like this compresses very well.
by azernik 10y ago
Are these files not compressed on disk? I'd imagine duplicate data like this compresses very well.
- kevingadd 10y agoThat's assuming the on-disk compression deduplicates. ZIP does not and 7z only does in 'solid' mode. Compression is not a silver bullet for this either - if you have a repeated 8MB sequence in 8 different parts of a 250MB file, whether or not the sequence gets deduplicated depends on the compression algorithm, dictionary size, speed setting, etc. The solution to all the duplicated files is to dedup at build time.