2 ms·
Compression works inside relatively short chunks of the files, compared to huge amounts of storage. Plus if you use zip, you're only looking at similarities wit
by bensummers 17y ago
Compression works inside relatively short chunks of the files, compared to huge amounts of storage. Plus if you use zip, you're only looking at similarities within individual files. I don't think the results would be useful.
Incidentally, ZFS also has an option to compress the data. You have a choice of a fast but not so wonderful algorithm, or gzip level 1 to 9. Since de dupe is at the block level, I believe you can combine the two.
- atamyrat 17y agoWhy it wouldn't be useful? While it might be true for some compression formats/programs, it is not the case for .tar.*, as the directory is archived to the single file first (tar), and then compressed. So if you have similarities within 2 different files, that will be exploited. I think to make something work for "large blocks" that already does a good job for "small chunks" is just finding the right values for parameters of compression algorithm used.
- bensummers 17y agoIt's not a useful measure because it's not the same thing at all. Dedup only works if the block is aligned to a ZFS block size. Compression will find blocks which have no particular alignment.
- atamyrat 17y agoIf they were same thing, you'd get the exact result, but not an approximation. tar.x will compress chunks of data that is smaller than zfs block size. ZFS dedup + ZFS compression will compress them as well, so what's the problem?