5 ms·

Compress the folder with your favorite zip program and the compression ratio would be very good approximation of performance gain you will get out of zfs dedup.

Compress the folder with your favorite zip program and the compression ratio would be very good approximation of performance gain you will get out of zfs dedup. Web archiving is another application that can benefit from this. You can crawl the same website 10 times a day and just store all of the files as it is.

bensummers 17y ago
Compression works inside relatively short chunks of the files, compared to huge amounts of storage. Plus if you use zip, you're only looking at similarities within individual files. I don't think the results would be useful. Incidentally, ZFS also has an option to compress the data. You have a choice of a fast but not so wonderful algorithm, or gzip level 1 to 9. Since de dupe is at the block level, I believe you can combine the two.
- atamyrat 17y ago
  Why it wouldn't be useful? While it might be true for some compression formats/programs, it is not the case for .tar.*, as the directory is archived to the single file first (tar), and then compressed. So if you have similarities within 2 different files, that will be exploited. I think to make something work for "large blocks" that already does a good job for "small chunks" is just finding the right values for parameters of compression algorithm used.
  - bensummers 17y ago
    It's not a useful measure because it's not the same thing at all. Dedup only works if the block is aligned to a ZFS block size. Compression will find blocks which have no particular alignment.
    - atamyrat 17y ago
      If they were same thing, you'd get the exact result, but not an approximation. tar.x will compress chunks of data that is smaller than zfs block size. ZFS dedup + ZFS compression will compress them as well, so what's the problem?
patrickgzill 17y ago
I don't think you understand how dedupe works for cases where it is a benefit. Imagine you have 10 VMware virtual machines, and all the VM files are stored on a ZFS server with dedupe support. Let's say that the base install image is 2GB in size of OS, GUI, Java libraries, etc. Non-dedupe scenario means 20GB of storage is used. Dedupe scenario means maybe, 2.5GB of storage is used due to slight differences in the way the blocks are arranged, etc. Now you add an 11th VM, same OS, same base setup. Storage goes up by maybe 100MB even though the VM has the same 2GB of files. Do you see how zipping a file would not show you that effect?
atamyrat 17y ago
note to self: Someone else compared ZFS dedup to compression on Slashdot and it scored 5/Interesting. On HN, it is downvoted and reply with technically wrong claim gets 4 points.

bensummers 17y ago

Compression works inside relatively short chunks of the files, compared to huge amounts of storage. Plus if you use zip, you're only looking at similarities within individual files. I don't think the results would be useful. Incidentally, ZFS also has an option to compress the data. You have a choice of a fast but not so wonderful algorithm, or gzip level 1 to 9. Since de dupe is at the block level, I believe you can combine the two.

atamyrat 17y ago
Why it wouldn't be useful? While it might be true for some compression formats/programs, it is not the case for .tar.*, as the directory is archived to the single file first (tar), and then compressed. So if you have similarities within 2 different files, that will be exploited. I think to make something work for "large blocks" that already does a good job for "small chunks" is just finding the right values for parameters of compression algorithm used.
- bensummers 17y ago
  It's not a useful measure because it's not the same thing at all. Dedup only works if the block is aligned to a ZFS block size. Compression will find blocks which have no particular alignment.
  - atamyrat 17y ago
    If they were same thing, you'd get the exact result, but not an approximation. tar.x will compress chunks of data that is smaller than zfs block size. ZFS dedup + ZFS compression will compress them as well, so what's the problem?

patrickgzill 17y ago

I don't think you understand how dedupe works for cases where it is a benefit. Imagine you have 10 VMware virtual machines, and all the VM files are stored on a ZFS server with dedupe support. Let's say that the base install image is 2GB in size of OS, GUI, Java libraries, etc. Non-dedupe scenario means 20GB of storage is used. Dedupe scenario means maybe, 2.5GB of storage is used due to slight differences in the way the blocks are arranged, etc. Now you add an 11th VM, same OS, same base setup. Storage goes up by maybe 100MB even though the VM has the same 2GB of files. Do you see how zipping a file would not show you that effect?

atamyrat 17y ago

note to self: Someone else compared ZFS dedup to compression on Slashdot and it scored 5/Interesting. On HN, it is downvoted and reply with technically wrong claim gets 4 points.