3 ms·

If you were going to do this for science, a good start would be to analyze the intro of shows in question, using rudimentary tools such as md5sum on a per-block

If you were going to do this for science, a good start would be to analyze the intro of shows in question, using rudimentary tools such as md5sum on a per-block basis. You would then quickly discover that the chances of huge gains would be small, without ever even looking into the ZFS aspects.

ccrush 12y ago
ZFS supports byte-level deduplication where it can compute anchor points so that it works on unaligned data. However, quite a few a few assumptions about the entropy of the data set could have been inferred from the "iTunes TV Show" file choice, which clearly means the files would be compressed and encrypted.
- TheCondor 12y ago
  Is that new? Zfs has supported block deduplication for a few years but I'm unaware if it supporting byte level deduplication.
  - ccrush 12y ago
    https://blogs.oracle.com/bonwick/entry/zfs_dedup https://blogs.oracle.com/bonwick/entry/zfs_dedup
    - lkateley 12y ago
      Not new.. dated 2009. Oracle seems to be doing alot of the lately where they are posting really old collaterial
- ryao 12y ago
  This is incorrect. Byte-level deduplication does work with anchor points, but it is extremely expensive. Jeff decided to let the application do byte-level deduplication and implement block-level deduplication instead. This gave us something that was reasonable to implement at the filesystem level.