3 ms·

Yes, ZFS deduplication is not very useful when trying to deduplicate encrypted and compressed video files. Encryption is supposed to make the contents of the fi

by ccrush 12y ago

Yes, ZFS deduplication is not very useful when trying to deduplicate encrypted and compressed video files. Encryption is supposed to make the contents of the file look random. Compression is itself a form of in-file deduplication. Where ZFS deduplication will be extremely useful: on a SAN in a video editing studio where originals are kept in raw format. Even then, file-level and block-level dedups will probably be the least effective means of deduplication. What we need is byte-level deduplication which computes anchor points from which data ought to be deduplicated. Even then, it sounds like the dedup feature would be scripted so that it be a part of the tooling, and would be updated with the team's projects. I would go as far as to say the author chose a deliberately eye-catching headline and coupled up a few Linux commands with some "hilarious" GIFs to address the wannabe geeks out there while displaying a staggering pile of shamelessness to even the remotely competent of his blog's visitors.

matrixagent 12y ago
The files are not encrypted. I mention the aspect of doing it with files that are already compressed and that I never expected it to work for that very reason. I just wanted to to for once actually do it and I don't see why you need to get so mad about that.
- ccrush 12y ago
  Unless you stripped the DRM from the files, yes, they are encrypted. At least according to the Wikipedia page on FairPlay [http://en.wikipedia.org/wiki/FairPlay http://en.wikipedia.org/wiki/FairPlay], all the iTunes files are encrypted. Also, wouldn't proving it doesn't work by setting up a deliberately flawed scenario amount to a bit of blogsturbation? Should I write an article about "finding out whether my computer will turn on even if it's unplugged"? Answer: no. But read my article about me TRYING it just to be sure. I think you should write a follow-up article where you splice together raw video files and include a similar segment in all the different files. Then, put that through the dedup test. I would actually be interested to see the ability of ZFS implementations to find worthwhile anchor points within files and do smart byte-level deduplication.
- theoh 12y ago
  I can see where he's coming from. Any differences in the data caused by encryption or compression of the final video files don't even come into play here unless the raw video data they are produced from is bit-for-bit identical from one episode to the next, say in a precise lossless format like CCIR 601. Maybe this is true at the production company but I wouldn't be surprised if it isn't. It would actually be somewhat interesting to know what the typical workflow at the production company is for gluing the title sequence together with the rest of the show. Your blog post doesn't do enough to clarify these issues.

by ccrush 12y ago

matrixagent 12y ago

The files are not encrypted. I mention the aspect of doing it with files that are already compressed and that I never expected it to work for that very reason. I just wanted to to for once actually do it and I don't see why you need to get so mad about that.

ccrush 12y ago
Unless you stripped the DRM from the files, yes, they are encrypted. At least according to the Wikipedia page on FairPlay [http://en.wikipedia.org/wiki/FairPlay http://en.wikipedia.org/wiki/FairPlay], all the iTunes files are encrypted. Also, wouldn't proving it doesn't work by setting up a deliberately flawed scenario amount to a bit of blogsturbation? Should I write an article about "finding out whether my computer will turn on even if it's unplugged"? Answer: no. But read my article about me TRYING it just to be sure. I think you should write a follow-up article where you splice together raw video files and include a similar segment in all the different files. Then, put that through the dedup test. I would actually be interested to see the ability of ZFS implementations to find worthwhile anchor points within files and do smart byte-level deduplication.
theoh 12y ago
I can see where he's coming from. Any differences in the data caused by encryption or compression of the final video files don't even come into play here unless the raw video data they are produced from is bit-for-bit identical from one episode to the next, say in a precise lossless format like CCIR 601. Maybe this is true at the production company but I wouldn't be surprised if it isn't. It would actually be somewhat interesting to know what the typical workflow at the production company is for gluing the title sequence together with the rest of the show. Your blog post doesn't do enough to clarify these issues.