3 ms·
Don't you want to dedup first, then compress?
by bradknowles 7y ago
Don't you want to dedup first, then compress?
- frio 7y agoProbably, yes. Sorry, I should’ve said “in no particular order” — this is part of the reason why I haven’t tried to tackle building something myself :).
- cben 7y agoWell, with `gzip --rsyncable` and similar (most compression tools have such a mode), it's technically possible to compress first then dedup! To be precise it's "chunk first, then compress", which still admits a 2nd "chunk, then dedup" pass to reuse data, but not as efficiently as a combined "chunk+dedup+compress each chunk"... `gzip --rsyncable` and friends are mostly useful for when you already have compressed files on your filesystem, and want to sync/backup them byte-for-byte. Of course the benefit is lost unless most compressed files in the world are produced with such mode, and sadly most aren't :-( The author of `zsync` did various experiments confirming gzip --rsyncable then sync is sub-optimal, and implemented a somewhat crazy "look inside" approach that can sync compressed files byte-for-byte AND very efficiently: > gzip --rsync does fairly well, with both rsync and zsync transferring about 410kB at the optimum point. zsync with the look-inside method does much better than either of these, with as little as 140K transferred. > -- http://zsync.moria.org.uk/paper/ch03s03.html http://zsync.moria.org.uk/paper/ch03s03.html IIUC though it's more of a "2-files sync" scenario like rsync, not applicable to "chunk everything then dedup" approach of borg and similar tools.