3 ms·
Well, with `gzip --rsyncable` and similar (most compression tools have such a mode), it's technically possible to compress first then dedup! To be precise it's
by cben 7y ago
Well, with `gzip --rsyncable` and similar (most compression tools have such a mode), it's technically possible to compress first then dedup!
To be precise it's "chunk first, then compress", which still admits a 2nd "chunk, then dedup" pass to reuse data, but not as efficiently as a combined "chunk+dedup+compress each chunk"...
`gzip --rsyncable` and friends are mostly useful for when you already have compressed files on your filesystem, and want to sync/backup them byte-for-byte. Of course the benefit is lost unless most compressed files in the world are produced with such mode, and sadly most aren't :-(
The author of `zsync` did various experiments confirming gzip --rsyncable then sync is sub-optimal, and implemented a somewhat crazy "look inside" approach that can sync compressed files byte-for-byte AND very efficiently:
> gzip --rsync does fairly well, with both rsync and zsync transferring about 410kB at the optimum point. zsync with the look-inside method does much better than either of these, with as little as 140K transferred.
> -- http://zsync.moria.org.uk/paper/ch03s03.html http://zsync.moria.org.uk/paper/ch03s03.html
IIUC though it's more of a "2-files sync" scenario like rsync, not applicable to "chunk everything then dedup" approach of borg and similar tools.