4 ms·
There are many rsync-based backup systems like this that assume you back up many times and don't restore often, so backups are efficient (deltas, etc) but resto
by modeless 10y ago
There are many rsync-based backup systems like this that assume you back up many times and don't restore often, so backups are efficient (deltas, etc) but restoring is not (just download the whole thing). I'd like to use an rsync-like mechanism for software distribution, where the problem is reversed: the software is updated infrequently, but there are many clients that all need to download updates, so "restoring" should be efficient. Is there any open source solution for that? Preferably where the server can be dumb, like an S3 bucket.
- phireal 10y agoLike delta RPMs [0]? Software updates are distributed as diffs and applied to the existing package, so only the differences in the new packages need to be sent each time. This is computationally ineffficient (calculating and applying the delta RPM) but bandwidth efficient (smaller transfers). It depends on what you mean by efficient. [0] http://www.mpipks-dresden.mpg.de/~mueller/docs/suse10.2/html/opensuse-manual_en/manual/sec.rpm.delta.html http://www.mpipks-dresden.mpg.de/~mueller/docs/suse10.2/html... First reference I found on Google
- rakoo 10y agoYou might want to have a look at zsync (http://zsync.moria.org.uk/ http://zsync.moria.org.uk/), which is exactly a reversed rsync: the seed server calculates a signatures file over a file, the clients download that signatures file (through any means, but usually HTTP) and calculate a diff on their side; once they have instructions they can download only the parts of what they need through HTTP Ranges. Which means that any dumb HTTP storage that respects Ranges will work. The big advantage over rsync, on top of being available over HTTP, is that the server only calculates the signatures file once for the life of the content, instead of calculating it for each and every client. I know some distributions use that, but it's not very well known... yet ?
- modeless 10y agoCool! Zsync looks like exactly what I want, except that it doesn't sync directory trees, only individual files.
- andyjpb 10y agoFrom the same era: http://www.benedikt-stockebrand.de/rdist-intro_en.html http://www.benedikt-stockebrand.de/rdist-intro_en.html
- MaBu 10y agoTry [BSDiff](http://www.daemonology.net/bsdiff/ http://www.daemonology.net/bsdiff/) it is meant for diffing binary files and it was used for incremental updates in Chrome. Now it seems they replaced it with their own algorithm [Courgette](https://www.chromium.org/developers/design-documents/software-updates-courgette https://www.chromium.org/developers/design-documents/softwar...). I use it to upload new versions of my java app since I have slow upload. Instead of always uploading ~40 MB JARs I just upload 1.8 MB diff which is of course much faster.
- extra88 10y agoIf you also use rsync for the restoring from rsnapshot-managed backups, they are just as efficient in terms of the data transfer being just byte-level deltas as the backups. But rsync can't work with a serverless data storage like S3.