4 ms·
Bit-sync: synchronizing arbitrary data using rsync algorithm in pure JavaScript
- zmj 6y agoThe rsync algorithm is pretty cool. I implemented it in C# a couple of years ago: https://github.com/zmj/rsync-delta https://github.com/zmj/rsync-delta
- cshipley 6y agoDoes this only work with files or can any source be used as long as it is a Stream? Basically, I'd love to have something like this for arbitrary data, not just files. Also, looks like the README has a broken link: https://librsync.github.io/rdiff.html https://librsync.github.io/rdiff.html
- zmj 6y agoYep, Streams or Pipes for IO. No seeking, forward-only reads - using this with network in/out was my goal. Thanks for pointing out the broken link. Looks like it's here now: https://librsync.github.io/page_rdiff.html https://librsync.github.io/page_rdiff.html
- hardwaresofton 6y agoRelated (rsync in the browser): https://webdeltasync.github.io/ https://webdeltasync.github.io/
- claytongulick 6y agoThis is a cool project! Looks like they use my bit-sync lib for the actual rsync part [1] which is neat. I should modernize this lib so that it's easier for them to add as a dependency. https://github.com/WebDeltaSync/WebRsync/blob/master/public/js/bit-sync.js https://github.com/WebDeltaSync/WebRsync/blob/master/public/...
- hardwaresofton 6y agoThanks for making bit-sync, it's awesome! It immediately got me thinking about cool things you could do in the browser with it! It's great that they didn't try to reinvent the wheel and used your work. The WebRSync doesn't seem to be particularly active so I don't think you have to worry about it, but if you find time to modernize your lib that'd be awesome -- just need the JS community to get their excited about doing cool stuff with rsync in the browser (sprinkle some WASM in there maybe somewhere).
- claytongulick 6y agoYep, the unexpected attention this received today has lit a fire under me. I'll get on it.
- tyingq 6y ago"The md5 algorithm needs to be rewritten with performance in mind" Poked around a little, and this area seems a bit under-researched since MD5 went out of favor for many use cases. For example, the SubtleCrypto digest doesn't support it. There's this WASM implementation[1], that shows 2-37 MB/sec, presumably on fairly modern hardware. But, I can also find an old pure Javascript implementation[2] that shows 5-7 MB/sec on an ancient Pentium4 running Opera[2]. Makes you wonder how it would fare with a recent Chrome/V8 and hardware. There's online tests, but with small inputs and Chrome is fast enough that I get a lot of NaN/Infinity. Maybe it would be better to use SHA-1 or some other digest supported by the built-in browser SubtleCrypto package. Wouldn't technically be rsync, but this package seems to be intended for use on both sides. [1] https://www.npmjs.com/package/md5-wasm https://www.npmjs.com/package/md5-wasm [2] http://www.myersdaily.org/joseph/javascript/md5-speed-test.html http://www.myersdaily.org/joseph/javascript/md5-speed-test.h...
- claytongulick 6y agoHi, author here. It's funny you mention this, I was just revisiting this lib and thinking that SubtleCrypto would be a better option. Also interesting idea to use the md5-wasm, thanks for the link. It's been years since I made this, probably due for a little love.
- blixt 6y agoAlso check out the Rabin rolling hash algorithm for smart chunking of files which can be useful to either dedupe dynamic content or efficiently sync files.
- londons_explore 6y agoI'd like to see something like this used for big JavaScript bundles. Many webapps now have 2MB+ of JavaScript, and lots of web companies do daily or even hourly software releases, which effectively means every time I visit your site I have to download many megabytes of JavaScript which takes many seconds. It's frustrating. Diffing the JavaScript and sending just the changed lines of code would save hundreds of millions of people a few seconds each day, and is totally worth it.
- tyingq 6y agoHTTP already has range requests, so it seems like you could do this in a relatively straightforward way.
- claytongulick 6y agoThat is a very interesting idea. I've been thinking about what I should do in a V2 of this lib.
- kevmo314 6y agoI really like this idea, but I'm not sure if it would work in practice. It's basically working against webpack's code splitting because those are tagged with hashes. Additionally, the first load will still be quite heavy, whereas with code splitting at least there's some semblance of a smaller initial load. Since it's not native to the browser, it would have a nontrivial initialization overhead as well. I think it could be a cool idea if it's offered as a static file hosting as a service kind of deal that requires zero config, but then again if you're the kind of person that wants a zero config static file hosting, you're probably not pushing daily/hourly software releases? It would be neat to have something like this more transparently implemented though. Seeing a 2MB+ module rebuild for a minor typo fix is aggravating.
- londons_explore 6y agoOne possible implementation: * For each bundle, webpack produces the compiled bundle and a seperate diff file between the latest release and every previously published release. Filenames could be [oldhash]-[newhash].js * The webpack loader can check if the browser has that specific module cached (I believe service workers are able to inspect the http cache like this), and then based on the version that is cached, request the correct filename for a diff to the latest version. * Then apply the diff and load the javascript as normal. I just did a quick test with a webapp of mine, and a small diff file made with the rsync algorithm (via rdiff-backup) of a webpack bundle was just 150 bytes.