3 ms·
rsync and Rabin chunking are both based on rolling hashes determining where exactly to cryptohash. rsync operates "in place" and benefits from chunks being in c
by mindslight 11y ago
rsync and Rabin chunking are both based on rolling hashes determining where exactly to cryptohash. rsync operates "in place" and benefits from chunks being in consistent order, whereas Rabin chunking uses precomputation to draw from unordered basis chunks (eg cross-file chunking).
- zbjornson 11y agoPardon my ignorance, but I'm still not quite understanding the applications and benefits. Distributed file hosting? High-concurrency reads? Torrent-like? "file sharing network" is a bit ambiguous to me. (I seem to have broke the demo with a 100 MB file, btw.)
- mindslight 11y ago(Not my project. I'm just commenting on the general techniques) Rabin chunking should get you quick cross-file partial-file deduplication. So if you're already storing a file and then download a modified version (from someone else) with changed metadata in the header, only the differing data should be transferred/stored.