5 ms·

Hyperdrive – a novel file sharing network

42 points by sgrytoyr 11y ago

nine_k 11y ago
«A file sharing network based on rabin file chunking and append only feeds of data verified by merkle trees.» As far as I understand, this is about efficient verified replication of files, so that only smallest changed chunks are actually replicated (unlike e.g. rsync).
- mindslight 11y ago
  rsync and Rabin chunking are both based on rolling hashes determining where exactly to cryptohash. rsync operates "in place" and benefits from chunks being in consistent order, whereas Rabin chunking uses precomputation to draw from unordered basis chunks (eg cross-file chunking).
  - zbjornson 11y ago
    Pardon my ignorance, but I'm still not quite understanding the applications and benefits. Distributed file hosting? High-concurrency reads? Torrent-like? "file sharing network" is a bit ambiguous to me. (I seem to have broke the demo with a 100 MB file, btw.)
    - mindslight 11y ago
      (Not my project. I'm just commenting on the general techniques) Rabin chunking should get you quick cross-file partial-file deduplication. So if you're already storing a file and then download a modified version (from someone else) with changed metadata in the header, only the differing data should be transferred/stored.
out_of_protocol 11y ago
Any idea how peer discovering work? Can't find any mention of it
- mafintosh 11y ago
  it's in the README examples. you can use discovery-channel, https://github.com/maxogden/discovery-channel https://github.com/maxogden/discovery-channel to do peer discovery which in turn uses multicast-dns and the bittorrent dht to find peers.
mankash666 11y ago
I'm unable to get the demo [1] to work. I'm using Firefox 42.0 on Mac OS 10.11.12 [1]: http://mafintosh.github.io/hyperdrive/ http://mafintosh.github.io/hyperdrive/
- mafintosh 11y ago
  yes i haven't tested the demo in firefox yet. there is an open issue for that on the hyperdrive repo. it should work in chrome though and there is no reason it shouldn't work in firefox once i find the bug.
Fice 11y ago
Looks similar to IPFS (https://ipfs.io/ https://ipfs.io/) but with less ambitious goals. And the IPFS team is also working on JS implementation that will run in browser: https://github.com/ipfs/js-ipfs https://github.com/ipfs/js-ipfs.
deleted 11y ago
[deleted]

42 points by sgrytoyr 11y ago

«A file sharing network based on rabin file chunking and append only feeds of data verified by merkle trees.» As far as I understand, this is about efficient verified replication of files, so that only smallest changed chunks are actually replicated (unlike e.g. rsync).

mindslight 11y ago
rsync and Rabin chunking are both based on rolling hashes determining where exactly to cryptohash. rsync operates "in place" and benefits from chunks being in consistent order, whereas Rabin chunking uses precomputation to draw from unordered basis chunks (eg cross-file chunking).
- zbjornson 11y ago
  Pardon my ignorance, but I'm still not quite understanding the applications and benefits. Distributed file hosting? High-concurrency reads? Torrent-like? "file sharing network" is a bit ambiguous to me. (I seem to have broke the demo with a 100 MB file, btw.)
  - mindslight 11y ago
    (Not my project. I'm just commenting on the general techniques) Rabin chunking should get you quick cross-file partial-file deduplication. So if you're already storing a file and then download a modified version (from someone else) with changed metadata in the header, only the differing data should be transferred/stored.

out_of_protocol 11y ago

Any idea how peer discovering work? Can't find any mention of it

mafintosh 11y ago
it's in the README examples. you can use discovery-channel, https://github.com/maxogden/discovery-channel https://github.com/maxogden/discovery-channel to do peer discovery which in turn uses multicast-dns and the bittorrent dht to find peers.

mankash666 11y ago

I'm unable to get the demo [1] to work. I'm using Firefox 42.0 on Mac OS 10.11.12 [1]: http://mafintosh.github.io/hyperdrive/ http://mafintosh.github.io/hyperdrive/

mafintosh 11y ago
yes i haven't tested the demo in firefox yet. there is an open issue for that on the hyperdrive repo. it should work in chrome though and there is no reason it shouldn't work in firefox once i find the bug.

Fice 11y ago

Looks similar to IPFS (https://ipfs.io/ https://ipfs.io/) but with less ambitious goals. And the IPFS team is also working on JS implementation that will run in browser: https://github.com/ipfs/js-ipfs https://github.com/ipfs/js-ipfs.

deleted 11y ago

[deleted]