5 ms·

Hyperswarm is a Kademlia DHT with hole-punching builtin. It's designed to arrange p2p connections. For context as to why you'd want this, when you create a p2p

by pfraze 8y ago

Hyperswarm is a Kademlia DHT with hole-punching builtin. It's designed to arrange p2p connections. For context as to why you'd want this, when you create a p2p connection (like with WebRTC) you typically need a "signaling server" to arrange the connection. Hyperswarm replaces that server with a distributed signaling system. We built it for the dat protocol, but it's reusable in a lot of other contexts. (In fact, the creator of the Manyverse mobile app which released yesterday [1] has voiced an interest in using to Hyperswarm [2].) 1. https://news.ycombinator.com/item?id=18065567 https://news.ycombinator.com/item?id=18065567 2. https://twitter.com/andrestaltz/status/1044973699940659200 https://twitter.com/andrestaltz/status/1044973699940659200

k__ 8y ago
First time I heared about such things was with Overnet many years ago, how come it never caught on? Or asked differently, what makes Hyperswarm better than Overnet? :)
- brad0 8y ago
  It could be the slow response times. With other DHTs such as the BitTorrent DHT it takes a long time to setup a connection to a peer.
  - k__ 8y ago
    AFAIK Overnet was build on Kademlia
- pfraze 8y ago
  I'd be interested to hear other perspectives on this, but I think p2p computing in general got skipped over when the cloud became a thing. DHTs have had steady use in the p2p world though, including with BitTorrent. DHTs are not without their problems. I know some other projects in our space decided it wasn't tenable to solve the Sybil attack and so they created a peer-routing system that bootstraps off of a blockchain. As brad0 pointed out, there have also been performance issues in some deployments. We're going to iterate on the security and performance and see how it goes.
  - kodablah 8y ago
    > I'd be interested to hear other perspectives on this, but I think p2p computing in general got skipped over when the cloud became a thing There are few commercial benefits/incentives for a DHT provider and few guarantees p2p tech can give a company requiring an always-on central presence. > I know some other projects in our space decided it wasn't tenable to solve the Sybil attack and so they created a peer-routing system that bootstraps off of a blockchain. One approach is to do the S/Kademlia thing where you require all peers to generate IDs (hash of a key) and require the it start with a certain number of 0 bits and then require another random X that, when xor'd w/ the ID, starts with a certain number of 0 bits.
    - pfraze 8y ago
      > There are few commercial benefits/incentives for a DHT provider and few guarantees p2p tech can give a company requiring an always-on central presence. There are quite a few commercial benefits! It's just that nobody's properly banked on them yet. P2P distributes costs, simplifies configuration & ops, and improves data privacy. For the politically-minded, P2P also distributes power more equitably by moving business logic to user devices. P2P will not remove the need for always-on presences. You need a peer of last resort. However, running a peer will be cheaper and easier than running a server. > One approach is to do the S/Kademlia thing where you require all peers to generate IDs (hash of a key) and require the it start with a certain number of 0 bits and then require another random X that, when xor'd w/ the ID, starts with a certain number of 0 bits. We're looking at the crypto-puzzle approach, but there needs to be a work-requirement asymmetry between honest nodes and attackers, which I think is only the case with the Eclipse attacks and not the Sybil attacks. Otherwise, the puzzles only provide security if honest nodes outspend attackers, and we don't want to turn our userbase into mining farms.
      - tracker1 8y ago
        Beyond that, there's nothing stopping a commercial entity from ensuring that there are enough baseline hosts in the DHT network from always on.
- wesleytodd 8y ago
  I hadn't heard of Overnet, so here is a wiki for others like me: https://en.wikipedia.org/wiki/Overnet https://en.wikipedia.org/wiki/Overnet That said, it seems like "what makes it better" is that it hasn't received a take-down because this isn't attempting to be a solution for copyright circumvention ;)
kodablah 8y ago
Had y'all considered using any of the js-libp2p (i.e. ipfs) stuff for this? I have used Go DHT lib w/ success and have used the connecting (not participating) part of the JS DHT. I will admit the dozens of JS modules all interconnected in the loose libp2p way is a bit annoying as is the fact they aren't all completed yet. Also, was any effort made to make this work in the browser sans browserify? Meaning, can I replace the networking component and maybe put in my own webrtc connector impls and still leverage all the kad stuff ala peer buckets/closeness?
- pfraze 8y ago
  The majority of Hyperswarm's code was written before libp2p. We had been sitting on it and waiting for the right time to deploy. We're not against working/merging with libp2p, but since we do like being able to iterate quickly within our team for now. This project isn't possible in current browsers. Browsers just don't expose the interfaces needed. That's why we work on https://beakerbrowser.com/ https://beakerbrowser.com/, which is the main project that Hyperswarm was built for.
  - kodablah 8y ago
    > This project isn't possible in current browsers. Browsers just don't expose the interfaces needed. While I agree on the networking front, with pluggable bidirectional networking impl that others can write, the DHT still has value. I am familiar w/ the Beaker project and definitely appreciate all of y'all's work (I did a much smaller PoC back in the day [0] when I thought MaidSafe had legs). I also definitely understand/appreciate the simplicity of some of the hyperswarm code and also understand fears of overabstraction for external use cases. 0 - https://github.com/cretz/safe-poc-browser https://github.com/cretz/safe-poc-browser
  - WorldMaker 8y ago
    The lack of browser support is the biggest stop to getting PWAs to better decentralized/robust-offline. Stronger PWA support would help a lot, especially in trying to minimize the number of wrappers like Electron and Cordova in the wild. Certainly most of the interfaces needed aren't high on web platform priorities, and it may not be Beaker's intent yet to push on WHATWG to adopt these technologies more platform-wide, but even the ability to bootstrap the DHT in an ugly, but entirely browser-capable fashion (a dumb REST API bootstrap call to known HTTP endpoints, as one off-the-cuff idea) could do a lot for the kinds of decentralized applications that could be built entirely as PWA with no other application wrapper needed. libp2p seems to have this goal in mind on their issue tracker, but so far as skimming the issues, I don't think I've seen a timeline for releasing it.
smaddock 8y ago
I see that the DHT relies on a package called 'ipv4-peers' [1]. Does it also support IPv6 peers? I know there's a BitTorrent BEP for IPv6 support, but I'm not quite sure if that relates to this DHT implementation. [1] https://github.com/hyperswarm/dht/blob/master/index.js#L4 https://github.com/hyperswarm/dht/blob/master/index.js#L4
- mafintosh 8y ago
  ipv6 is on the roadmap :) i just started spec'ing that out today actually.
bennofs 8y ago
How does the hole punching work? Is there a high level overview of the difference between this implementation and textbook Kademlia? I am not sure in which part of the code to look for this (is the hole punching part of the dht package?) so any pointers are appreciated
- mafintosh 8y ago
  Yea holepunching is implemented in the dht-rpc package. There is an explanatory comment there as well, https://github.com/mafintosh/dht-rpc/blob/master/lib/io.js#L47-L68 https://github.com/mafintosh/dht-rpc/blob/master/lib/io.js#L... (hope that helps!) We are planning to blog more about how the internals work. The "first class" hole punching is def one of the features I'm most exciting about.
fenwick67 8y ago
How does this package compare with https://www.npmjs.com/package/hyperdht https://www.npmjs.com/package/hyperdht ? Both seem to depend on https://www.npmjs.com/package/dht-rpc https://www.npmjs.com/package/dht-rpc for most of the work.
- mafintosh 8y ago
  Author of both here. I need to update the README of hyperdht, but we basically just iterated that a bit and moved it hyperswarm/dht to as it's part of the stack now. I'll update the hyperdht README to reflect that.
  - TACIXAT 8y ago
    Curious for more details on how the dht signaling / hole punching works. I'm working on a peer to peer chat and file sharing system right now and upnp / nat pmp are pretty wack. I'd be interested in using it if there ever is a Go port.

by pfraze 8y ago

k__ 8y ago

First time I heared about such things was with Overnet many years ago, how come it never caught on? Or asked differently, what makes Hyperswarm better than Overnet? :)

brad0 8y ago
It could be the slow response times. With other DHTs such as the BitTorrent DHT it takes a long time to setup a connection to a peer.
- k__ 8y ago
  AFAIK Overnet was build on Kademlia
pfraze 8y ago
I'd be interested to hear other perspectives on this, but I think p2p computing in general got skipped over when the cloud became a thing. DHTs have had steady use in the p2p world though, including with BitTorrent. DHTs are not without their problems. I know some other projects in our space decided it wasn't tenable to solve the Sybil attack and so they created a peer-routing system that bootstraps off of a blockchain. As brad0 pointed out, there have also been performance issues in some deployments. We're going to iterate on the security and performance and see how it goes.
- kodablah 8y ago
  > I'd be interested to hear other perspectives on this, but I think p2p computing in general got skipped over when the cloud became a thing There are few commercial benefits/incentives for a DHT provider and few guarantees p2p tech can give a company requiring an always-on central presence. > I know some other projects in our space decided it wasn't tenable to solve the Sybil attack and so they created a peer-routing system that bootstraps off of a blockchain. One approach is to do the S/Kademlia thing where you require all peers to generate IDs (hash of a key) and require the it start with a certain number of 0 bits and then require another random X that, when xor'd w/ the ID, starts with a certain number of 0 bits.
  - pfraze 8y ago
    > There are few commercial benefits/incentives for a DHT provider and few guarantees p2p tech can give a company requiring an always-on central presence. There are quite a few commercial benefits! It's just that nobody's properly banked on them yet. P2P distributes costs, simplifies configuration & ops, and improves data privacy. For the politically-minded, P2P also distributes power more equitably by moving business logic to user devices. P2P will not remove the need for always-on presences. You need a peer of last resort. However, running a peer will be cheaper and easier than running a server. > One approach is to do the S/Kademlia thing where you require all peers to generate IDs (hash of a key) and require the it start with a certain number of 0 bits and then require another random X that, when xor'd w/ the ID, starts with a certain number of 0 bits. We're looking at the crypto-puzzle approach, but there needs to be a work-requirement asymmetry between honest nodes and attackers, which I think is only the case with the Eclipse attacks and not the Sybil attacks. Otherwise, the puzzles only provide security if honest nodes outspend attackers, and we don't want to turn our userbase into mining farms.
    - tracker1 8y ago
      Beyond that, there's nothing stopping a commercial entity from ensuring that there are enough baseline hosts in the DHT network from always on.
wesleytodd 8y ago
I hadn't heard of Overnet, so here is a wiki for others like me: https://en.wikipedia.org/wiki/Overnet https://en.wikipedia.org/wiki/Overnet That said, it seems like "what makes it better" is that it hasn't received a take-down because this isn't attempting to be a solution for copyright circumvention ;)

kodablah 8y ago

Had y'all considered using any of the js-libp2p (i.e. ipfs) stuff for this? I have used Go DHT lib w/ success and have used the connecting (not participating) part of the JS DHT. I will admit the dozens of JS modules all interconnected in the loose libp2p way is a bit annoying as is the fact they aren't all completed yet. Also, was any effort made to make this work in the browser sans browserify? Meaning, can I replace the networking component and maybe put in my own webrtc connector impls and still leverage all the kad stuff ala peer buckets/closeness?

pfraze 8y ago
The majority of Hyperswarm's code was written before libp2p. We had been sitting on it and waiting for the right time to deploy. We're not against working/merging with libp2p, but since we do like being able to iterate quickly within our team for now. This project isn't possible in current browsers. Browsers just don't expose the interfaces needed. That's why we work on https://beakerbrowser.com/ https://beakerbrowser.com/, which is the main project that Hyperswarm was built for.
- kodablah 8y ago
  > This project isn't possible in current browsers. Browsers just don't expose the interfaces needed. While I agree on the networking front, with pluggable bidirectional networking impl that others can write, the DHT still has value. I am familiar w/ the Beaker project and definitely appreciate all of y'all's work (I did a much smaller PoC back in the day [0] when I thought MaidSafe had legs). I also definitely understand/appreciate the simplicity of some of the hyperswarm code and also understand fears of overabstraction for external use cases. 0 - https://github.com/cretz/safe-poc-browser https://github.com/cretz/safe-poc-browser
- WorldMaker 8y ago
  The lack of browser support is the biggest stop to getting PWAs to better decentralized/robust-offline. Stronger PWA support would help a lot, especially in trying to minimize the number of wrappers like Electron and Cordova in the wild. Certainly most of the interfaces needed aren't high on web platform priorities, and it may not be Beaker's intent yet to push on WHATWG to adopt these technologies more platform-wide, but even the ability to bootstrap the DHT in an ugly, but entirely browser-capable fashion (a dumb REST API bootstrap call to known HTTP endpoints, as one off-the-cuff idea) could do a lot for the kinds of decentralized applications that could be built entirely as PWA with no other application wrapper needed. libp2p seems to have this goal in mind on their issue tracker, but so far as skimming the issues, I don't think I've seen a timeline for releasing it.

smaddock 8y ago

I see that the DHT relies on a package called 'ipv4-peers' [1]. Does it also support IPv6 peers? I know there's a BitTorrent BEP for IPv6 support, but I'm not quite sure if that relates to this DHT implementation. [1] https://github.com/hyperswarm/dht/blob/master/index.js#L4 https://github.com/hyperswarm/dht/blob/master/index.js#L4

mafintosh 8y ago
ipv6 is on the roadmap :) i just started spec'ing that out today actually.

bennofs 8y ago

How does the hole punching work? Is there a high level overview of the difference between this implementation and textbook Kademlia? I am not sure in which part of the code to look for this (is the hole punching part of the dht package?) so any pointers are appreciated

mafintosh 8y ago
Yea holepunching is implemented in the dht-rpc package. There is an explanatory comment there as well, https://github.com/mafintosh/dht-rpc/blob/master/lib/io.js#L47-L68 https://github.com/mafintosh/dht-rpc/blob/master/lib/io.js#L... (hope that helps!) We are planning to blog more about how the internals work. The "first class" hole punching is def one of the features I'm most exciting about.

fenwick67 8y ago

How does this package compare with https://www.npmjs.com/package/hyperdht https://www.npmjs.com/package/hyperdht ? Both seem to depend on https://www.npmjs.com/package/dht-rpc https://www.npmjs.com/package/dht-rpc for most of the work.

mafintosh 8y ago
Author of both here. I need to update the README of hyperdht, but we basically just iterated that a bit and moved it hyperswarm/dht to as it's part of the stack now. I'll update the hyperdht README to reflect that.
- TACIXAT 8y ago
  Curious for more details on how the dht signaling / hole punching works. I'm working on a peer to peer chat and file sharing system right now and upnp / nat pmp are pretty wack. I'd be interested in using it if there ever is a Go port.