4 ms·

Is there any library/framework that helps build p2p things? That could help the ecosystem the same way web frameworks have helped build websites faster and more

Is there any library/framework that helps build p2p things? That could help the ecosystem the same way web frameworks have helped build websites faster and more secure than ever.

asp2insp 11y ago
ipfs just released their beta which includes a golang library for interacting with their DHT/p2p network
- cdata 11y ago
  Here is a link to go-libp2p, which I believe is what you are referring to: https://github.com/ipfs/go-libp2p https://github.com/ipfs/go-libp2p
api 11y ago
We are working on a pluggable SDK version of ZeroTier with a tiny embedded IP stack that will allow p2p using standard unmodified network code, libraries, and protocols. Full interoperability with servers and desktops and other members of virtual networks is also nice. Basically the app is just a "device." An embeddable IP stack sounds fat but it's actually tiny. Whole ball of wax is similar in footprint to other p2p libraries that don't allow normal protocols and full code reuse. I'm a bit partial of course, but the approach is amazingly cool. It will be for Linux, Mac, iOS, Android, and eventually Windows. https://www.zerotier.com/product-netcon.shtml https://www.zerotier.com/product-netcon.shtml Currently GPL with dual licensing for proprietary closed use, which may or may not change later depending on how other things go. Linux and iOS are working. Android and Mac next. This gives you totally easy p2p transport but not things like a distributed database or identity, etc. But there are lots of other projects for that that become easier to use when you have transparent p2p.
lewisl9029 11y ago
I'm not aware of any batteries-included frameworks, but here are some of the p2p-ish libraries that I've either worked with personally, or looked into and found promising, along with some commentary: Telehash: http://telehash.org/ http://telehash.org/ Provides a p2p-capable messaging stack. I've worked with an earlier version (v2) for Toc Messenger [1]. V2 included an integrated DHT for public peer discovery along with the p2p messaging stack. Was actually surprisingly robust and reliable considering how young the project was at the time. V3 seems to have pivoted towards the private mesh networking side, and spun out the DHT into a separate project, but V3 itself doesn't seem to have a clear story for integrating with an external, public DHT the last time I checked, so the intended use cases aren't quite the same as V2. [1] http://toc.im/ http://toc.im/ Matrix: http://matrix.org/ http://matrix.org/ Doesn't allow for fully peer-to-peer messaging, but is decentralized and federated like XMPP. Probably the closest thing on this list to a framework (handles both messaging and storage). Last time I looked at it, my main concern was how much implicit trust users needed to place in homeserver providers (messages and user profiles are stored in plaintext). Unless this has changed, I don't find it viable for any app that values user privacy. Even federated networks eventually gravitate towards a large degree of centralization once the tech becomes mainstream (see email). If any servers are used at all in a privacy-minded app, they need to be zero-knowledge as far as I'm concerned, for actual user data at the very least, if not metadata as well (I realize the latter is an unsolved problem). RemoteStorage: https://remotestorage.io/ https://remotestorage.io/ A decentralized, federated protocol and library for data & file storage. Also used this for Toc. Worked as advertised but I'm looking to explore fully p2p alternatives for my next project. The main problems I ran into (asides from the non-fully-p2p architecture) was the rather clunky API (for instance, it requires developers to provide a schema to fully specify objects they want to store, but doesn't provide a versioning and migration mechanism to take advantage of the schema specs, although it is planned), and the fact that encryption for data at rest wasn't implemented (until I finished working on a custom encryption layer for Toc built on top of the library). Kinto: http://kinto.readthedocs.org/en/latest/ http://kinto.readthedocs.org/en/latest/ Similar to RemoteStorage, but decentralized discovery is planned and not implemented yet. It's built by Mozilla and apparently used in Firefox and Firefox OS (I'd assume for Firefox Sync?). They also have a comparison table with RemoteStorage as a part of their docs if you're interested in a more detailed (but possibly impartial) comparison [2]. [2] http://kinto.readthedocs.org/en/latest/overview.html#comparison-with-other-solutions http://kinto.readthedocs.org/en/latest/overview.html#compari... Swarm: https://github.com/gritzko/swarm https://github.com/gritzko/swarm A decentralized data replication library that's built on ops-based CRDTs [3]. Some very impressive work here. Although I recently found out that V1 of the library no longer supports fully p2p replication due to architectural changes, which made it a lot less interesting for my use cases. [3] https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type#Operation-based_CRDTs https://en.wikipedia.org/wiki/Conflict-free_replicated_data_... Replikativ: https://github.com/replikativ/replikativ https://github.com/replikativ/replikativ The one I'm most excited about personally. Similar to Swarm, a CRDT-based data replication library, but even more ambitious (they envision an open data exchange whereby users fully control their own data, and can specify any number of applications to have access to that data, instead of the status quo where user data is kept in closed, per-application silos). There's a comparison with other alternatives, including Swarm, in their README [4]. They also plan to have pure browser-based p2p replication through WebRTC, which is exactly what I'm looking for. It's built in and for Clojure and ClojureScript, but JavaScript support is in the works. As if I needed any more incentive to get started on my first ClojureScript app. =) [4] https://github.com/replikativ/replikativ#alternatives https://github.com/replikativ/replikativ#alternatives GunJS: http://gun.js.org/#step1 http://gun.js.org/#step1 A decentralized, replicated graph database. The pitch is very impressive sounding, but I'm hesitant to actually use this because they promise a lot of replication magic, but the replication is built on a home-grown algorithm rather than something rigorously peer-reviewed and implemented industry-wide like CRDTs. I'd love to hear from anyone who has actual working experience with it though. IPFS: https://ipfs.io/ https://ipfs.io/ IPFS and its suite of associated libraries has already been mentioned elsewhere in this thread, so I won't elaborate on it.
- haddr 11y ago
  I've had one project where we evaluated P2P libraries for this kind of message exchange, and I remember that telehash was the most promising. The project however was finally abandonded :( I think today I would give IPFS a try.
- marknadal 11y ago
  Author of gun.js here, great comments/concerns. First, Aphyr (Kyle Kingsbury of Call Me Maybe fame, Jepsen database testing suite) did some tweets about GUN ( https://twitter.com/aphyr/status/646302398575587332?ref_src=twsrc%5Etfw https://twitter.com/aphyr/status/646302398575587332?ref_src=... ), and I met with him at a conference in Berlin where we both were talking. He had some concerns (wanted the conflict resolution to be able to be plug-in-play with your-own-CRDT, and if journaling is disabled then some stale/old updates would be dropped), but overall seemed positive about GUN (check his own tweets). I hope to have him do a more serious review at some point, but I also hope that his initial thoughts provides some initial industry peer review. Tailing the first point, part of the reason why we didn't receive a more critical response from him is because we make NO claims of serializability or linearizability, which is what most academic research focuses on. In fact, GUN takes a somewhat controversial view with regards to classical things like Strong Consistency. For more information on how GUN's algorithm works, check out this recent podcast we did with the LondonJS community - https://youtu.be/qJNDplwJ8aQ?t=32m45s https://youtu.be/qJNDplwJ8aQ?t=32m45s . Second, GUN does use CRDTs. All CRDT means is "Convergent Replicated Data Type", which is a fancy way of saying that you'll get the same end result regardless of the ordering of the operations involved. Because GUN is offline-first it is incredibly important that it can retry an update multiple times to handle network partitions - this fundamentally results in operations not having a consistent ordering, especially when dealing with multiple peers updating (offline) simultaneously. If you watch the "Holy Grail Demo" (1min long, https://youtu.be/-i-11T5ZI9o https://youtu.be/-i-11T5ZI9o ) you see that this does not wind up being a problem. Third, there are other CRDTs that can be implemented on top of GUN's core. A specific instance of where GUN's default will fail you is in commutative operations (something Kyle was also concerned about), by default you cannot do increment (adding numbers) operations with GUN - partly because GUN treats a single value as being atomic. If you have a HN upvote count and you naively were to use GUN's out of the box CRDT, and two people locally added 1 to their local value at the same time... they would both converge to the same end result (5 + 1 = 6) but the intended result is 7. This intention-preserving CRDT is called a Commutative Replicated Data Type, which a specific variant on the general category. However, this does not mean that it is impossible to do this with GUN - you can, because GUN's underlying CRDT is designed to be emergent, allowing for other CRDTs to be built on top. At some point we'll (or somebody in the community) will provide these as plug-in-play module/extensions to GUN, so that way you won't have to build them yourself. Fourth, "home-grown" is somewhat true, but I think that is giving me too much credit. GUN's underlying conflict resolution algorithm is based on a hybrid lexical, vector, and timestamp clock - hardly magic at all, in fact they are somewhat naive intentionally. The nice thing is that they "just work" for the majority of web based apps right out of the box, which is probably why it seems magical. However, they won't work (alone) for more complicated business logic, and that is why we encourage the UNIX/NodeJS philosophy of adding those algorithms on top. It is important that your bread-and-butter machine is not based on a monolithic black box (like a lot of databases are), instead you should understand the base of the system and then plug your business-specific needs on top. Fifth, we're looking for more academic and peer review! I've already had numerous discussions with a Distributed Systems PhD at Carnegie Mellon that got poached by Uber's self driving car division, he's trying to help me find some people to put together a paper. That said, if you know anybody or could help connect me into that world it would be greatly appreciated. Thanks so much for including us in the list, please shout if you have any more questions! :)
jnpn 11y ago
https://www.ethereum.org/ https://www.ethereum.org/
rzzzt 11y ago
TomP2P is a Java library for building DHTs: http://tomp2p.net/ http://tomp2p.net/
EGreg 11y ago
http://qbix.com/platform http://qbix.com/platform
whimful 11y ago
you might like to check out secure-scuttlebutt : http://ssbc.github.io/scuttlebot/ http://ssbc.github.io/scuttlebot/ it's a gossip based network / database. Check out our first active app : http://github.com/ssbc/patchwork http://github.com/ssbc/patchwork
martindale 11y ago
I've been working on exactly this mission over the past several years in a framework I'm calling Maki: https://maki.io https://maki.io We're approaching our 0.3 release, which will provide the minimum viable framework for developing fully peer-to-peer applications. There is much work to be done with regards to documentation and design, so we'd love to welcome anyone who wants to help to our community.

Is there any library/framework that helps build p2p things? That could help the ecosystem the same way web frameworks have helped build websites faster and more secure than ever.

asp2insp 11y ago

ipfs just released their beta which includes a golang library for interacting with their DHT/p2p network

cdata 11y ago
Here is a link to go-libp2p, which I believe is what you are referring to: https://github.com/ipfs/go-libp2p https://github.com/ipfs/go-libp2p

api 11y ago

We are working on a pluggable SDK version of ZeroTier with a tiny embedded IP stack that will allow p2p using standard unmodified network code, libraries, and protocols. Full interoperability with servers and desktops and other members of virtual networks is also nice. Basically the app is just a "device." An embeddable IP stack sounds fat but it's actually tiny. Whole ball of wax is similar in footprint to other p2p libraries that don't allow normal protocols and full code reuse. I'm a bit partial of course, but the approach is amazingly cool. It will be for Linux, Mac, iOS, Android, and eventually Windows. https://www.zerotier.com/product-netcon.shtml https://www.zerotier.com/product-netcon.shtml Currently GPL with dual licensing for proprietary closed use, which may or may not change later depending on how other things go. Linux and iOS are working. Android and Mac next. This gives you totally easy p2p transport but not things like a distributed database or identity, etc. But there are lots of other projects for that that become easier to use when you have transparent p2p.

lewisl9029 11y ago

I'm not aware of any batteries-included frameworks, but here are some of the p2p-ish libraries that I've either worked with personally, or looked into and found promising, along with some commentary: Telehash: http://telehash.org/ http://telehash.org/ Provides a p2p-capable messaging stack. I've worked with an earlier version (v2) for Toc Messenger [1]. V2 included an integrated DHT for public peer discovery along with the p2p messaging stack. Was actually surprisingly robust and reliable considering how young the project was at the time. V3 seems to have pivoted towards the private mesh networking side, and spun out the DHT into a separate project, but V3 itself doesn't seem to have a clear story for integrating with an external, public DHT the last time I checked, so the intended use cases aren't quite the same as V2. [1] http://toc.im/ http://toc.im/ Matrix: http://matrix.org/ http://matrix.org/ Doesn't allow for fully peer-to-peer messaging, but is decentralized and federated like XMPP. Probably the closest thing on this list to a framework (handles both messaging and storage). Last time I looked at it, my main concern was how much implicit trust users needed to place in homeserver providers (messages and user profiles are stored in plaintext). Unless this has changed, I don't find it viable for any app that values user privacy. Even federated networks eventually gravitate towards a large degree of centralization once the tech becomes mainstream (see email). If any servers are used at all in a privacy-minded app, they need to be zero-knowledge as far as I'm concerned, for actual user data at the very least, if not metadata as well (I realize the latter is an unsolved problem). RemoteStorage: https://remotestorage.io/ https://remotestorage.io/ A decentralized, federated protocol and library for data & file storage. Also used this for Toc. Worked as advertised but I'm looking to explore fully p2p alternatives for my next project. The main problems I ran into (asides from the non-fully-p2p architecture) was the rather clunky API (for instance, it requires developers to provide a schema to fully specify objects they want to store, but doesn't provide a versioning and migration mechanism to take advantage of the schema specs, although it is planned), and the fact that encryption for data at rest wasn't implemented (until I finished working on a custom encryption layer for Toc built on top of the library). Kinto: http://kinto.readthedocs.org/en/latest/ http://kinto.readthedocs.org/en/latest/ Similar to RemoteStorage, but decentralized discovery is planned and not implemented yet. It's built by Mozilla and apparently used in Firefox and Firefox OS (I'd assume for Firefox Sync?). They also have a comparison table with RemoteStorage as a part of their docs if you're interested in a more detailed (but possibly impartial) comparison [2]. [2] http://kinto.readthedocs.org/en/latest/overview.html#comparison-with-other-solutions http://kinto.readthedocs.org/en/latest/overview.html#compari... Swarm: https://github.com/gritzko/swarm https://github.com/gritzko/swarm A decentralized data replication library that's built on ops-based CRDTs [3]. Some very impressive work here. Although I recently found out that V1 of the library no longer supports fully p2p replication due to architectural changes, which made it a lot less interesting for my use cases. [3] https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type#Operation-based_CRDTs https://en.wikipedia.org/wiki/Conflict-free_replicated_data_... Replikativ: https://github.com/replikativ/replikativ https://github.com/replikativ/replikativ The one I'm most excited about personally. Similar to Swarm, a CRDT-based data replication library, but even more ambitious (they envision an open data exchange whereby users fully control their own data, and can specify any number of applications to have access to that data, instead of the status quo where user data is kept in closed, per-application silos). There's a comparison with other alternatives, including Swarm, in their README [4]. They also plan to have pure browser-based p2p replication through WebRTC, which is exactly what I'm looking for. It's built in and for Clojure and ClojureScript, but JavaScript support is in the works. As if I needed any more incentive to get started on my first ClojureScript app. =) [4] https://github.com/replikativ/replikativ#alternatives https://github.com/replikativ/replikativ#alternatives GunJS: http://gun.js.org/#step1 http://gun.js.org/#step1 A decentralized, replicated graph database. The pitch is very impressive sounding, but I'm hesitant to actually use this because they promise a lot of replication magic, but the replication is built on a home-grown algorithm rather than something rigorously peer-reviewed and implemented industry-wide like CRDTs. I'd love to hear from anyone who has actual working experience with it though. IPFS: https://ipfs.io/ https://ipfs.io/ IPFS and its suite of associated libraries has already been mentioned elsewhere in this thread, so I won't elaborate on it.

haddr 11y ago
I've had one project where we evaluated P2P libraries for this kind of message exchange, and I remember that telehash was the most promising. The project however was finally abandonded :( I think today I would give IPFS a try.
marknadal 11y ago
Author of gun.js here, great comments/concerns. First, Aphyr (Kyle Kingsbury of Call Me Maybe fame, Jepsen database testing suite) did some tweets about GUN ( https://twitter.com/aphyr/status/646302398575587332?ref_src=twsrc%5Etfw https://twitter.com/aphyr/status/646302398575587332?ref_src=... ), and I met with him at a conference in Berlin where we both were talking. He had some concerns (wanted the conflict resolution to be able to be plug-in-play with your-own-CRDT, and if journaling is disabled then some stale/old updates would be dropped), but overall seemed positive about GUN (check his own tweets). I hope to have him do a more serious review at some point, but I also hope that his initial thoughts provides some initial industry peer review. Tailing the first point, part of the reason why we didn't receive a more critical response from him is because we make NO claims of serializability or linearizability, which is what most academic research focuses on. In fact, GUN takes a somewhat controversial view with regards to classical things like Strong Consistency. For more information on how GUN's algorithm works, check out this recent podcast we did with the LondonJS community - https://youtu.be/qJNDplwJ8aQ?t=32m45s https://youtu.be/qJNDplwJ8aQ?t=32m45s . Second, GUN does use CRDTs. All CRDT means is "Convergent Replicated Data Type", which is a fancy way of saying that you'll get the same end result regardless of the ordering of the operations involved. Because GUN is offline-first it is incredibly important that it can retry an update multiple times to handle network partitions - this fundamentally results in operations not having a consistent ordering, especially when dealing with multiple peers updating (offline) simultaneously. If you watch the "Holy Grail Demo" (1min long, https://youtu.be/-i-11T5ZI9o https://youtu.be/-i-11T5ZI9o ) you see that this does not wind up being a problem. Third, there are other CRDTs that can be implemented on top of GUN's core. A specific instance of where GUN's default will fail you is in commutative operations (something Kyle was also concerned about), by default you cannot do increment (adding numbers) operations with GUN - partly because GUN treats a single value as being atomic. If you have a HN upvote count and you naively were to use GUN's out of the box CRDT, and two people locally added 1 to their local value at the same time... they would both converge to the same end result (5 + 1 = 6) but the intended result is 7. This intention-preserving CRDT is called a Commutative Replicated Data Type, which a specific variant on the general category. However, this does not mean that it is impossible to do this with GUN - you can, because GUN's underlying CRDT is designed to be emergent, allowing for other CRDTs to be built on top. At some point we'll (or somebody in the community) will provide these as plug-in-play module/extensions to GUN, so that way you won't have to build them yourself. Fourth, "home-grown" is somewhat true, but I think that is giving me too much credit. GUN's underlying conflict resolution algorithm is based on a hybrid lexical, vector, and timestamp clock - hardly magic at all, in fact they are somewhat naive intentionally. The nice thing is that they "just work" for the majority of web based apps right out of the box, which is probably why it seems magical. However, they won't work (alone) for more complicated business logic, and that is why we encourage the UNIX/NodeJS philosophy of adding those algorithms on top. It is important that your bread-and-butter machine is not based on a monolithic black box (like a lot of databases are), instead you should understand the base of the system and then plug your business-specific needs on top. Fifth, we're looking for more academic and peer review! I've already had numerous discussions with a Distributed Systems PhD at Carnegie Mellon that got poached by Uber's self driving car division, he's trying to help me find some people to put together a paper. That said, if you know anybody or could help connect me into that world it would be greatly appreciated. Thanks so much for including us in the list, please shout if you have any more questions! :)

jnpn 11y ago

https://www.ethereum.org/ https://www.ethereum.org/

rzzzt 11y ago

TomP2P is a Java library for building DHTs: http://tomp2p.net/ http://tomp2p.net/

EGreg 11y ago

http://qbix.com/platform http://qbix.com/platform

whimful 11y ago

you might like to check out secure-scuttlebutt : http://ssbc.github.io/scuttlebot/ http://ssbc.github.io/scuttlebot/ it's a gossip based network / database. Check out our first active app : http://github.com/ssbc/patchwork http://github.com/ssbc/patchwork

martindale 11y ago

I've been working on exactly this mission over the past several years in a framework I'm calling Maki: https://maki.io https://maki.io We're approaching our 0.3 release, which will provide the minimum viable framework for developing fully peer-to-peer applications. There is much work to be done with regards to documentation and design, so we'd love to welcome anyone who wants to help to our community.