12 ms·
Ask HN: Recommendations for Peer to Peer Frameworks?
I'm looking for peer to peer framework recommendations to build a distributed app. In particular, a framework that supports seeds and peer discovery, message passing, maybe firewall / UPNP support, etc.
I'm not particular about language (Java, Go, Javascript, etc), since it's for a new project. I found this thread from 2016 (https://news.ycombinator.com/item?id=11145691), it had some interesting recommendations, but quite a few were out of date, and with the big push to distributed apps nowadays, I thought it might be a good time to readdress the topic.
Some of the solutions form the other thread or that I've found:
- Smokesignal (Javascript / Node, not updated in >10 months) https://github.com/marcelklehr/smokesignal
- ZeroTier: https://www.zerotier.com/product-netcon.shtml
- Maki: https://maki.io
- Telehash: http://telehash.org/
- A whole host of blockchain projects, Bitcoin, Ethereum, HyperLedger, Scorex (Scala blockchain)
Bitcoin is an interesting option, it certainly has a large user base and is well tested and secure code, but forking it has it's own set of issues.
- dnomad 9y agoI2P works well enough.
- Dowwie 9y agoHow about IPFS? OpenBazaar is an IPFS case study. They recently released a revamped v2 built with Go.
- zilchers 9y agoThat's interesting, I hadn't thought about IPFS - it doesn't really have a strong message passing semantic so much as it has good replication properties, but maybe you could build a distributed log on top of it.
- jjallen 9y agoBy good "replication properties" you mean "having to run many scripts/nodes yourself" to get widely replicated that is true. We found IPFS in general to be awesome software that just wasn't quite ready for production. There's IPFS-cluster[0] but that's still in capital-letters ALPHA. https://github.com/ipfs/ipfs-cluster https://github.com/ipfs/ipfs-cluster
- jjallen 9y agoThey are working on this: https://github.com/orbitdb/ipfs-log https://github.com/orbitdb/ipfs-log
- xendipity 9y agoYeah, I'd look at IPFS's networking library, libp2p. It's the peer-to-peer backbone of IPFS and has a big community and active development https://libp2p.io/ https://libp2p.io/. It's also implemented in both go and js (as most of IPFS is)
- eeZah7Ux 9y agoThe wire protocol is practically almost not documented.
- jjallen 9y agoWe moved away from libp2p at the project I work on because of major instability problems in browsers. Our dapp that used libp2p would crash after a few minutes and was generally not easy to use. EDIT: I'm referring to the JS version here.
- lgierth 9y agoSorry to hear you had move to something else -- my guess is these crashes were from WebRTC, which isn't very performant in browsers yet. Especially with more than a handful of WebRTC connections, we've seen it crash regularly. We've switched the default browser-to-browser transport in js-libp2p to a websockets relay recently, so you should give it another try.
- ComputerGuru 9y agoDid you try to investigate the stack trace or file a bug upstream? No snark intended!
- jjallen 9y agoYes, of course, and it was WebRTC. We ended up with a better solution (encoding the metadata we were storing in IPFS in a bytes32 stored in the Ethereum blockchain). There were extra scripts and setup. Overall moving off simplified our stack, so it was probably a good move despite libp2p's crashing. Could be totally fixed by now for all I know. That said, it was believed that most Protocol Labs resources were/would be devoted to Filecoin, which is probably a safe bet.
- jjallen 9y agoI'd love to read technical details about what they built with it and how they manage replication. Will look for some and repost if I find.
- lgierth 9y agoLook at the IPFS Pubsub and Y.js integration, that e.g. PeerPad uses to build distributed text editing: https://github.com/ipfs-shipyard/peer-pad/blob/master/docs/ARCHITECTURE.md https://github.com/ipfs-shipyard/peer-pad/blob/master/docs/A...
- adamnemecek 9y agoLook into crdt it’s the shit for collaborative editing (idk if that’s what you ares looking for tho).
- malloryerik 9y agoDo you happen to know of a decent and easy-ish crdt library in Go?
- adamnemecek 9y agoCheck github there’s plenty.
- zilchers 9y agoCRDT is definitely a good pattern for the data passing itself, but one of the really tough parts of these distributed apps is the plumbing work, just getting peers to reliability communicate. That's part of what I'm looking for.
- billylo 9y agoI am using https://hypelabs.io/ https://hypelabs.io/ Pretty good for small payloads.
- rauljordan 9y agoLook into libp2p created by the Protocol Labs team (created IPFS). We're thinking of moving to this framework internally at Ethereum.
- TACIXAT 9y agoSecond vote for libp2p. It looks very mature and generic enough to build whatever off of it. It has implementations in js and go. If you end up rolling your own, I found js to have the most building blocks (pmp and upnp libraries).
- Dowwie 9y agoThere's an implementation in Rust: https://github.com/libp2p/rust-libp2p https://github.com/libp2p/rust-libp2p
- fro0116 9y agoCan anyone point to some higher-level libraries or open-source apps built on top of libp2p?
- lgierth 9y agoApart from the obvious js-ipfs and go-ipfs, check out ipfs-cluster and LivePeer. libp2p is actually meant to be pretty high-level so I'm not aware of more high-level libraries.
- lgierth 9y agoActually, a few minutes later, diggan set my understanding of your question straight :) Check out peer-crdt, the ipfs/y.js bridge, ipfs-pubsub-room, and orbitdb!
- no_identd 9y ago>We're thinking of moving to this framework internally at Ethereum. Why not GNUnet? Check the recently released 181 pages long dissertation on it: https://grothoff.org/christian/habil.pdf https://grothoff.org/christian/habil.pdf
- 0XAFFE 9y agoMaybe Scuttlebutt? https://www.scuttlebutt.nz/ https://www.scuttlebutt.nz/ Scuttlebutt is advertised as a social network, but in core it is an eternal distributed log. You can pass either public or private messages and assets around.
- zilchers 9y agoThis is a great suggestion, have started looking into it. Thanks!
- freshhawk 9y agoScuttlebutt and Dat (https://datproject.org/ https://datproject.org/) are definitely looking like good bets right now.
- yoshuaw 9y agoAlso perhaps fun to note is that we're currently hacking on a Rust version of Dat. Hopefully we'll be able to bring the Dat protocol to more languages and platforms soon! https://github.com/datrs https://github.com/datrs
- robotmay 9y agoOh man that makes me happy. I just posted about that this morning. Hopefully I can find a way to contribute!
- deleted 9y ago[deleted]
- DyslexicAtheist 9y agoKademlia DHT: https://en.wikipedia.org/wiki/Kademlia https://en.wikipedia.org/wiki/Kademlia https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia...
- petegordon 9y agoI’m curious what your goal is. Do you want the complete immutability of transactions like bitcoin/blockchain; or do you want the peer/seed capability like BitTorrent and IPFS with a DHT (Kademlia)? I’d be interested in seeing a good list of the technology choices that can make up a P2P project if anyone knows of one, please share. But you definitely should checkout IPFS and Kademlia.
- mcjiggerlog 9y agoMaybe check out Dat[0] - especially useful if you need some kind of distributed data store. The main implementation is in Javascript. [0] https://datproject.org/ https://datproject.org/
- robotmay 9y agoI really wish there was an additional implementation in something like Rust (personal preference, but C would be good as it's fairly easy to bind to) for DAT, because what they're doing is rather cool and the experience is tidy.
- jondashkyle 9y agoThat’s in the works: https://github.com/datrs https://github.com/datrs
- robotmay 9y agoAwesome. I'm really excited for that, I'll have to try to contribute.
- indescions_2018 9y agoAwesome List of P2P resources: https://github.com/kgryte/awesome-peer-to-peer https://github.com/kgryte/awesome-peer-to-peer Personal bias tilts to web tech. WebRTC will require you to centrally implement all your peer broker and logic. But take a look at something like Peer5 Video CDN to appreciate scalability: https://www.peer5.com/ https://www.peer5.com/
- inglor 9y agoHey, Benji from Peer5 here - our product is really cool and helps you scale your video delivery but I don't believe it works in this case. We solve the problem of distributing HLS/DASH delivery over large p2p swarms and scaling video. Doesn't sound like it could work for OP. Thanks a lot for keeping us in mind though :)
- adam_ellsworth 9y agoHey Benji. I saw you guys setting up at NAB yesterday. Would there be a good day this week to come by and talk shop?
- inglor 9y agoHey Adam, I'm not attending NAB this year - Shachar and Hadar would love to talk (shachar@peer5.com or hadar@peer5.com) in terms of technical people attending :)
- cjbprime 9y agoWebTorrent.
- wslh 9y agoRe blockchain technologies: while there is an obvious intersection between the world of blockchain and P2P, blockchain technologies are slow because BFT adds many requirements to the game. Basically blockchain technologies use P2P to reach a common state and this could not be necessary in your use case. You can take a look at OpenBazaar and Particl for P2P implementations.
- no_identd 9y agoThe 181 pages long GNUnet dissertation very recently got released: https://grothoff.org/christian/habil.pdf https://grothoff.org/christian/habil.pdf I'd strongly recommend going for GNUnet, nobody takes me serious when I say this, but it WILL surplant IP eventually.
- woah 9y agoLink broken.
- no_identd 9y agohttps://pdfs.semanticscholar.org/153d/578869863450766a87072702243a38f3abe2.pdf https://pdfs.semanticscholar.org/153d/578869863450766a870727... Previous Hacker News discussion here: https://news.ycombinator.com/item?id=15877908 https://news.ycombinator.com/item?id=15877908
- fouc 9y agohttp://web.archive.org/web/20180327164208if_/https://grothoff.org/christian/habil.pdf http://web.archive.org/web/20180327164208if_/https://grothof...
- Ixiaus 9y agoI agree. I think gnunet gets a lot right.
- yid 9y ago> but it WILL surplant IP eventually. The painful move from IPv4 to IPv6 suggests that this is unlikely. More likely is an overlay over IP, TCP, or even HTTPS.
- no_identd 9y ago>The painful move from IPv4 to IPv6 suggests that this is unlikely. More likely is an overlay over IP, TCP, or even HTTPS. GNUnet can function as an overlay and an _underlay_ simultaneously, quoting from the dissertation: > GNUnet is an overlay network, as it is initially supposed to primarily operate over the existing Internet network. However, GNUnet does not assume that this is always the case. For example, GNUnet can also operate directly over WLAN or Bluetooth to create an ad-hoc wireless mesh network, and with the GNUnet “PT/VPN” subsystems we can run TCP/IP over GNUnet. So in this configuration, GNUnet would be an underlay network, a bit like MPLS or B.A.T.M.A.N. [NAL07]. In reality, we in fact expect to see both at the same time: some peers will run over the existing Internet, while others may connect to GNUnet on Layer 2.
- hapnin 9y agoBlockstack looks like it would fit here: https://github.com/blockstack/blockstack/blob/master/README.md https://github.com/blockstack/blockstack/blob/master/README....
- abreckle 9y ago+1 for Blockstack
- martindale 9y agoMaki 0.3 will include components from Fabric: https://fabric.fm https://fabric.fm — automatically available to anyone upgrading from 0.2. :)
- tribler 9y ago> supports seeds and peer discovery, message passing, maybe firewall / UPNP support, etc. Our audacious 'IPv8' work is now deployed live. Might fit you needs. Includes above features in a low-complexity approach. https://github.com/Tribler/py-ipv8 https://github.com/Tribler/py-ipv8 Our fresh IETF Internet Standard draft of this work: https://tools.ietf.org/html/draft-pouwelse-trustchain-00 https://tools.ietf.org/html/draft-pouwelse-trustchain-00
- smaddock 9y agoIf you plan on having any collaborative editing features, Yjs [1] is a p2p framework utilizing CRDTs with support for IPFS. [1] https://github.com/y-js/yjs https://github.com/y-js/yjs
- mempko 9y agoHow about something [completely different](http://firestr.com http://firestr.com)
- alexmorley 9y agoMaybe checkout the dat project https://datproject.org/ https://datproject.org/ which is whats used by BeakerBroswer etc.
- moonbug22 9y agoIdiot
- JamesLeonis 9y agoThe WebTorrent Project [1] has a whole suite of tools for decentralized applications based around BitTorrent. It's not a framework per-se, but it breaks up the protocol into libraries for your use. If you're interested in Ethereum, it has other non-blockchain technologies that you can use for your apps. It has the Swarm network for storing bits of data, the Whisper network for chat and messaging, as well as Public-Key signing to verify messages. I'm using the latter functionality for a demo to replace password as the security mechanism. [1]: https://github.com/webtorrent https://github.com/webtorrent
- monocasa 9y agoThe virtual synchrony protocols don't get enough love. Conceptually, they're sort of like if you performed PAXOS to determine ordering for a distributed finite state machine. Let's you build all sorts of distributed applications where you trust the nodes. https://en.wikipedia.org/wiki/Virtual_synchrony https://en.wikipedia.org/wiki/Virtual_synchrony http://www.jgroups.org/ http://www.jgroups.org/ https://archive.codeplex.com/?p=vsync https://archive.codeplex.com/?p=vsync
- shpx 9y agohttps://urbit.org/docs/ https://urbit.org/docs/ The idea with urbit (amongst others, it's a big system) is for everyone to have a personal server that's easier to administrate than a linux server. It's been posted a bunch of times https://hn.algolia.com/?query=urbit&sort=byPopularity&prefix&page=0&dateRange=all&type=all https://hn.algolia.com/?query=urbit&sort=byPopularity&prefix... I've also heard about https://zeronet.io/ https://zeronet.io/ and https://beakerbrowser.com/ https://beakerbrowser.com/
- tomp2p 9y agoTomP2P (https://tomp2p.net/ https://tomp2p.net/) is a library handling firewall / UPNP support, however, the focus is academic/experimental.
- sova 9y agoZeroMQ http://zguide.zeromq.org/py:all http://zguide.zeromq.org/py:all ... "In One Hundred Words: ZeroMQ (also known as ØMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fan-out, pub-sub, task distribution, and request-reply. It's fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems. ZeroMQ is from iMatix and is LGPLv3 open source."
- mayamatrix 9y agoWorth having a look into the Holochain framework: https://holochain.org/ https://holochain.org/ Depending on the application(s) you have in mind.
- PkBuzios 9y agoHolochain is interesting for having a different approach then the blockchain-based frameworks, which are inefficient
- jeswin 9y agoSecure Scuttlebutt. It's probably the only one seeing everyday use to get things done. Patchwork, the flagship app is as good as Facebook/Twitter. Scuttlers often use git over SSB, even. What SSB gets right and the others don't, is the incentive to mirror or replicate. Virtual currency is a weak incentive in an early stage network - but friendship isn't; you mirror your friends' data. The basic API to program the dapp can be seen here - http://scuttlebot.io/ http://scuttlebot.io/ But for more advanced uses see https://youtu.be/f_baWUW4R8Y https://youtu.be/f_baWUW4R8Y
- w3news 9y agoCheck this great talk: https://youtu.be/izQFMADw70w https://youtu.be/izQFMADw70w
- devxpy 9y agoI recently discovered zyre. Its from the same guy that brought us the wonderful zeromq library While it does a lot of the connection and communication part, it lacks a way to bypass nat routers. Here is a book on zyre http://zguide.zeromq.org/php:chapter8 http://zguide.zeromq.org/php:chapter8 I also found this thing called pwnat which bypasses nat routers without the need for UDP hole punching ! https://github.com/samyk/pwnat https://github.com/samyk/pwnat I feel like if someone made a high level wrapper combining these two, we can have a very good framework.
- povilasb 9y agoAnother project you might find useful, that we at https://www.maidsafe.net https://www.maidsafe.net are actively working on, is https://github.com/maidsafe/crust/ https://github.com/maidsafe/crust/. It's a generic peer-to-peer networking library written in Rust. * it connects two peers together * various NAT traversal techniques are implemented: hole punching, UPnP/IGD, etc. * it supports both TCP and UDP. In case of UDP it uses uTP: http://www.bittorrent.org/beps/bep_0029.html * all messages are encrypted * it implements automatic peer discovery on LAN * it caches it's previous connections and is able to reuse them in the future * etc.
- doomjunky 9y agoPeer Name Resolution Protocol (PNRP) - Developed by Microsoft - Open Specification - DHT-based - IPv6 only - Windows support since Windows XP SP 3 - .NET support since .NET Framework 3.5 (System.Net.PeerToPeer) [1]: https://en.wikipedia.org/wiki/Peer_Name_Resolution_Protocol https://en.wikipedia.org/wiki/Peer_Name_Resolution_Protocol [2]: https://msdn.microsoft.com/en-us/library/aa371699.aspx https://msdn.microsoft.com/en-us/library/aa371699.aspx "PNRP Namespace Provider API" [3]: https://msdn.microsoft.com/en-us/library/bb726971.aspx https://msdn.microsoft.com/en-us/library/bb726971.aspx "Peer Name Resolution Protocol"
- zilchers 9y agoMicrosoft had a few cool things, but it all seemed .net specific. Check out this in WCF https://msdn.microsoft.com/en-us/library/cc297274.aspx https://msdn.microsoft.com/en-us/library/cc297274.aspx