8 ms·

Hypercore protocol: a distributed (P2P) append-only log

203 points by thinkmassive 6y ago

js4ever 6y ago
Hypercore / Hyperswarm seems super interesting... But last time I tried it few months ago it was ... Hyperslow
- jbverschoor 6y ago
  Or hypofast
donor20 6y ago
Good lord. Hyper everything. These are often (maybe not this time) navel gazing tech solutions in search of an actual product (other than child porn / dark web type stuff). I asked myself, I wonder if the word blockchain is on their home page. YES! I'd love to benchmark whatever they are doing against ZeroMQ - I think they have hyperswarm for the pub / sub solution. Anyone know how it runs (latency / throughput)?
- pfraze 6y ago
  I wrote a little bit about the product side of hypercore here- https://www.patreon.com/posts/paul-reveals-is-44665610 https://www.patreon.com/posts/paul-reveals-is-44665610
  - jcynix 6y ago
    Hmm, Somehow our networking tech has this big gap in-between LANs and Clouds ... I think there is more between LANs and Clouds, e.g. encrypted emails, or VPN tunnels which could be set up between sender and recipient, last but not least HTTPS or SFTP access to documents (which could still be encrypted Zip files. You'd have to share keys/secrets, but you'll need that for secure P2P connections anyways.
    - pfraze 6y ago
      Sure, but I think it can be easier to do. VPNs require some setup so it's not as ad-hoc as you want for sharing files between two offices. The HTTPS server is what many offices do now when they deal with a lot of secure document exchange to clients (law firms, accountants, etc). Again, not bad but requires them to setup an on-prem public server to have equivalent security to peer-to-peer. The opportunity is to take some steps out of the process.
- viraptor 6y ago
  > I wonder if the word blockchain is on their home page. YES! It's only used for comparison to something you may be already familiar with. Why is that a problem?
  - bogwog 6y ago
    Because 90% of things that mention "blockchain" are scams or scam-adjacent. Not saying that this is a scam, but it's good to have some skepticism.
    - dboreham 6y ago
      Not warranted in my experience. I've never heard anyone in the community (except perhaps...er..me) talk about integration between Dat and a blockchain.
    - pwdisswordfish4 6y ago
      The Dat people are notoriously opposed to blockchain hype. That doesn't mean you should actually go build anything on top of their projects (for reasons to do with what the top post in this thread touch on—they don't know much about stability or focus).
  - donor20 6y ago
    Hyper / mega / etc all are hype up type words Blockchain - tons of hype A ton of techno folks build products that do privacy / security but don't really have a use case or user base perhaps (outside of the unsavory types). Blockchain has this same issue. A lot of use cases around payments for kidnapping, drug deals etc and endless endless scams. Wasn't there a "megaupload" file sharing site where the guy went to jail for computer fraud (perhaps unfairly). Anyways, I would folks on what your product does, and just avoid the hyper blockchain mega drive space swarm language.
    - SturgeonsLaw 6y ago
      The example you gave, Megaupload, wasn't a peer to peer blockchain buzzword bingo site, it was just a plain old Web 2.0 era website with a founder who turned a blind eye (or perhaps encouraged) pirated media to be uploaded. Google Kim Dotcom if you're interested in his story. He hasn't done any jail time, despite the US DOJ's attempts to extradite him, although that really leads to the question as to whether jail is an appropriate punishment for copyright infringement.
conradev 6y ago
All of the decentralization/P2P companies I have seen focus on platform before product, seemingly with the attitude of “if we build it, developers will come”. Has this worked out for any of them so far? Regardless, the companies do great R&D, so I’m glad that people are investing in them. Are there any companies building product first that also happen to be P2P in nature?
- pfraze 6y ago
  We're pretty product-focused. We started with Beaker Browser, then recently have been working on CLI tools for private file sharing [1] and now we're working toward an app with the same goal [2] 1. https://www.patreon.com/posts/hyp-command-line-44923749 2. https://www.patreon.com/posts/paul-reveals-is-44665610
  - wngr 6y ago
    How is this patreon thing going for you? dat is a pretty neat protocol, but it seems to have trouble coming out of its niche. Are there any products (as in $$$) coming out of the community? Or maybe some bigger sponsors (think decentralized web)?
    - pfraze 6y ago
      We started it this month and are at $150/mo, not a bad start and we appreciate the help. We're trying it as a way to bridge us to the next step. I love Beaker but I think it was strategically too long-term. The hyp CLI and uplink are us turning our focus to better immediate utility for people, which we think is the right near-term strategy to breaking out of the niche.
  - CuriouslyC 6y ago
    Beaker browser does look very cool. It would be even cooler if you used WebRTC to build p2p browsing into a website or browser extension.
    - jeswin 6y ago
rklaehn 6y ago
Any progress on the rust implementation? There was a lot of activity a few months ago, but now it seems that things have stalled.
- karissa 6y ago
  Hi rklaehn, we're actively looking for contributors! Check out the #rust channel in the hypercore discord https://discord.gg/KAaPRsMm https://discord.gg/KAaPRsMm
sambroner 6y ago
Has anyone used this enough to share perf characteristics? This is coming up twice today, but I’m interested in using a P2P append only log as a total order broadcast for use with (my day job) Fluid Framework. Decentralizing the broadcast would make Fluid more aligned with the Ink and Switch essays about offline first. (Worth a read if you haven’t! I’d link, but I’m on mobile. It appears ink and switch does sponsor Hypercore)
tgirod 6y ago
I've recently installed beaker browser to have a look and poke around hypercore. Unfortunately, so far I couldn't load a single page from https://explore.beakerbrowser.com/ https://explore.beakerbrowser.com/ - even though those pages are supposed to be kept online thanks to hashbase.io Such a pity, I'd really like to see something like this take off!
- pfraze 6y ago
  That's all the old version of the protocol that we keep up for archival EDIT- took down the old explore site with a redirect to avoid future confusion
Fiahil 6y ago
Looks interesting! Quick question: it's a distributed log, how do you establish consensus to keep its entries ordered? If it's not ordered, then it's a set, not a log. Still cool, but not as impressive.
- z3ncyberpunk 6y ago
  log time-stamp?
- anchpop 6y ago
  Only one person using one device is allowed to add to the log. Each entry contains the hash of the entry before it (including that entry's field where it contains the previous hash, etc.). If you add entries that have a conflicting ordering, you're assumed to be a bad actor and get blocked by the network. If you want to have multiple devices or multiple people editing the same data, you need to give them all a separate log and use some kind of CRDT system to combine them
  - Fiahil 6y ago
    So, there's no consensus because writes are coming from a single process. I must say, I'm a little bit disappointed : that's not the world-changing algorithm I was hoping for.
    - rakoo 6y ago
      They're working on it, with a rather elegant solution: https://www.datprotocol.com/deps/0008-multiwriter/ https://www.datprotocol.com/deps/0008-multiwriter/
      - brunoqc 6y ago
        That document looks stale. Last changelog entry is 2018-11-17.
superzamp 6y ago
A related thing I'd love to see take off is companies offering log-based interactions instead of HTTP APIs only. Some apps are such a poor fit for HTTP that you end up with a convoluted mess of web-hooks as soon as some elements of async appear. Obviously, these web-hook contraptions are always home-grown and offer nothing near what you get with a proper log system.
syntonym2 6y ago
If I want to use one of the hyper* tools to partially syncronize files p2p on a LAN, which one should I use? The hyper* world seems to be very fragmented right now. There is the dat [0] project, which started 2013 and shares files between computers p2p. In may 2020 the dat protocol was renamed to the hypercore protocol [1] and dat "will continue as the collective of teams and projects that have evolved from the original Dat CLI project.". hypercore-protocol [2] links to multiple applications for file sharing (none of them the dat CLI tool). Hyperdrive [3] "help[s] you share files quickly and safely, directly from your computer [...] -- all P2P". The github [4] mentions the hyperdrive daemon als a batteries included experience, but the hyperdrive daemon has a deprecation notice and tells you to use hyperspace. Hyperspace [5] "provides remote access to Hypercores and a Hyperswarm instance" and "exposes a simple RPC interface". The documentation is very technical and seems to be aimed at developers, not using it as a tool. Digging around in the github organisation [6] or by stumbling upon the patreon [7] one can find the hyp CLI tool which is "A CLI for peer-to-peer file sharing (and more) using the Hypercore Protocol". The first commit is 9 days old. On the twitter of the author one can also find hyperbeam [8], which is integrated into hyp. Here on HN one can also find an announcement for "uplink" [9]. The tech looks pretty cool, but the vast amount of different projects makes it difficult to grasp. From all of these tools the dat tool seems to be the most advanced, but not actively maintained. It's not linked on any of the hyper* sites and doesn't seem to be the recommended way to use the hypercore protocol to share files p2p. While I would like to use the tech, I'm pretty lost on how to do that today. [0] https://docs.datproject.org/ [1] https://blog.datproject.org/2020/05/15/dat-protocol-renamed-hypercore-protocol/ [2] https://hypercore-protocol.org/ [3] https://hypercore-protocol.org/#hyperdrive [4] https://github.com/hypercore-protocol/hyperdrive [5] https://github.com/hypercore-protocol/hyperspace [6] https://github.com/hypercore-protocol/cli [7] https://www.patreon.com/posts/hyp-command-line-44923749 [8] https://github.com/mafintosh/hyperbeam [9] https://www.patreon.com/posts/paul-reveals-is-44665610
- dwiel 6y ago
  First off, as someone who has spent many hours trying to use dat, I agree 100% with your assessment. I have been using dat command line tool to p2p synchronize files on a LAN (and also over the internet) for a year or two now. The latest version is more stable for me than than the previous version and I really love it. It synchronizes files across multiple computers nearly instantly.
jokoon 6y ago
It's pretty nice to see that there are now several P2P techs. This one seems pretty good, but unless you have a single client and unless the protocol is defined to do a single thing and do it well, I can't see it thrive. For example bittorrent is a good tech because it only does one thing and users can understand it. Same for IPFS. Softwares like limewire or kazaa thrived because they were simple enough to use. Same goes for protocols. In my view, decentralization requires: * a client that runs on user's hardware * a p2p database and filesystem, with this kind of append-only log and verification, that runs on the client * server can still accelerate access access time Generally with this kind of system, the goal is that each user can publish data/content either in a public/private manner, without being dependent on servers. The only problem is being able to attract users and compete with popular platforms. If such a protocol+software can attract users who want a "privacy-aware" alternative to platforms like facebook, I could see it work. Sometimes I wonder why there isn't more money invested to build such platforms, because the p2p internet seems to be the only viable solution to fight privacy issues.
- sull 6y ago
  Yes and no. The tech is like the “Internet” and is largely born from academic work and continues forward as fundamental and foundational p2p networking advancements. It’s not a consumer product itself but obviously builders have and will fill that void. Beaker is the best example but eventually more and some projects you may have heard about are actually using some aspects of hypercore.
gioscarab 6y ago
It would be cool to run one of those P2P technologies over a protocol like PJON: https://github.com/gioblu/PJON https://github.com/gioblu/PJON until the network infrastructure is centralized P2P is relatively useless, it would be cool to see it applied on top of a private network infrastructure made by people.
anentropic 6y ago
> Somehow our networking tech has this big gap in-between LANs and Clouds I remember we used to have FTP we used to use it for all sorts of things... deploying websites, sending files to the print company etc

203 points by thinkmassive 6y ago

js4ever 6y ago

Hypercore / Hyperswarm seems super interesting... But last time I tried it few months ago it was ... Hyperslow

jbverschoor 6y ago
Or hypofast

donor20 6y ago

Good lord. Hyper everything. These are often (maybe not this time) navel gazing tech solutions in search of an actual product (other than child porn / dark web type stuff). I asked myself, I wonder if the word blockchain is on their home page. YES! I'd love to benchmark whatever they are doing against ZeroMQ - I think they have hyperswarm for the pub / sub solution. Anyone know how it runs (latency / throughput)?

pfraze 6y ago
I wrote a little bit about the product side of hypercore here- https://www.patreon.com/posts/paul-reveals-is-44665610 https://www.patreon.com/posts/paul-reveals-is-44665610
- jcynix 6y ago
  Hmm, Somehow our networking tech has this big gap in-between LANs and Clouds ... I think there is more between LANs and Clouds, e.g. encrypted emails, or VPN tunnels which could be set up between sender and recipient, last but not least HTTPS or SFTP access to documents (which could still be encrypted Zip files. You'd have to share keys/secrets, but you'll need that for secure P2P connections anyways.
  - pfraze 6y ago
    Sure, but I think it can be easier to do. VPNs require some setup so it's not as ad-hoc as you want for sharing files between two offices. The HTTPS server is what many offices do now when they deal with a lot of secure document exchange to clients (law firms, accountants, etc). Again, not bad but requires them to setup an on-prem public server to have equivalent security to peer-to-peer. The opportunity is to take some steps out of the process.
viraptor 6y ago
> I wonder if the word blockchain is on their home page. YES! It's only used for comparison to something you may be already familiar with. Why is that a problem?
- bogwog 6y ago
  Because 90% of things that mention "blockchain" are scams or scam-adjacent. Not saying that this is a scam, but it's good to have some skepticism.
  - dboreham 6y ago
    Not warranted in my experience. I've never heard anyone in the community (except perhaps...er..me) talk about integration between Dat and a blockchain.
  - pwdisswordfish4 6y ago
    The Dat people are notoriously opposed to blockchain hype. That doesn't mean you should actually go build anything on top of their projects (for reasons to do with what the top post in this thread touch on—they don't know much about stability or focus).
- donor20 6y ago
  Hyper / mega / etc all are hype up type words Blockchain - tons of hype A ton of techno folks build products that do privacy / security but don't really have a use case or user base perhaps (outside of the unsavory types). Blockchain has this same issue. A lot of use cases around payments for kidnapping, drug deals etc and endless endless scams. Wasn't there a "megaupload" file sharing site where the guy went to jail for computer fraud (perhaps unfairly). Anyways, I would folks on what your product does, and just avoid the hyper blockchain mega drive space swarm language.
  - SturgeonsLaw 6y ago
    The example you gave, Megaupload, wasn't a peer to peer blockchain buzzword bingo site, it was just a plain old Web 2.0 era website with a founder who turned a blind eye (or perhaps encouraged) pirated media to be uploaded. Google Kim Dotcom if you're interested in his story. He hasn't done any jail time, despite the US DOJ's attempts to extradite him, although that really leads to the question as to whether jail is an appropriate punishment for copyright infringement.

conradev 6y ago

All of the decentralization/P2P companies I have seen focus on platform before product, seemingly with the attitude of “if we build it, developers will come”. Has this worked out for any of them so far? Regardless, the companies do great R&D, so I’m glad that people are investing in them. Are there any companies building product first that also happen to be P2P in nature?

pfraze 6y ago
We're pretty product-focused. We started with Beaker Browser, then recently have been working on CLI tools for private file sharing [1] and now we're working toward an app with the same goal [2] 1. https://www.patreon.com/posts/hyp-command-line-44923749 2. https://www.patreon.com/posts/paul-reveals-is-44665610
- wngr 6y ago
  How is this patreon thing going for you? dat is a pretty neat protocol, but it seems to have trouble coming out of its niche. Are there any products (as in $$$) coming out of the community? Or maybe some bigger sponsors (think decentralized web)?
  - pfraze 6y ago
    We started it this month and are at $150/mo, not a bad start and we appreciate the help. We're trying it as a way to bridge us to the next step. I love Beaker but I think it was strategically too long-term. The hyp CLI and uplink are us turning our focus to better immediate utility for people, which we think is the right near-term strategy to breaking out of the niche.
- CuriouslyC 6y ago
  Beaker browser does look very cool. It would be even cooler if you used WebRTC to build p2p browsing into a website or browser extension.
  - jeswin 6y ago

rklaehn 6y ago

Any progress on the rust implementation? There was a lot of activity a few months ago, but now it seems that things have stalled.

karissa 6y ago
Hi rklaehn, we're actively looking for contributors! Check out the #rust channel in the hypercore discord https://discord.gg/KAaPRsMm https://discord.gg/KAaPRsMm

sambroner 6y ago

Has anyone used this enough to share perf characteristics? This is coming up twice today, but I’m interested in using a P2P append only log as a total order broadcast for use with (my day job) Fluid Framework. Decentralizing the broadcast would make Fluid more aligned with the Ink and Switch essays about offline first. (Worth a read if you haven’t! I’d link, but I’m on mobile. It appears ink and switch does sponsor Hypercore)

tgirod 6y ago

I've recently installed beaker browser to have a look and poke around hypercore. Unfortunately, so far I couldn't load a single page from https://explore.beakerbrowser.com/ https://explore.beakerbrowser.com/ - even though those pages are supposed to be kept online thanks to hashbase.io Such a pity, I'd really like to see something like this take off!

pfraze 6y ago
That's all the old version of the protocol that we keep up for archival EDIT- took down the old explore site with a redirect to avoid future confusion

Fiahil 6y ago

Looks interesting! Quick question: it's a distributed log, how do you establish consensus to keep its entries ordered? If it's not ordered, then it's a set, not a log. Still cool, but not as impressive.

z3ncyberpunk 6y ago
log time-stamp?
anchpop 6y ago
Only one person using one device is allowed to add to the log. Each entry contains the hash of the entry before it (including that entry's field where it contains the previous hash, etc.). If you add entries that have a conflicting ordering, you're assumed to be a bad actor and get blocked by the network. If you want to have multiple devices or multiple people editing the same data, you need to give them all a separate log and use some kind of CRDT system to combine them
- Fiahil 6y ago
  So, there's no consensus because writes are coming from a single process. I must say, I'm a little bit disappointed : that's not the world-changing algorithm I was hoping for.
  - rakoo 6y ago
    They're working on it, with a rather elegant solution: https://www.datprotocol.com/deps/0008-multiwriter/ https://www.datprotocol.com/deps/0008-multiwriter/
    - brunoqc 6y ago
      That document looks stale. Last changelog entry is 2018-11-17.

superzamp 6y ago

A related thing I'd love to see take off is companies offering log-based interactions instead of HTTP APIs only. Some apps are such a poor fit for HTTP that you end up with a convoluted mess of web-hooks as soon as some elements of async appear. Obviously, these web-hook contraptions are always home-grown and offer nothing near what you get with a proper log system.

syntonym2 6y ago

If I want to use one of the hyper* tools to partially syncronize files p2p on a LAN, which one should I use? The hyper* world seems to be very fragmented right now. There is the dat [0] project, which started 2013 and shares files between computers p2p. In may 2020 the dat protocol was renamed to the hypercore protocol [1] and dat "will continue as the collective of teams and projects that have evolved from the original Dat CLI project.". hypercore-protocol [2] links to multiple applications for file sharing (none of them the dat CLI tool). Hyperdrive [3] "help[s] you share files quickly and safely, directly from your computer [...] -- all P2P". The github [4] mentions the hyperdrive daemon als a batteries included experience, but the hyperdrive daemon has a deprecation notice and tells you to use hyperspace. Hyperspace [5] "provides remote access to Hypercores and a Hyperswarm instance" and "exposes a simple RPC interface". The documentation is very technical and seems to be aimed at developers, not using it as a tool. Digging around in the github organisation [6] or by stumbling upon the patreon [7] one can find the hyp CLI tool which is "A CLI for peer-to-peer file sharing (and more) using the Hypercore Protocol". The first commit is 9 days old. On the twitter of the author one can also find hyperbeam [8], which is integrated into hyp. Here on HN one can also find an announcement for "uplink" [9]. The tech looks pretty cool, but the vast amount of different projects makes it difficult to grasp. From all of these tools the dat tool seems to be the most advanced, but not actively maintained. It's not linked on any of the hyper* sites and doesn't seem to be the recommended way to use the hypercore protocol to share files p2p. While I would like to use the tech, I'm pretty lost on how to do that today. [0] https://docs.datproject.org/ [1] https://blog.datproject.org/2020/05/15/dat-protocol-renamed-hypercore-protocol/ [2] https://hypercore-protocol.org/ [3] https://hypercore-protocol.org/#hyperdrive [4] https://github.com/hypercore-protocol/hyperdrive [5] https://github.com/hypercore-protocol/hyperspace [6] https://github.com/hypercore-protocol/cli [7] https://www.patreon.com/posts/hyp-command-line-44923749 [8] https://github.com/mafintosh/hyperbeam [9] https://www.patreon.com/posts/paul-reveals-is-44665610

dwiel 6y ago
First off, as someone who has spent many hours trying to use dat, I agree 100% with your assessment. I have been using dat command line tool to p2p synchronize files on a LAN (and also over the internet) for a year or two now. The latest version is more stable for me than than the previous version and I really love it. It synchronizes files across multiple computers nearly instantly.

jokoon 6y ago

It's pretty nice to see that there are now several P2P techs. This one seems pretty good, but unless you have a single client and unless the protocol is defined to do a single thing and do it well, I can't see it thrive. For example bittorrent is a good tech because it only does one thing and users can understand it. Same for IPFS. Softwares like limewire or kazaa thrived because they were simple enough to use. Same goes for protocols. In my view, decentralization requires: * a client that runs on user's hardware * a p2p database and filesystem, with this kind of append-only log and verification, that runs on the client * server can still accelerate access access time Generally with this kind of system, the goal is that each user can publish data/content either in a public/private manner, without being dependent on servers. The only problem is being able to attract users and compete with popular platforms. If such a protocol+software can attract users who want a "privacy-aware" alternative to platforms like facebook, I could see it work. Sometimes I wonder why there isn't more money invested to build such platforms, because the p2p internet seems to be the only viable solution to fight privacy issues.

sull 6y ago
Yes and no. The tech is like the “Internet” and is largely born from academic work and continues forward as fundamental and foundational p2p networking advancements. It’s not a consumer product itself but obviously builders have and will fill that void. Beaker is the best example but eventually more and some projects you may have heard about are actually using some aspects of hypercore.

gioscarab 6y ago

It would be cool to run one of those P2P technologies over a protocol like PJON: https://github.com/gioblu/PJON https://github.com/gioblu/PJON until the network infrastructure is centralized P2P is relatively useless, it would be cool to see it applied on top of a private network infrastructure made by people.

anentropic 6y ago

> Somehow our networking tech has this big gap in-between LANs and Clouds I remember we used to have FTP we used to use it for all sorts of things... deploying websites, sending files to the print company etc