6 ms·

P2panda: P2P protocol for secure, energy-efficient local-first web applications

217 points by rapnie 3y ago

mdhb 3y ago
Seeing that this was put together by one of the co-authors of the recently released MLS (Message Layer Security) spec and backed by an EU funding grant are both interesting tidbits that got me more interested in this. This would indeed allow for a very different internet and would provide some viable options untethered to big tech as it exists today.
- raphaelrobert 3y ago
  I case you refer to me as the MLS co-author, I'm not directly affiliated with the p2panda project. I think p2panda uses OpenMLS (OSS implementation of the MLS protocol, https://openmls.tech https://openmls.tech), hence the connection. I did however exchange with this friendly an motivated team!
  - mdhb 3y ago
    Thanks for clarifying that. I’m sorry if I misunderstood but I really appreciate you pointing out those facts.
  - andreasdz 3y ago
    Yes, it's been a pleasure to work with OpenMLS, they consulted us with the integration into p2panda! As mentioned in my other comment: while most low-level parts have already been written, we still need to work on our high-level APIs, which should allow for easy integration of data encryption when using our SDKs. We aim at doing this in our next funding period.
actionfromafar 3y ago
Should I be this disappointed that there is no C or C++ library?
- emilecantin 3y ago
  Well, it says WEB applications, so presumably this is something that will run in a browser, which implies some flavour of Javascript. Tangentially, I've been interested in this space for a while, and I have a few in-progress projects built on CouchDB. My aim was to offer the offline app for free and monetize through sync / sharing features. This would've mirrored my cost structure (hosting), so I thought it was a pretty fair way to monetize a product.
  - nofollow 3y ago
    easy to find from the homepage: https://p2panda.org/libraries/ https://p2panda.org/libraries/ > The core library is shipped both as a Rust crate p2panda-rs with WebAssembly bindings and a NPM package p2panda-js with TypeScript definitions running in NodeJS or any modern web browser.
- wmf 3y ago
  [flagged]
  - msm_ 3y ago
    I'm pretty sure it's not. Maybe you wish it was, but it's not. C is still commonly used as a "lowest common denominator" language, and other languages often use bindings to native C libraries. This is still very common, and well supported in almost all modern languages (sometimes even completely embraced by some implementations, like cpython). Even Signal has official C bindings (libsignal-protocol-c). To reiterate my point, one may hope that C goes away, but objectively, in the foreseeable future, it continues to be a backbone of most software.
    - TeeMassive 3y ago
      Rust has C bindings too
      - zer8k 3y ago
Jhsto 3y ago
I wonder how complimentary this could be for etcd, which tends to not work so great for structured data.
brunoqc 3y ago
Is it blockchain?
- orthecreedence 3y ago
  No. Just old fashioned p2p with signed transaction chains. No consensus, as far as I can tell.
  - tommiegannert 3y ago
    Isn't "consensus" just another word for "(eventual) consistency"? I don't see how you could build a distributed database without tackling the consensus problem.
    - lifty 3y ago
      More information can be found here: https://p2panda.org/specification/replication https://p2panda.org/specification/replication. The basic data sync is based on an append only log. But I can't find any information on what are the strategies for dealing with conflicts. If any of the contributors see this, could you comment on this please. Thanks!
      - andreasdz 3y ago
        Check out our section on "Operations", this is the data type we've built on top of the append-only log structure for multi-writer and conflict free data editing: https://p2panda.org/learn/operations https://p2panda.org/learn/operations
    - orthecreedence 3y ago
      > Isn't "consensus" just another word for "(eventual) consistency"? In my head, there's a distinction. Consensus can be used for non-commutative operations, whereas signed CRDT chains can operate entirely without any form of "agreement" other than how they are applied (see https://news.ycombinator.com/item?id=30560573 https://news.ycombinator.com/item?id=30560573). I wouldn't call distributed CRDTs "consensus."
jedahan 3y ago
I wish I could read the specification offline. It's split into a lot of smaller pages.
chaostheory 3y ago
I’ve been waiting for something like this. There are still many places even in developed countries where a stable network connection isn’t guaranteed
- orthecreedence 3y ago
  You might like Holochain (https://www.holochain.org/ https://www.holochain.org/) as well, it's in a similar vein.
andreasdz 3y ago
Oh wow, we didn't expect p2panda to pop up here (I'm adz, one of the core contributors). Let us know if you have any questions :-) This week we're wrapping up a long development phase for a new "aquadoggo" version (https://github.com/p2panda/aquadoggo/ https://github.com/p2panda/aquadoggo/) - our p2panda node reference implementation. The release will come with a dynamic GraphQL query (filtering, cursor pagination, text search and ordering), blob support, a better configuration interface and a full p2p networking stack (QUIC) with NAT traversal techniques and relays based on libp2p. Next steps in our current funding period is a showcase app for Android (built with Flutter and Rust FFI bindings from our core libraries), a bee sighting database for scientists in the Brazilian Amazon (in collaboration with https://www.meli-bees.org/ https://www.meli-bees.org/). Further down the lane we're continuing our integration of MLS - as some high-level APIs are still missing - and a capability system. We're definitely not out of ideas yet!
- mdhb 3y ago
  If that means you will also have a Dart / Flutter package available for development I would be very interested.
  - andreasdz 3y ago
    We've started implementing one here as part of the "Meli" Android App project: https://github.com/p2panda/meli/ https://github.com/p2panda/meli/ - as soon as we're happy with the API we would factor it out in a separate repository and publish it!
    - mdhb 3y ago
      Excellent, I did end up taking a look. Hopefully using the FFI approach worked out ok for you folks in the end
    - kaichao 3y ago
      Curios to know why Meli only has Android support? Since Flutter + Rust (FFI) seems a good solution for multi moblie platform support.
someguy7250 3y ago
Wow! This might just be what I've been hoping for, since 2020. What I hoped for was more of a generic backend for multiplayer games. But this is even better. It looks like a next-gen bittorrent which natively supports operations? How did the EU support something like this without worrying about misuse? It would be fun to build something like a Blender plugin that allows people to cooperate on free 3D models and animations. But it's also unclear whether anyone would be legally liable when something goes wrong. (Edit: It sounds like we want to avoid legal issues by having everyone run their own local instances. I like that feeling of hosting our own servers. But something just feels odd. Since when are we allowed to be so open? What's the catch?)
CGamesPlay 3y ago
Looks interesting, but there doesn’t seem to be any information about what CRDT it’s using, or how it handles concurrent editing at all. The only operation appears to be “set key”, so it looks pretty basic. Is this the target state or is this going to be expanded upon in the future?
- capableweb 3y ago
  https://p2panda.org/specification/data-types/documents https://p2panda.org/specification/data-types/documents > A document is made up of operations published by one or many authors > Every operation has a previous field containing a document_view_id which refers to document state at the moment the operation was encoded > DO3: An operation which refers to the current operation in its previous field MUST be sorted next. As far as I understand, it would be up to the client to present potential conflicts and won't be a valid document until it has been resolved.
- andreasdz 3y ago
  We're using our own CRDT called "Operations" giving us multi-writer conflict-free editing. It's a simple key/value map with a last-write win rule while we keep some sort of vector clock for every write to understand what every peer has seen when they updated the Document. This might be limiting for some cases, though so far we could model many applications already with such simple CRDT. It is also possible to add your own or already existing CRDT frameworks on top of p2panda.
mble 3y ago
What is the difference compared to IPFS and Torrent?

217 points by rapnie 3y ago

mdhb 3y ago

Seeing that this was put together by one of the co-authors of the recently released MLS (Message Layer Security) spec and backed by an EU funding grant are both interesting tidbits that got me more interested in this. This would indeed allow for a very different internet and would provide some viable options untethered to big tech as it exists today.

raphaelrobert 3y ago
I case you refer to me as the MLS co-author, I'm not directly affiliated with the p2panda project. I think p2panda uses OpenMLS (OSS implementation of the MLS protocol, https://openmls.tech https://openmls.tech), hence the connection. I did however exchange with this friendly an motivated team!
- mdhb 3y ago
  Thanks for clarifying that. I’m sorry if I misunderstood but I really appreciate you pointing out those facts.
- andreasdz 3y ago
  Yes, it's been a pleasure to work with OpenMLS, they consulted us with the integration into p2panda! As mentioned in my other comment: while most low-level parts have already been written, we still need to work on our high-level APIs, which should allow for easy integration of data encryption when using our SDKs. We aim at doing this in our next funding period.

actionfromafar 3y ago

Should I be this disappointed that there is no C or C++ library?

emilecantin 3y ago
Well, it says WEB applications, so presumably this is something that will run in a browser, which implies some flavour of Javascript. Tangentially, I've been interested in this space for a while, and I have a few in-progress projects built on CouchDB. My aim was to offer the offline app for free and monetize through sync / sharing features. This would've mirrored my cost structure (hosting), so I thought it was a pretty fair way to monetize a product.
- nofollow 3y ago
  easy to find from the homepage: https://p2panda.org/libraries/ https://p2panda.org/libraries/ > The core library is shipped both as a Rust crate p2panda-rs with WebAssembly bindings and a NPM package p2panda-js with TypeScript definitions running in NodeJS or any modern web browser.
wmf 3y ago
[flagged]
- msm_ 3y ago
  I'm pretty sure it's not. Maybe you wish it was, but it's not. C is still commonly used as a "lowest common denominator" language, and other languages often use bindings to native C libraries. This is still very common, and well supported in almost all modern languages (sometimes even completely embraced by some implementations, like cpython). Even Signal has official C bindings (libsignal-protocol-c). To reiterate my point, one may hope that C goes away, but objectively, in the foreseeable future, it continues to be a backbone of most software.
  - TeeMassive 3y ago
    Rust has C bindings too
    - zer8k 3y ago

Jhsto 3y ago

I wonder how complimentary this could be for etcd, which tends to not work so great for structured data.

brunoqc 3y ago

Is it blockchain?

orthecreedence 3y ago
No. Just old fashioned p2p with signed transaction chains. No consensus, as far as I can tell.
- tommiegannert 3y ago
  Isn't "consensus" just another word for "(eventual) consistency"? I don't see how you could build a distributed database without tackling the consensus problem.
  - lifty 3y ago
    More information can be found here: https://p2panda.org/specification/replication https://p2panda.org/specification/replication. The basic data sync is based on an append only log. But I can't find any information on what are the strategies for dealing with conflicts. If any of the contributors see this, could you comment on this please. Thanks!
    - andreasdz 3y ago
      Check out our section on "Operations", this is the data type we've built on top of the append-only log structure for multi-writer and conflict free data editing: https://p2panda.org/learn/operations https://p2panda.org/learn/operations
  - orthecreedence 3y ago
    > Isn't "consensus" just another word for "(eventual) consistency"? In my head, there's a distinction. Consensus can be used for non-commutative operations, whereas signed CRDT chains can operate entirely without any form of "agreement" other than how they are applied (see https://news.ycombinator.com/item?id=30560573 https://news.ycombinator.com/item?id=30560573). I wouldn't call distributed CRDTs "consensus."

jedahan 3y ago

I wish I could read the specification offline. It's split into a lot of smaller pages.

chaostheory 3y ago

I’ve been waiting for something like this. There are still many places even in developed countries where a stable network connection isn’t guaranteed

orthecreedence 3y ago
You might like Holochain (https://www.holochain.org/ https://www.holochain.org/) as well, it's in a similar vein.

andreasdz 3y ago

Oh wow, we didn't expect p2panda to pop up here (I'm adz, one of the core contributors). Let us know if you have any questions :-) This week we're wrapping up a long development phase for a new "aquadoggo" version (https://github.com/p2panda/aquadoggo/ https://github.com/p2panda/aquadoggo/) - our p2panda node reference implementation. The release will come with a dynamic GraphQL query (filtering, cursor pagination, text search and ordering), blob support, a better configuration interface and a full p2p networking stack (QUIC) with NAT traversal techniques and relays based on libp2p. Next steps in our current funding period is a showcase app for Android (built with Flutter and Rust FFI bindings from our core libraries), a bee sighting database for scientists in the Brazilian Amazon (in collaboration with https://www.meli-bees.org/ https://www.meli-bees.org/). Further down the lane we're continuing our integration of MLS - as some high-level APIs are still missing - and a capability system. We're definitely not out of ideas yet!

mdhb 3y ago
If that means you will also have a Dart / Flutter package available for development I would be very interested.
- andreasdz 3y ago
  We've started implementing one here as part of the "Meli" Android App project: https://github.com/p2panda/meli/ https://github.com/p2panda/meli/ - as soon as we're happy with the API we would factor it out in a separate repository and publish it!
  - mdhb 3y ago
    Excellent, I did end up taking a look. Hopefully using the FFI approach worked out ok for you folks in the end
  - kaichao 3y ago
    Curios to know why Meli only has Android support? Since Flutter + Rust (FFI) seems a good solution for multi moblie platform support.

someguy7250 3y ago

Wow! This might just be what I've been hoping for, since 2020. What I hoped for was more of a generic backend for multiplayer games. But this is even better. It looks like a next-gen bittorrent which natively supports operations? How did the EU support something like this without worrying about misuse? It would be fun to build something like a Blender plugin that allows people to cooperate on free 3D models and animations. But it's also unclear whether anyone would be legally liable when something goes wrong. (Edit: It sounds like we want to avoid legal issues by having everyone run their own local instances. I like that feeling of hosting our own servers. But something just feels odd. Since when are we allowed to be so open? What's the catch?)

CGamesPlay 3y ago

Looks interesting, but there doesn’t seem to be any information about what CRDT it’s using, or how it handles concurrent editing at all. The only operation appears to be “set key”, so it looks pretty basic. Is this the target state or is this going to be expanded upon in the future?

capableweb 3y ago
https://p2panda.org/specification/data-types/documents https://p2panda.org/specification/data-types/documents > A document is made up of operations published by one or many authors > Every operation has a previous field containing a document_view_id which refers to document state at the moment the operation was encoded > DO3: An operation which refers to the current operation in its previous field MUST be sorted next. As far as I understand, it would be up to the client to present potential conflicts and won't be a valid document until it has been resolved.
andreasdz 3y ago
We're using our own CRDT called "Operations" giving us multi-writer conflict-free editing. It's a simple key/value map with a last-write win rule while we keep some sort of vector clock for every write to understand what every peer has seen when they updated the Document. This might be limiting for some cases, though so far we could model many applications already with such simple CRDT. It is also possible to add your own or already existing CRDT frameworks on top of p2panda.

mble 3y ago

What is the difference compared to IPFS and Torrent?