4 ms·

Ask HN: P2P Databases?

I had an idea for a project that would be best served by publishing a database that all the clients would sync locally and query against directly. I've looked around a bit and I've found BigchainDB, however it's a quite heavy load. I want to update the authoritative database and have it generate deltas that are synced to the clients via a p2p like network, rather then having anything 'mined'. Is anyone aware of anything existing that would be of use?

pkhuong 5y ago
https://github.com/backtrace-labs/verneuil/ https://github.com/backtrace-labs/verneuil/ is one way to address the diffing / read replica part of the problem. I believe it's compatible with gossipping: most of the data is in small content-addressed chunks, with small manifests that tell clients what chunks to fetch and how to reassemble them to recreate a sqlite database. There's already client-side caching to persistent storage, and chunks can be fetched on demand. Sharing replication data P2P, while retaining the simplicity of a single authoritative writer per database, is explicitly part of the project's long-term goals!
- culpable_pickle 5y ago
  This is super cool. Thank you.
billconan 5y ago
https://orbitdb.org/ https://orbitdb.org/
- hecturchi 5y ago
  And https://github.com/ipfs/go-ds-crdt https://github.com/ipfs/go-ds-crdt
  - culpable_pickle 5y ago
    Fascinating. Thank's for letting me know about this!
- culpable_pickle 5y ago
  ipfs is something new to investigate. Have you used this? I'm a little concerned about the data pinning costs.
bbrks 5y ago
Couchbase Lite offers delta-sync and P2P replication over any transport. Has client libraries for all mobile OSes, Java, and recently C. CBL is backed largely by Sqlite with features layered on top. Local query support, server replication and a bunch of other nice features useful for an offline/P2p database. Disclaimer: Employee working on the server-side Sync Gateway component for Couchbase Mobile.
- culpable_pickle 5y ago
  This is super cool. Thanks for letting me know about it!
tacostakohashi 5y ago
Ignite works as you describe: https://ignite.apache.org/ https://ignite.apache.org/ I wouldn't really recommend this approach, I would think more in terms of subscriptions and topics and less of a 'database'.
- culpable_pickle 5y ago
  Thanks for this! I will give my approach more thought with a messaging system model.
GauntletWizard 5y ago
Git. Seriously, git is, depending on your data model, possibly the best delta compression and sync model you can find. There pitfalls and sync issues to consider, but if it's just going to be published by one authority, git is a great solution.
- culpable_pickle 5y ago
  This is an interesting idea. I wanted to use p2p distribution to save bandwidth costs, but maybe I have to re-evaluate with git in mind. Thanks for the idea!
  - GauntletWizard 5y ago
    Git can be immensely P2P, though there's a coordination problem that would be involved. There was an attempt at "Gittorrent" at one point, Don't know the current status or if there was ever a working prototype.
_448 5y ago
_448 5y ago

pkhuong 5y ago

https://github.com/backtrace-labs/verneuil/ https://github.com/backtrace-labs/verneuil/ is one way to address the diffing / read replica part of the problem. I believe it's compatible with gossipping: most of the data is in small content-addressed chunks, with small manifests that tell clients what chunks to fetch and how to reassemble them to recreate a sqlite database. There's already client-side caching to persistent storage, and chunks can be fetched on demand. Sharing replication data P2P, while retaining the simplicity of a single authoritative writer per database, is explicitly part of the project's long-term goals!

culpable_pickle 5y ago
This is super cool. Thank you.

billconan 5y ago

https://orbitdb.org/ https://orbitdb.org/

hecturchi 5y ago
And https://github.com/ipfs/go-ds-crdt https://github.com/ipfs/go-ds-crdt
- culpable_pickle 5y ago
  Fascinating. Thank's for letting me know about this!
culpable_pickle 5y ago
ipfs is something new to investigate. Have you used this? I'm a little concerned about the data pinning costs.

bbrks 5y ago

Couchbase Lite offers delta-sync and P2P replication over any transport. Has client libraries for all mobile OSes, Java, and recently C. CBL is backed largely by Sqlite with features layered on top. Local query support, server replication and a bunch of other nice features useful for an offline/P2p database. Disclaimer: Employee working on the server-side Sync Gateway component for Couchbase Mobile.

culpable_pickle 5y ago
This is super cool. Thanks for letting me know about it!

tacostakohashi 5y ago

Ignite works as you describe: https://ignite.apache.org/ https://ignite.apache.org/ I wouldn't really recommend this approach, I would think more in terms of subscriptions and topics and less of a 'database'.

culpable_pickle 5y ago
Thanks for this! I will give my approach more thought with a messaging system model.

GauntletWizard 5y ago

Git. Seriously, git is, depending on your data model, possibly the best delta compression and sync model you can find. There pitfalls and sync issues to consider, but if it's just going to be published by one authority, git is a great solution.

culpable_pickle 5y ago
This is an interesting idea. I wanted to use p2p distribution to save bandwidth costs, but maybe I have to re-evaluate with git in mind. Thanks for the idea!
- GauntletWizard 5y ago
  Git can be immensely P2P, though there's a coordination problem that would be involved. There was an attempt at "Gittorrent" at one point, Don't know the current status or if there was ever a working prototype.

_448 5y ago