5 ms·

Really awesome demo - the Beaker/Dat/Hypercore ecosystem is amazing. Real-time p2p sync of anything you can imagine. I was able to build a 'Slack clone' as a o

Really awesome demo - the Beaker/Dat/Hypercore ecosystem is amazing. Real-time p2p sync of anything you can imagine. I was able to build a 'Slack clone' as a one-day hack, using Hypercore + Electron: https://github.com/lachenmayer/p2p-slack-clone-poc https://github.com/lachenmayer/p2p-slack-clone-poc I think we'll be seeing a lot of these kind of projects soon :)

marknadal 9y ago
Hypercore has realtime sync, but is limited to log structured data. Dat can do diffs on data, but is meant for large datasets, not realtime changes. This sounds like it would be difficult (or not scalable) to build something like a Wiki, features in social networks, Trello, or other apps (i.e., anything with shared mutable state). How would you do this? The reason I ask is that me, Mafintosh, Juan at IPFS, Dominic with Scuttlebutt, Feross at WebTorrent, Substack, and others all met back in 2014 for our different P2P projects. All of us had slightly different approaches. Juan and others seemed mostly interested in hash addressing, which I think is great but doesn't solve the end problem of data sync. Seems like Dat deals with that fairly well, but not for highly mutable data (versus large scientific files). Meanwhile we ( https://github.com/amark/gun https://github.com/amark/gun ) tackled that problem first, because it seems like CRDTs are the most relevant for killing traditional centralized Facebook/Twitter/Reddit/gDocs like apps, and hashing is more applicable for killing centralized YouTube/imgur like apps. Both are necessary, but it certainly seems harder/easier based on which underlying P2P tools you use. You are one of the few people actually jumping in and building end apps (thanks for sharing your chat app!), and we need more people doing that (not just all of us who are trying to rebuild the underlying architecture). So I would be curious to hear your experience comparing the different use cases behind dat/IPFS/gun/WebTorrent/etc. it would make for an interesting and informative comparison article. Thoughts?
- maxogden 9y ago
  The real-time syncing immutable, append-only log that hypercore provides can also be accessed randomly, allowing for lots of cool parallel/distributed log processing architectures (Kappa architecture), similar to Kafka (which is also based on immutable, append-only logs). We have just focused on syncing a filesystem at first because we had a strong use case there, but you could totally build a CRDT on top of an append only log.
  - marknadal 9y ago
    Hey Max, long time no chat since Hash the Planet! Was it you or Mathias that demoed running an Operating System while streaming it live from remote peers? Those were some insanely sick demos. Is random access something you guys have added since then? Or, could you clarify how it reduces the overhead on apps that would have shared mutable state (even if it is composed of immutable streams)? Don't you still have to back-scan through the log to recompose the current view? Which then wouldn't scale for read performance. Or is there an alternative approach? This was one of the main things I was discussing with Dominic Tarr about Scuttlebutt back then. We had done a lot of event-sourcing (immutable logs) at my previous startup, but had problems with compaction or generating views/state of those streams. Which is what lead me to the CRDT approach as the base, not the thing on top. I know Tim Caswell/Creationix is using DAT for one of his products he is building with @coolaj86 . But that was 4 years ago now, that me and Dominic were talking about those problems? Does random access solve that? Would love to read about it, shoot me some links! Also, Dominic and I are starting to have Web Crypto meetings with some folks at MIT (anvil/trust, ex-riak/basho) and they are in town (the Bay Area) this week. If you are around, we should all get together again to discuss identity, security, crypto, and P2P!

marknadal 9y ago

Hypercore has realtime sync, but is limited to log structured data. Dat can do diffs on data, but is meant for large datasets, not realtime changes. This sounds like it would be difficult (or not scalable) to build something like a Wiki, features in social networks, Trello, or other apps (i.e., anything with shared mutable state). How would you do this? The reason I ask is that me, Mafintosh, Juan at IPFS, Dominic with Scuttlebutt, Feross at WebTorrent, Substack, and others all met back in 2014 for our different P2P projects. All of us had slightly different approaches. Juan and others seemed mostly interested in hash addressing, which I think is great but doesn't solve the end problem of data sync. Seems like Dat deals with that fairly well, but not for highly mutable data (versus large scientific files). Meanwhile we ( https://github.com/amark/gun https://github.com/amark/gun ) tackled that problem first, because it seems like CRDTs are the most relevant for killing traditional centralized Facebook/Twitter/Reddit/gDocs like apps, and hashing is more applicable for killing centralized YouTube/imgur like apps. Both are necessary, but it certainly seems harder/easier based on which underlying P2P tools you use. You are one of the few people actually jumping in and building end apps (thanks for sharing your chat app!), and we need more people doing that (not just all of us who are trying to rebuild the underlying architecture). So I would be curious to hear your experience comparing the different use cases behind dat/IPFS/gun/WebTorrent/etc. it would make for an interesting and informative comparison article. Thoughts?

maxogden 9y ago
The real-time syncing immutable, append-only log that hypercore provides can also be accessed randomly, allowing for lots of cool parallel/distributed log processing architectures (Kappa architecture), similar to Kafka (which is also based on immutable, append-only logs). We have just focused on syncing a filesystem at first because we had a strong use case there, but you could totally build a CRDT on top of an append only log.
- marknadal 9y ago
  Hey Max, long time no chat since Hash the Planet! Was it you or Mathias that demoed running an Operating System while streaming it live from remote peers? Those were some insanely sick demos. Is random access something you guys have added since then? Or, could you clarify how it reduces the overhead on apps that would have shared mutable state (even if it is composed of immutable streams)? Don't you still have to back-scan through the log to recompose the current view? Which then wouldn't scale for read performance. Or is there an alternative approach? This was one of the main things I was discussing with Dominic Tarr about Scuttlebutt back then. We had done a lot of event-sourcing (immutable logs) at my previous startup, but had problems with compaction or generating views/state of those streams. Which is what lead me to the CRDT approach as the base, not the thing on top. I know Tim Caswell/Creationix is using DAT for one of his products he is building with @coolaj86 . But that was 4 years ago now, that me and Dominic were talking about those problems? Does random access solve that? Would love to read about it, shoot me some links! Also, Dominic and I are starting to have Web Crypto meetings with some folks at MIT (anvil/trust, ex-riak/basho) and they are in town (the Bay Area) this week. If you are around, we should all get together again to discuss identity, security, crypto, and P2P!