4 ms·

Since we’re on the subject of logs and embarassingly parallel distributed systems, I know someone who’s also in NYC who’s been building a project exactly along

by EGreg 2y ago

Since we’re on the subject of logs and embarassingly parallel distributed systems, I know someone who’s also in NYC who’s been building a project exactly along these lines. It’s called gossiplog and it uses Prolly trees to make some interesting results. https://www.npmjs.com/package/@canvas-js/gossiplog https://www.npmjs.com/package/@canvas-js/gossiplog Joel Gustafson started this stuff at MIT and used to work at Protocol Labs. It’s very straightforward. By any chance sewen do you know him? I first became aware of this guy’s work when he posted “merklizing the key value store for fun and profit” or something like that. Afterwards I looked at log protocols, including SLEEP protocol for Dat/Hypercore/ pear and time-travel DBs that track diffs, including including Dolt and even Quadrable. https://news.ycombinator.com/item?id=36265429 https://news.ycombinator.com/item?id=36265429 Gossiplog’s README says exactly what this article says— everything is a log underneath and if you can sync that (using prolly tree techniques) people can just focus on business logic and get sync for free!

sewen 2y ago
Never encountered it before, but it looks cool. I think they are trying to solve a related problem. "We can consolidate the work by making a generic log that has networking and syncing built-in. This can be used by developers to make automatically-decentralized apps without writing a single line of networking code." At a first glance, I would say that Gossiplog is a bit more low level, targeting developers of databases and queues, to save them from re-building a log every time. But then there are elements of sharing the log between components. Worth a deeper look, but seems a bit lower level abstraction.
- EGreg 2y ago
  It’s part of his higher-level framework called Canvas. Check this out: https://joelgustafson.com/posts/2024-09-30/introduction-to-causal-logs https://joelgustafson.com/posts/2024-09-30/introduction-to-c... And this: https://github.com/canvasxyz/canvas https://github.com/canvasxyz/canvas
vdm 2y ago
Thank you @EGreg for sharing this.
- deleted 2y ago
  [deleted]
- EGreg 2y ago
  Def. I geek out on this stuff, as I am building my own distributed systems. I have had discussions with a lot of people in the space, like Leslie Lamport, Petar Maymounkov etc. You might like this interview: https://www.youtube.com/watch?v=JWrRqUkJpMQ https://www.youtube.com/watch?v=JWrRqUkJpMQ This is what I’m working on now: https://intercoin.org/intercloud.pdf https://intercoin.org/intercloud.pdf
hem777 2y ago
There’s also OrbitDB https://github.com/orbitdb/orbitdb https://github.com/orbitdb/orbitdb which to my understanding has been a pioneer for p2p logs, databases and CRDTs.

by EGreg 2y ago

sewen 2y ago

Never encountered it before, but it looks cool. I think they are trying to solve a related problem. "We can consolidate the work by making a generic log that has networking and syncing built-in. This can be used by developers to make automatically-decentralized apps without writing a single line of networking code." At a first glance, I would say that Gossiplog is a bit more low level, targeting developers of databases and queues, to save them from re-building a log every time. But then there are elements of sharing the log between components. Worth a deeper look, but seems a bit lower level abstraction.

EGreg 2y ago
It’s part of his higher-level framework called Canvas. Check this out: https://joelgustafson.com/posts/2024-09-30/introduction-to-causal-logs https://joelgustafson.com/posts/2024-09-30/introduction-to-c... And this: https://github.com/canvasxyz/canvas https://github.com/canvasxyz/canvas

vdm 2y ago

Thank you @EGreg for sharing this.

deleted 2y ago
[deleted]
EGreg 2y ago
Def. I geek out on this stuff, as I am building my own distributed systems. I have had discussions with a lot of people in the space, like Leslie Lamport, Petar Maymounkov etc. You might like this interview: https://www.youtube.com/watch?v=JWrRqUkJpMQ https://www.youtube.com/watch?v=JWrRqUkJpMQ This is what I’m working on now: https://intercoin.org/intercloud.pdf https://intercoin.org/intercloud.pdf

hem777 2y ago

There’s also OrbitDB https://github.com/orbitdb/orbitdb https://github.com/orbitdb/orbitdb which to my understanding has been a pioneer for p2p logs, databases and CRDTs.