4 ms·
Curious: What do you need something like that for? It sounds like some kind of synced backup of a dataset represented by files on S3? With what you describe, i
by iamwil 3y ago
Curious: What do you need something like that for? It sounds like some kind of synced backup of a dataset represented by files on S3?
With what you describe, it kinda sounds like a Merkle-DAG CRDT. https://arxiv.org/pdf/2004.00107.pdf https://arxiv.org/pdf/2004.00107.pdf
The paper describes a Merkle-DAG that can sync two copies together, because it's also a CRDT. I don't think it's exactly what you want, but I think if you're building an implementation, it gives you the core ideas of what you're looking for.
I also detail Merkle-DAGs in my search.
https://interjectedfuture.com/crdts-turned-inside-out/ https://interjectedfuture.com/crdts-turned-inside-out/
If you're looking for something off-the-shelf, I haven't looked for implementations in the wild, so I can't make recommendations.
- atombender 3y agoI can't describe the exact use case, but suffice to say I'd like to incrementally replicate (in one direction) data, but I don't desire to track what has changed or having to index the data. I'm attracted to the simplicity behind Merkle trees where you can use hashes as a much smaller proxy for the upstream data, and even describe use the root-level hash as a "signature" representing the state of the dataset. A naive, non-Merkle-y solution would be to associate each datum with a modification timestamp, and simply grab any data where "updated_at > $last_timestamp"; this would require an index to filter efficiently, however. Another simple solution would be a log that tracked every insert, and a process could then tail this log; this would require all the bookeeping this entails. Thanks, I will check out those links.