3 ms·

I tried to create a collaborative (the most important point), local-first, end-to-end encrypted app. I really tried. But it's hard, really hard. Like all distr

I tried to create a collaborative (the most important point), local-first, end-to-end encrypted app. I really tried. But it's hard, really hard. Like all distributed systems. CRDTs are far from a panacea: * No implementation is compatible with another * There is very few implementation in languages that are not JS. * The biggest problem is when 2 documents are diverging for too much time (think a blockchain fork) like if you go offline for 4 days while your coworker work on the same document as you do. You have to come back to differential sync, like Git. And it's just for sync, then to move to P2P you have to handle NAT traversals, distributed identities, rendez-vous places and much more.

LukeEF 6y ago
You speak my pain... took so much pain to overcome these problems when delivering TerminusDB... taking best ideas from blockchain, git and rolling them all together... this white paper we wrote might remind you of past suffering... https://github.com/terminusdb/terminusdb-server/blob/dev/docs/whitepaper/terminusdb.pdf https://github.com/terminusdb/terminusdb-server/blob/dev/doc...
dspillett 6y ago
For a lot of people for a lot of applications and tools, this level of sync isn't all that necessary though. If you are working on something yourself and not as part of a team then the age-old save-document-and-reopen-elsewhere method could work as well as it ever did. You don't have to have it open on your phone, laptop, desktop, and fire stick all at the same time with immediate sync. Have your offline documents on each machine, automatically back them up to "the cloud". If you edit one in two places because of disconnection issues then do a manual merge like we did in the old days (hopefully it won't happen often enough to be that much of a pain). In version 2+ of your product, once the base features are proven viable (and for commercial use, profitable), start to worry about automatic sync/merge/other with a granularity smaller than a whole document. A lot of offline-first efforts are trying to solve every possible use case from day one, from single person with a laptop to huge international teams all working on the same asset at the same time and everything in between & beside, and in that direction madness lies. Implementing a useful, viable, relatively minimal product, and expanding its capabilities later would make more sense, though of course that suggestion is immediately countered by fears that the next clone of the idea will implement something first and be crowned king leading the thought process back to "must do it all, must do it all now".

LukeEF 6y ago

You speak my pain... took so much pain to overcome these problems when delivering TerminusDB... taking best ideas from blockchain, git and rolling them all together... this white paper we wrote might remind you of past suffering... https://github.com/terminusdb/terminusdb-server/blob/dev/docs/whitepaper/terminusdb.pdf https://github.com/terminusdb/terminusdb-server/blob/dev/doc...

dspillett 6y ago

For a lot of people for a lot of applications and tools, this level of sync isn't all that necessary though. If you are working on something yourself and not as part of a team then the age-old save-document-and-reopen-elsewhere method could work as well as it ever did. You don't have to have it open on your phone, laptop, desktop, and fire stick all at the same time with immediate sync. Have your offline documents on each machine, automatically back them up to "the cloud". If you edit one in two places because of disconnection issues then do a manual merge like we did in the old days (hopefully it won't happen often enough to be that much of a pain). In version 2+ of your product, once the base features are proven viable (and for commercial use, profitable), start to worry about automatic sync/merge/other with a granularity smaller than a whole document. A lot of offline-first efforts are trying to solve every possible use case from day one, from single person with a laptop to huge international teams all working on the same asset at the same time and everything in between & beside, and in that direction madness lies. Implementing a useful, viable, relatively minimal product, and expanding its capabilities later would make more sense, though of course that suggestion is immediately countered by fears that the next clone of the idea will implement something first and be crowned king leading the thought process back to "must do it all, must do it all now".