2 ms·

I'm completely into the idea that your data should be document-centric, this I've seen working at a very large scale (i.e. 100+ MB JSON blobs being updated at 6

by blixt 2y ago

I'm completely into the idea that your data should be document-centric, this I've seen working at a very large scale (i.e. 100+ MB JSON blobs being updated at 60 FPS with a large team of people editing at the same time) with the structure we built at Framer. What I'm still not sure about is how superior CRDTs are if you're building a centralized service. CRDTs can be great for offline mode or syncing data across long time periods, and not necessarily for an online and realtime multiplayer experience. I know they work just fine in such a situation, but they do come with additional cost in logical complexity, memory, and compute. Does anyone have experience with doing first "old school" sequential data patching (which would need some additional work to support undo and synchronization with clients that are trying to make simultaneous changes) and then switching to CRDT? Last time I tried it was too costly in every way, so I'm curious.

dartos 2y ago
Specifically yjs provides a great experience for building CRDT editors. I think it’s a big selling point. I recently built a centralized, multiplayer, editor as a POC. I did one iteration with hand rolled operational transforms (what google docs uses) and one with yjs. Yjs just kind of works, where OTs require me to handle different kinds of data in bespoke ways and resolving conflicts is much more difficult due to the manual, one off nature of OT based systems. Yjs’s CRDT approach is much simpler to get going with and, imo, CRDTs are easier to reason about than OTs. Yjs’s implementation of CRDTs is quite a bit more space efficient than other implementations I’ve seen. I haven’t dealt with either in a production scenario yet, but I’m going to be moving forward with Yjs crdts. There are native bindings as well.
- blixt 2y ago
  Yeah I think CRDT or OT can work well for multiplayer rich text editing for example, but then I would scope the domain to just that in such a way that you still keep the bulk of your document "dumb", e.g. as JSON patches on a big JSON object, maybe with some special logic for nicer array insertion/deletion. > OTs require me to handle different kinds of data in bespoke ways I'll admit I haven't tried to implement a general OT solution, but CRDTs are also extremely complex functions on the base building blocks of your data. I have tried Yjs and it does indeed hide away a lot of this complexity for you, but I would be surprised if there isn't an equivalent OT library (a quick Google turns up https://www.npmjs.com/package/@otjs/state-machine https://www.npmjs.com/package/@otjs/state-machine). Furthermore the CRDTs uses quite a lot of memory as your project scales up, so I'd keep an eye on that. I'd be curious to see more of your project even if it's still in progress, is there already something live?
  - dartos 2y ago
    > I'll admit I haven't tried to implement a general OT solution, but CRDTs are also extremely complex functions on the base building blocks of your data. I can’t really speak to CRDTs more generally, but yjs provides a few fairly general data structures which “just work.” In my (non extensive) experience with OTs, you need to write bespoke, application specific conflict resolution, but I haven’t tried that OTJS library. > I'd be curious to see more of your project even if it's still in progress, is there already something live? Not yet, hoping to have something by the end of the year. I have the domain name “outlines.work” if you feel like checking it out later. The yjs parts are very similar to this example repo, though: https://github.com/satoren/y-phoenix-channel/tree/main https://github.com/satoren/y-phoenix-channel/tree/main

by blixt 2y ago

dartos 2y ago

Specifically yjs provides a great experience for building CRDT editors. I think it’s a big selling point. I recently built a centralized, multiplayer, editor as a POC. I did one iteration with hand rolled operational transforms (what google docs uses) and one with yjs. Yjs just kind of works, where OTs require me to handle different kinds of data in bespoke ways and resolving conflicts is much more difficult due to the manual, one off nature of OT based systems. Yjs’s CRDT approach is much simpler to get going with and, imo, CRDTs are easier to reason about than OTs. Yjs’s implementation of CRDTs is quite a bit more space efficient than other implementations I’ve seen. I haven’t dealt with either in a production scenario yet, but I’m going to be moving forward with Yjs crdts. There are native bindings as well.

blixt 2y ago
Yeah I think CRDT or OT can work well for multiplayer rich text editing for example, but then I would scope the domain to just that in such a way that you still keep the bulk of your document "dumb", e.g. as JSON patches on a big JSON object, maybe with some special logic for nicer array insertion/deletion. > OTs require me to handle different kinds of data in bespoke ways I'll admit I haven't tried to implement a general OT solution, but CRDTs are also extremely complex functions on the base building blocks of your data. I have tried Yjs and it does indeed hide away a lot of this complexity for you, but I would be surprised if there isn't an equivalent OT library (a quick Google turns up https://www.npmjs.com/package/@otjs/state-machine https://www.npmjs.com/package/@otjs/state-machine). Furthermore the CRDTs uses quite a lot of memory as your project scales up, so I'd keep an eye on that. I'd be curious to see more of your project even if it's still in progress, is there already something live?
- dartos 2y ago
  > I'll admit I haven't tried to implement a general OT solution, but CRDTs are also extremely complex functions on the base building blocks of your data. I can’t really speak to CRDTs more generally, but yjs provides a few fairly general data structures which “just work.” In my (non extensive) experience with OTs, you need to write bespoke, application specific conflict resolution, but I haven’t tried that OTJS library. > I'd be curious to see more of your project even if it's still in progress, is there already something live? Not yet, hoping to have something by the end of the year. I have the domain name “outlines.work” if you feel like checking it out later. The yjs parts are very similar to this example repo, though: https://github.com/satoren/y-phoenix-channel/tree/main https://github.com/satoren/y-phoenix-channel/tree/main