4 ms·

Hi Raph, your GitHub comment about "Why CRDT didn't work out as well for collaborative editing in xi-editor" [1] was sent to me several times as an argument no

Hi Raph, your GitHub comment about "Why CRDT didn't work out as well for collaborative editing in xi-editor" [1] was sent to me several times as an argument not to use CRDTs. I agree with you that CRDTs might be too complex for the way the xi-editor used it (although I loved the idea, and appreciate that you tried it). But the title of the Hackernews post tells a very different story. So many people misunderstood WHY CRDTs didn't work out well for the xi-editor. At the very least, my article shows that CRDTs are well suited for shared editing. You brought up some valid points why CRDTs are not well suited for having different editor components communicate with each other (language server, indentation, syntax highlighting, ..). Although, I still think there is a lot of merit in using CRDTs as a data model for a code editor (or any kind of editor). Not for editor components concurrently modifying the editor model, but just as an collaboration-aware model. • Marijn considered a CRDT as a data model for CodeMirror 6 because positions in collaborative software can be better expressed [2]. A position in the Yjs model is simply a reference to the unique id of the character. • Even without collaboration, Yjs servers as a highly efficient selective Undo/Redo manager. Each item on the Undo/Redo stack just consumes a couple of bytes. Furthermore, most existing implementations don't support selective Undo/Redo. This is free when using Yjs as a data model. • Some components can work in the background and annotate the CRDT model (not manipulate it). For example, a code analyzer that runs on a remote computer could annotate a function and notify the user about potential problems. The position of the annotation will still be valid if users modify the model concurrently in a distributed environment. [1]: https://news.ycombinator.com/item?id=19886883 https://news.ycombinator.com/item?id=19886883 [2]: https://marijnhaverbeke.nl/blog/collaborative-editing-cm.html https://marijnhaverbeke.nl/blog/collaborative-editing-cm.htm...

raphlinus 6y ago
I'm not sure what to add. I definitely agree that CRDT is viable for collaborative editing, I'm just saying people need to be aware of the issues. Regarding position and undo, these are problems that can also be solved just fine using OT techniques, and are (imho) simpler when there is a central authority that can order all revisions into a globally consistent sequence. I think it's inevitable that people are going to misunderstand arguments, given the complexity of the underlying space and the paucity of good learning resources. So thanks for your writeup, it adds to the discussion.
- kevinjahns 6y ago
  I think we can agree on that. CRDT vs OT is a complex topic with lots of pros and cons for either side.
- josephg 6y ago
  Just to add my 2c, having written sharejs and sharedb and working on and off on OT systems for the past 10 years or so: After playing with a simple implementation of Martin Kleppmann’s newest automerge work, I was wrong to doubt CRDTs. I’m seeing about 6M ops / second in a prototype text engine in rust. I have much more to say on this - probably a blog post more to say. But I think CRDTs are the future for many workloads, and I have a strong sense of despair seeing how much time I’ve wasted investing in approaches which won’t feature strongly in the future. Automerge is really good.
  - kevinjahns 6y ago
    > I have a strong sense of despair seeing how much time I’ve wasted investing in approaches which won’t feature strongly in the future. Your comment makes me sad on several levels.. Firstly, ShareJS and OT types was the first open framework to build collaborative applications on the web. Your work inspired me to work on shared editing - I just wanted it to work over WebRTC. Until 2015, JavaScript engines implemented different garbage collection approaches that wouldn't allow efficient CRDT implementations (as they need to handle millions of objects). The idea of distributed applications on the web just popped up a couple of years ago. WebRTC didn't even exist when you started ShareJS. Even Websockets were still a bit experimental. OT was the right technology at the time. Lastly, I'd love for you to try out Yjs. The current JavaScript implementation handles 2.5 million operations / second. A Rust/C implementation would surely handle more than that as it is not limited by automatic garbage collection. You can compare Automerge's current performance branch with Yjs' current implementation in [1]. To be fair, their implementation is not finished. If I understood correctly, they want to load the compressed format directly into memory. I played with this idea a few years ago and represented the Yjs model in an ArrayBuffer. This approach will improve load-time, but the performance will be intolerable in other aspects (e.g. when applying document updates, transforming the document to a String/JSON, or when computing diffs). The performance of Yjs is the result of more than five years research. I still have a couple of ideas to improve performance significantly. Although, my article clearly shows that it is definitely good enough now. [1]: https://github.com/dmonad/crdt-benchmarks/pull/4 https://github.com/dmonad/crdt-benchmarks/pull/4

raphlinus 6y ago

I'm not sure what to add. I definitely agree that CRDT is viable for collaborative editing, I'm just saying people need to be aware of the issues. Regarding position and undo, these are problems that can also be solved just fine using OT techniques, and are (imho) simpler when there is a central authority that can order all revisions into a globally consistent sequence. I think it's inevitable that people are going to misunderstand arguments, given the complexity of the underlying space and the paucity of good learning resources. So thanks for your writeup, it adds to the discussion.

kevinjahns 6y ago
I think we can agree on that. CRDT vs OT is a complex topic with lots of pros and cons for either side.
josephg 6y ago
Just to add my 2c, having written sharejs and sharedb and working on and off on OT systems for the past 10 years or so: After playing with a simple implementation of Martin Kleppmann’s newest automerge work, I was wrong to doubt CRDTs. I’m seeing about 6M ops / second in a prototype text engine in rust. I have much more to say on this - probably a blog post more to say. But I think CRDTs are the future for many workloads, and I have a strong sense of despair seeing how much time I’ve wasted investing in approaches which won’t feature strongly in the future. Automerge is really good.
- kevinjahns 6y ago
  > I have a strong sense of despair seeing how much time I’ve wasted investing in approaches which won’t feature strongly in the future. Your comment makes me sad on several levels.. Firstly, ShareJS and OT types was the first open framework to build collaborative applications on the web. Your work inspired me to work on shared editing - I just wanted it to work over WebRTC. Until 2015, JavaScript engines implemented different garbage collection approaches that wouldn't allow efficient CRDT implementations (as they need to handle millions of objects). The idea of distributed applications on the web just popped up a couple of years ago. WebRTC didn't even exist when you started ShareJS. Even Websockets were still a bit experimental. OT was the right technology at the time. Lastly, I'd love for you to try out Yjs. The current JavaScript implementation handles 2.5 million operations / second. A Rust/C implementation would surely handle more than that as it is not limited by automatic garbage collection. You can compare Automerge's current performance branch with Yjs' current implementation in [1]. To be fair, their implementation is not finished. If I understood correctly, they want to load the compressed format directly into memory. I played with this idea a few years ago and represented the Yjs model in an ArrayBuffer. This approach will improve load-time, but the performance will be intolerable in other aspects (e.g. when applying document updates, transforming the document to a String/JSON, or when computing diffs). The performance of Yjs is the result of more than five years research. I still have a couple of ideas to improve performance significantly. Although, my article clearly shows that it is definitely good enough now. [1]: https://github.com/dmonad/crdt-benchmarks/pull/4 https://github.com/dmonad/crdt-benchmarks/pull/4