6 ms·

Just use OT like normal people, it’s been proven to work. No tombstones, no infinite storage requirements or forced “compaction”, fairly easy to debug, algorith

Just use OT like normal people, it’s been proven to work. No tombstones, no infinite storage requirements or forced “compaction”, fairly easy to debug, algorithm is moderate to complex but there are reference open source implementations to cross check against. You need a server for OT but you’re always going to have a server anyway, one extra websocket won’t hurt you. We regularly have 30-50k websockets connected at a time. CRDTs are a meme and are not for serious applications.

chrisweekly 7mo ago
"CRDTs are a meme and are not for serious applications." That is one hot take!
- argee 7mo ago
  Let's balance the discussion a bit. https://josephg.com/blog/crdts-are-the-future/ https://josephg.com/blog/crdts-are-the-future/
  - chrisweekly 7mo ago
    Great link! Thanks for sharing.
antics 7mo ago
Author here, I did not specifically mention OT in the article, since our main focus was to help people understand the downsides of the currently-most-popular system, which is built on CRDTs. BUT, since you mention it, I'll say a bit here. It sounds like you have your own experience, and we'd love to hear about that. But OUR experience was: (1) we found (contrary to popular belief) that OT actually does not require a centralized server, (2) we found it to be harder to implement OT exactly right vs CRDTs, and (3) we found many (though not all) of the problems that CRDTs have, are also problems in practice for OT—although in fairness to OT, we think the problems CRDTs have in general are vastly worse to the end-user experience. If there's interest I'm happy to write a similar article entirely dedicated to OT. But, for (3), as intuition, we found a lot of the problems that both CRDTs and OT have seem to arise from a fundamental impedance mismatch between the in-memory representation of the state of a modern editor, and the representation that is actually synchronized. That is, when you apply an op (CRDT) or a transform (OT), you have to transform the change into a (to use ProseMirror as an example) valid `Transaction` on an `EditorState`. This is not always easy in either case, and to do it right you might have to think very hard about things like "how to preserve position mappings," and other parts of editor state that are crucial to (say) plugins that manage locations of comment marks or presence cursors. With all of that said, OT is definitely much closer to what modern editors need, in my opinion at least. The less-well-known algorithm we ended up recommending here (which I will call "Marjin Collab", after its author) is essentially a very lightweight OT, without the "transformation" step.
- samlinnfer 7mo ago
  Having a central server is not necessary, but we have one anyway and we use it, especially if you have a permissions system. It lets us use the "Google wave" algorithm which vastly simplifies things. https://svn.apache.org/repos/asf/incubator/wave/whitepapers/operational-transform/operational-transform.html https://svn.apache.org/repos/asf/incubator/wave/whitepapers/... > This is not always easy in either case, and to do it right you might have to think very hard about things like "how to preserve position mappings," and other parts of editor state that are crucial to (say) plugins that manage locations of comment marks or presence cursors. Maintaining text editor state is normal. Yes you do need to convert the OT messages into whatever diff format your editor requires (and back), but that's the standard glue code. The nice thing about OT is that you can just feed the positions of marks into the OT algorithm to get the new positional value. Worst case, you just have the server send the server side position when sending the OT event and the client just displays the server side position.
  - DonHopkins 7mo ago
    Josh eloquently explains how Google Wave's DACP (Distributed Application Canceling Protocol) works: https://www.youtube.com/watch?v=4Z4RKRLaSug https://www.youtube.com/watch?v=4Z4RKRLaSug
- ianhorn 7mo ago
  I always mentally slotted prosemirror-collab/your recommended solution in the OT category. What’s the difference between the “rebase” step and the “transformation” step you’re saying it doesn’t need?
  - antics 7mo ago
    Great question. Matt has a comment about this here, and he has an actual PhD on the subject! So rather than doing a worse job explaining I will leave it to him to explain: https://news.ycombinator.com/user?id=mweidner https://news.ycombinator.com/user?id=mweidner
- gritzko 7mo ago
MrJohz 7mo ago
Are there any major libraries for OT? I've been looking into this recently for a project at work, and OT would be completely sufficient for our use case, and does look simpler overall, but from what I could tell, we'd need to write a lot of stuff ourselves. The only vaguely active-looking project in JS at least seems to be DocNode (https://www.docukit.dev/docnode https://www.docukit.dev/docnode), and that looks very cool but also very early days.
- samlinnfer 7mo ago
  https://github.com/josephg/sharejs https://github.com/josephg/sharejs https://github.com/FirebaseExtended/firepad https://github.com/FirebaseExtended/firepad https://github.com/Operational-Transformation/ot.js https://github.com/Operational-Transformation/ot.js https://github.com/ottypes/docs https://github.com/ottypes/docs Really nice demo: https://operational-transformation.github.io https://operational-transformation.github.io
- antics 7mo ago
  Author here. I think it depends what you're doing! OT is a true distributed systems algorithm and to my knowledge there are no projects that implement true, distributed OT with strong support for modern rich text editor SDKs like ProseMirror. ShareJS, for example, is abandoned, and predates most modern editors. If you are using a centralized server and ProseMirror, there are several OT and pseudo-OT implementations. Most popularly, there is prosemirror-collab[4], which is basically "OT without the stuff you don't need with an authoritative source for documents." Practically speaking that means "OT without T", but because it does not transform the ops to be order-independent, it has an extra step on conflict where the user has to rebase changes and re-submit. This is can cause minor edit starvation of less-connected clients. prosemirror-collab-commit[5] fixes this by performing the rebasing on the server... so it's still "OT without the T", but also with an authoritative conflict resolution pseudo-T at the end. I personally recommend prosemirror-collab-commit, it's what we use, and it's extremely fast and predictable. If you just want something pedogocically helpful, the blessed upstream collaborative editing solution for CodeMirror is OT. See author's blog post[1], the @codemirror/collab package[2], and the live demo[3]. In general this implementation is quite good and worth reading if you are interested in this kind of thing. ShareJS and OTTypes are both very readable and very good, although we found them very challenging to adopt in a real-world ProseMirror-based editor. [1]: https://marijnhaverbeke.nl/blog/collaborative-editing-cm.html https://marijnhaverbeke.nl/blog/collaborative-editing-cm.htm... [2]: https://codemirror.net/docs/ref/#collab https://codemirror.net/docs/ref/#collab [3]: https://codemirror.net/examples/collab/ https://codemirror.net/examples/collab/ [4]: https://github.com/ProseMirror/prosemirror-collab https://github.com/ProseMirror/prosemirror-collab [5]: https://github.com/stepwisehq/prosemirror-collab-commit https://github.com/stepwisehq/prosemirror-collab-commit
r0b05 7mo ago
What is OT?
- hsx 7mo ago
  Operational Transformation: https://en.wikipedia.org/wiki/Operational_transformation https://en.wikipedia.org/wiki/Operational_transformation
- deleted 7mo ago
  [deleted]
fritzo 7mo ago
Agreed. In my limited experience, conflict resolution rules are very domain specific, whereas CTDTs encourage a lazy attitude that "if it's associative and commutative it must be correct".
leksak 7mo ago
What does OT stand for so I can learn more?
- cantaloupe 7mo ago
  Operational Transformation
josephg 7mo ago
> CRDTs are a meme and are not for serious applications. You don't think Figma is a serious application? By all means, use OT. I worked on OT software for many years - and my work on OT types, ShareJS and ShareDB is still in production all over the place. But I don't think there's anything you can do with OT that you can't do just as well with CRDTs. The only real benefit of OT is that its simpler to reason about. Maybe that's enough.
- GermanJablo 7mo ago
  > You don't think Figma is a serious application? I don't know where this popular belief came from. The Figma blog literally says "Figma isn't using true CRDTs"[1]. > The only real benefit of OT is that its simpler to reason about. That's incorrect. When you free yourself from the P2P restriction that CRDTs are subject to, there's a huge amount of metadata you can get rid of, just to mention one benefit. [1] https://www.figma.com/blog/how-figmas-multiplayer-technology-works/ https://www.figma.com/blog/how-figmas-multiplayer-technology...
  - nickthesick 7mo ago
    > I don't know where this popular belief came from. It may be worth reading the whole paragraph of the blog you referenced... > Figma isn't using true CRDTs though. CRDTs are designed for decentralized systems where there is no single central authority to decide what the final state should be. There is some unavoidable performance and memory overhead with doing this. Since Figma is centralized (our server is the central authority), we can simplify our system by removing this extra overhead and benefit from a faster and leaner implementation. > It’s also worth noting that Figma's data structure isn't a single CRDT. Instead it's inspired by multiple separate CRDTs and uses them in combination to create the final data structure that represents a Figma document (described below). So, it's multiple CRDTs, not just one. And they've made some optimizations on top, but that doesn't make it not a CRDT?
    - GermanJablo 7mo ago
      Nowhere does it say it's multiple CRDTs. It says "isn't a single CRDT" and that "it's inspired by multiple separate CRDTs." A bit confusing, I agree. By the way, I work at Figma.
deleted 7mo ago
[deleted]

chrisweekly 7mo ago

"CRDTs are a meme and are not for serious applications." That is one hot take!

argee 7mo ago
Let's balance the discussion a bit. https://josephg.com/blog/crdts-are-the-future/ https://josephg.com/blog/crdts-are-the-future/
- chrisweekly 7mo ago
  Great link! Thanks for sharing.

antics 7mo ago

Author here, I did not specifically mention OT in the article, since our main focus was to help people understand the downsides of the currently-most-popular system, which is built on CRDTs. BUT, since you mention it, I'll say a bit here. It sounds like you have your own experience, and we'd love to hear about that. But OUR experience was: (1) we found (contrary to popular belief) that OT actually does not require a centralized server, (2) we found it to be harder to implement OT exactly right vs CRDTs, and (3) we found many (though not all) of the problems that CRDTs have, are also problems in practice for OT—although in fairness to OT, we think the problems CRDTs have in general are vastly worse to the end-user experience. If there's interest I'm happy to write a similar article entirely dedicated to OT. But, for (3), as intuition, we found a lot of the problems that both CRDTs and OT have seem to arise from a fundamental impedance mismatch between the in-memory representation of the state of a modern editor, and the representation that is actually synchronized. That is, when you apply an op (CRDT) or a transform (OT), you have to transform the change into a (to use ProseMirror as an example) valid `Transaction` on an `EditorState`. This is not always easy in either case, and to do it right you might have to think very hard about things like "how to preserve position mappings," and other parts of editor state that are crucial to (say) plugins that manage locations of comment marks or presence cursors. With all of that said, OT is definitely much closer to what modern editors need, in my opinion at least. The less-well-known algorithm we ended up recommending here (which I will call "Marjin Collab", after its author) is essentially a very lightweight OT, without the "transformation" step.

samlinnfer 7mo ago
Having a central server is not necessary, but we have one anyway and we use it, especially if you have a permissions system. It lets us use the "Google wave" algorithm which vastly simplifies things. https://svn.apache.org/repos/asf/incubator/wave/whitepapers/operational-transform/operational-transform.html https://svn.apache.org/repos/asf/incubator/wave/whitepapers/... > This is not always easy in either case, and to do it right you might have to think very hard about things like "how to preserve position mappings," and other parts of editor state that are crucial to (say) plugins that manage locations of comment marks or presence cursors. Maintaining text editor state is normal. Yes you do need to convert the OT messages into whatever diff format your editor requires (and back), but that's the standard glue code. The nice thing about OT is that you can just feed the positions of marks into the OT algorithm to get the new positional value. Worst case, you just have the server send the server side position when sending the OT event and the client just displays the server side position.
- DonHopkins 7mo ago
  Josh eloquently explains how Google Wave's DACP (Distributed Application Canceling Protocol) works: https://www.youtube.com/watch?v=4Z4RKRLaSug https://www.youtube.com/watch?v=4Z4RKRLaSug
ianhorn 7mo ago
I always mentally slotted prosemirror-collab/your recommended solution in the OT category. What’s the difference between the “rebase” step and the “transformation” step you’re saying it doesn’t need?
- antics 7mo ago
  Great question. Matt has a comment about this here, and he has an actual PhD on the subject! So rather than doing a worse job explaining I will leave it to him to explain: https://news.ycombinator.com/user?id=mweidner https://news.ycombinator.com/user?id=mweidner
gritzko 7mo ago

MrJohz 7mo ago

Are there any major libraries for OT? I've been looking into this recently for a project at work, and OT would be completely sufficient for our use case, and does look simpler overall, but from what I could tell, we'd need to write a lot of stuff ourselves. The only vaguely active-looking project in JS at least seems to be DocNode (https://www.docukit.dev/docnode https://www.docukit.dev/docnode), and that looks very cool but also very early days.

samlinnfer 7mo ago
https://github.com/josephg/sharejs https://github.com/josephg/sharejs https://github.com/FirebaseExtended/firepad https://github.com/FirebaseExtended/firepad https://github.com/Operational-Transformation/ot.js https://github.com/Operational-Transformation/ot.js https://github.com/ottypes/docs https://github.com/ottypes/docs Really nice demo: https://operational-transformation.github.io https://operational-transformation.github.io
antics 7mo ago
Author here. I think it depends what you're doing! OT is a true distributed systems algorithm and to my knowledge there are no projects that implement true, distributed OT with strong support for modern rich text editor SDKs like ProseMirror. ShareJS, for example, is abandoned, and predates most modern editors. If you are using a centralized server and ProseMirror, there are several OT and pseudo-OT implementations. Most popularly, there is prosemirror-collab[4], which is basically "OT without the stuff you don't need with an authoritative source for documents." Practically speaking that means "OT without T", but because it does not transform the ops to be order-independent, it has an extra step on conflict where the user has to rebase changes and re-submit. This is can cause minor edit starvation of less-connected clients. prosemirror-collab-commit[5] fixes this by performing the rebasing on the server... so it's still "OT without the T", but also with an authoritative conflict resolution pseudo-T at the end. I personally recommend prosemirror-collab-commit, it's what we use, and it's extremely fast and predictable. If you just want something pedogocically helpful, the blessed upstream collaborative editing solution for CodeMirror is OT. See author's blog post[1], the @codemirror/collab package[2], and the live demo[3]. In general this implementation is quite good and worth reading if you are interested in this kind of thing. ShareJS and OTTypes are both very readable and very good, although we found them very challenging to adopt in a real-world ProseMirror-based editor. [1]: https://marijnhaverbeke.nl/blog/collaborative-editing-cm.html https://marijnhaverbeke.nl/blog/collaborative-editing-cm.htm... [2]: https://codemirror.net/docs/ref/#collab https://codemirror.net/docs/ref/#collab [3]: https://codemirror.net/examples/collab/ https://codemirror.net/examples/collab/ [4]: https://github.com/ProseMirror/prosemirror-collab https://github.com/ProseMirror/prosemirror-collab [5]: https://github.com/stepwisehq/prosemirror-collab-commit https://github.com/stepwisehq/prosemirror-collab-commit

r0b05 7mo ago

What is OT?

hsx 7mo ago
Operational Transformation: https://en.wikipedia.org/wiki/Operational_transformation https://en.wikipedia.org/wiki/Operational_transformation
deleted 7mo ago
[deleted]

fritzo 7mo ago

Agreed. In my limited experience, conflict resolution rules are very domain specific, whereas CTDTs encourage a lazy attitude that "if it's associative and commutative it must be correct".

leksak 7mo ago

What does OT stand for so I can learn more?

cantaloupe 7mo ago
Operational Transformation

josephg 7mo ago

> CRDTs are a meme and are not for serious applications. You don't think Figma is a serious application? By all means, use OT. I worked on OT software for many years - and my work on OT types, ShareJS and ShareDB is still in production all over the place. But I don't think there's anything you can do with OT that you can't do just as well with CRDTs. The only real benefit of OT is that its simpler to reason about. Maybe that's enough.

GermanJablo 7mo ago
> You don't think Figma is a serious application? I don't know where this popular belief came from. The Figma blog literally says "Figma isn't using true CRDTs"[1]. > The only real benefit of OT is that its simpler to reason about. That's incorrect. When you free yourself from the P2P restriction that CRDTs are subject to, there's a huge amount of metadata you can get rid of, just to mention one benefit. [1] https://www.figma.com/blog/how-figmas-multiplayer-technology-works/ https://www.figma.com/blog/how-figmas-multiplayer-technology...
- nickthesick 7mo ago
  > I don't know where this popular belief came from. It may be worth reading the whole paragraph of the blog you referenced... > Figma isn't using true CRDTs though. CRDTs are designed for decentralized systems where there is no single central authority to decide what the final state should be. There is some unavoidable performance and memory overhead with doing this. Since Figma is centralized (our server is the central authority), we can simplify our system by removing this extra overhead and benefit from a faster and leaner implementation. > It’s also worth noting that Figma's data structure isn't a single CRDT. Instead it's inspired by multiple separate CRDTs and uses them in combination to create the final data structure that represents a Figma document (described below). So, it's multiple CRDTs, not just one. And they've made some optimizations on top, but that doesn't make it not a CRDT?
  - GermanJablo 7mo ago
    Nowhere does it say it's multiple CRDTs. It says "isn't a single CRDT" and that "it's inspired by multiple separate CRDTs." A bit confusing, I agree. By the way, I work at Figma.

deleted 7mo ago

[deleted]