7 ms·

Loro's rich text CRDT

207 points by czx111331 3y ago

rony989 3y ago
[dead]
rubymamis 3y ago
Slightly off-topic - I don't think real-time collaboration is suitable for text-based formats. I believe collaboration similar to working with git is superior: 1. Fork the text 2. Submit proposal 3. Review 4. Merge/Cancel EDIT: To slightly expand on this - there are many reasons for this intuition - the main, IMO, is that people like to work on text privately before showing it to people. Also, the mental fear of your text interrupted by someone else. There might be even more reasons.
- madeofpalk 3y ago
  This might be true for writing an essay. But plenty of collaborative text-editing happens on a much smaller/informal/ad hoc scale. My personal experience of using Google Docs/Sheets strongly disagrees with you.
  - layer8 3y ago
    Asynchronous editing on a centralized platform only infrequently causes actual conflicts in practice, like multiple users editing the same sentence at the same time. Either the editing is actually sequential in time, or independent parts of the document are being edited. With asynchronous decentralized/offline editing, conflicts become more likely.
- jitl 3y ago
  What's great about CRDTs is that they're really good at both real-time and asynchronous collaboration, unlike the operational transform system used by Google Docs. Asynchronous collaboration is Peritext's major motivation [1]: > We interviewed eight people who regularly collaborate professionally on documents such as news articles, and several told us that they found real-time collaboration a stressful experience: they felt performative, self-conscious of others witnessing their messy work-in-progress, or irritated when a collaborator acted on suggestions before the editing pass was complete. When doing creative work, they preferred to have space to ideate and experiment in private, sharing their progress only when they are ready to do so. > With asynchronous collaboration, this is possible: a user may work in isolation on their own copy of a document for a while, without seeing other users’ real-time updates; sometime later, when they are ready to share their work, they can choose to merge it with their collaborators’ edits. Several such copies may exist side-by-side, and some might never be merged (e.g. if the user changed their mind about a set of edits). [1]: https://www.inkandswitch.com/peritext/ https://www.inkandswitch.com/peritext/
  - rubymamis 3y ago
    So they mention my exact concerns and addressed them, very cool. But their solution in practice doesn't look very different from what we can already achieve with git (apart from seeing your collaborator changes in real-time, which I'm not sure how substantial it is), or am I missing something? At the end of the day, how will this look to the end user? > a user may work in isolation on their own copy of a document for a while, without seeing other users’ real-time updates; sometime later, when they are ready to share their work, they can choose to merge it with their collaborators’ edits. Several such copies may exist side-by-side, and some might never be merged (e.g. if the user changed their mind about a set of edits). Again, this sounds almost exactly like what we already achieve using git, so why do we need CRDTs for that?
    - spencerflem 3y ago
      The merging is automatic, for one
      - josephg 3y ago
        Yeah; and the merging is correct in all cases. You don't get spurious conflicts like you can with git. Also the same system can work both in realtime and offline scenarios. And CRDTs can handle a lot more than just plain text editing. That said, one thing git does that I like is that sometimes I want conflict markers to be added to my document when concurrent edits happen on the same line. CRDTs (and REGs like this) store strictly more information than git does, so theoretically it should be possible to make a CRDT which adds conflict markers too like git does. But as far as I know, nobody has built that yet! I really hope someone does, because it would be really neat. I really want a git style version control system built on top of CRDTs with optional merge conflicts.
        bmacho 3y ago
        > Yeah; and the merging is correct in all cases. No, it's obviously not. I'd say most of the time the merging is incorrect, that is, it is in contrast to both of the parties intention. Unlike git, which actually notifies them, Loro just silently modifies both changes into something nonsense/wrong.
- mikebelanger 3y ago
  Git is still great for developers, who are comfortable with the command-line and are willing to learn all of Git's concepts. But let's be honest, Git isn't the most friendly tool for most computer users, many of whom have never touched any command-line interface, let alone Git. And yes, there's git-GUIs, but I find those are even more complex.
- ghizmandier 3y ago
  Hey, sounds like you'd love what we're building @ Ellipsus — https://ellipsus.com https://ellipsus.com We started off your same exact assumptions and built a text editor that combines the best of both worlds: you can collaborate in real-time on the same piece; or branch off of it, work on your own and then have it reviewed and merged back into the main branch. That and other features that should make writing together a pleasure. Reach out if you want to give it a spin, we'd appreciate your feedback!
  - rubymamis 3y ago
    Looks awesome! Love the website. I signed up for the wait list.
NeutralForest 3y ago
Looks dope, could be nice for collaborative writing like a multi-author blog post or for docs.
jitl 3y ago
I'm curious about this line describing REG: > The REG algorithm excels with its fast local update speeds and eliminate concerns about tombstone collection in CRDTs. For instance, if an operation has been synchronized across all endpoints, no new operations will occur concurrently with it, allowing it to be safely removed from the history. If you remove these ops from history, does that remove the ability to time travel (per the home page "An antidote to regret, enabling historical edits traversal") or merge branches? How can we be sure an operation is synchronized? If dropping these ops is necessary for speed/storage optimization but disables time-travel, is it possible to put the removed historical/tombstone ops into a "cold storage" that's optional and only loaded for time-travel use?
- czx111331 3y ago
  > If you remove these ops from history, does that remove the ability to time travel (per the home page "An antidote to regret, enabling historical edits traversal") or merge branches? Yes. But squash can be supported. > How can we be sure an operation is synchronized? In Loro, we not only record the real-world timestamp efficiently, similar to Git, but also capture the DAG information. This approach ensures that if an operation (op) is particularly old, it will have many other ops depending on it. By utilizing both pieces of information, we can determine the operations that are likely synced across all peers. For peers like servers, it's feasible to preserve all operations. However, we can remove some operations in scenarios such as opening the document online for the first time. > If dropping these ops is necessary for speed/storage optimization but disables time-travel, is it possible to put the removed historical/tombstone ops into a "cold storage" that's optional and only loaded for time-travel use? Yes. This is not supported at the moment, but we hope to implement it before version 1.0.
- josephg 3y ago
  Hi! I invented replayable event graphs. I'm writing a paper at the moment about it, which hopefully should be out in a month or so. Send me a private email and I can mail you the current draft if you like. > If you remove these ops from history, does that remove the ability to time travel Yes it does. You also need the ops from history to be able to merge changes. You can only merge changes so long as you have the operations going back to the point at which the fork happened. > is it possible to put the removed historical/tombstone ops into a "cold storage" that's optional and only loaded for time-travel use? Absolutely. And this is very practically useful. For example, you could have a web page which loads the current state of a document (just a string. Unlike CRDTs, it needs no additional metadata!). Then if some merge happens while you have the document open, the browser could just fetch the operations from the server back as far as it needs to be able to merge. But in normal operation, none of the historical operations need to be loaded at all. All this said, with text documents the overhead of just keeping the historical operations is pretty tiny anyway. In my testing using diamond types (same algorithm, different library), storing the entire set of historical operations usually increases the file size by less than 50% compared to just storing the final text string. Its much more efficient on disk than git, and more efficient than other CRDTs like automerge and Yjs. So I think most of the time its easier to just keep the history around and not worry about the complexity.
lewisjoe 3y ago
It's great work improvising over Peritext using joseph's latest CRDT work. Much needed literature in "applying CRDTs for richtext" space. But I'm surprised why this one too hasn't focussed a lot on rich-text block elements (like lists, tables & sections) as much as it focussed on text attributes (like bold and italics).
- injuly 3y ago
  I've gone through the Peritext paper several times, and attempted to implement support for lists/tables/sections myself, but found it to be difficult owing to the limited scope of the original paper. I reached out to Martin Kleppman about this, and this is what he told me: "Thanks for your message. I've written up a document on how to extend Patreon with nested block elements such as bullet points. It's not properly published yet, but you can find a draft here: https://martinkl.notion.site/Block-elements-in-rich-text-CRDT-for-Peritext-2-a3b69f886dbc4ad1abe81cea0b3e6623?pvs=4 https://martinkl.notion.site/Block-elements-in-rich-text-CRD... My colleagues are currently in the process of integrating this algorithm into Automerge, and creating bindings to the ProseMirror rich text editor." Personally, I found the document to be very helpful.
  - fnordsensei 3y ago
    I've recently implemented a rich text editor on top of Automerge and Peritext that supports blocks, inline-blocks (or whatever you call them: images, tables, etc.), unaware that this was in the works. The implementation ended up _very_ close to what's described in the linked post.
- mweidner 3y ago
  The demo they show uses the Quill rich-text editor, which handles block elements analogously to text attributes: a block's type is determined by the attributes on its trailing newline. E.g., for a block that is part of an ordered list, the newline's format is `{ list: "ordered" }`. https://quilljs.com/docs/delta/#line-formatting https://quilljs.com/docs/delta/#line-formatting
- hwillis 3y ago
doublerabbit 3y ago
Hmm, not helpful. I'm on iOS, so there is no console. > Application error: a client-side exception has occurred (see the browser console for more information).
erlend_sh 3y ago
Would be nice to see cola included in the benchmarks: https://nomad.foo/blog/cola https://nomad.foo/blog/cola
mikebelanger 3y ago
Looks neat! Would there be a way of intercepting state and making 'snapshots' into a more traditional format, like SQL, or even a JSON file? It sounds like this defaults to the server storing the whole state in their binary format, ditto the client-side portion of it. Nothing wrong with the format, but this is an early project, and nobody wants their data in something that's potentially unstable, or something that might get corrupted.
- CodeGroyper 3y ago
  Supposedly you can always make a deep copy or a backup of anything you have.
- czx111331 3y ago
  We are carefully stabilizing our encoding format and will have a clear storage format documentation introduced in version 1.0. I agree that a more transparent format can provide users with a better sense of control, and we will try to create a human-readable format for exporting CRDT data (the kind that includes operation history). As for the application state, Loro already supports direct export in json format.
  - mikebelanger 3y ago
    Ok, I appreciate you reaching back. And my bad, I didn't see that it already supports direct export of JSON.

207 points by czx111331 3y ago

[dead]

Slightly off-topic - I don't think real-time collaboration is suitable for text-based formats. I believe collaboration similar to working with git is superior: 1. Fork the text 2. Submit proposal 3. Review 4. Merge/Cancel EDIT: To slightly expand on this - there are many reasons for this intuition - the main, IMO, is that people like to work on text privately before showing it to people. Also, the mental fear of your text interrupted by someone else. There might be even more reasons.

madeofpalk 3y ago
This might be true for writing an essay. But plenty of collaborative text-editing happens on a much smaller/informal/ad hoc scale. My personal experience of using Google Docs/Sheets strongly disagrees with you.
- layer8 3y ago
  Asynchronous editing on a centralized platform only infrequently causes actual conflicts in practice, like multiple users editing the same sentence at the same time. Either the editing is actually sequential in time, or independent parts of the document are being edited. With asynchronous decentralized/offline editing, conflicts become more likely.
jitl 3y ago
What's great about CRDTs is that they're really good at both real-time and asynchronous collaboration, unlike the operational transform system used by Google Docs. Asynchronous collaboration is Peritext's major motivation [1]: > We interviewed eight people who regularly collaborate professionally on documents such as news articles, and several told us that they found real-time collaboration a stressful experience: they felt performative, self-conscious of others witnessing their messy work-in-progress, or irritated when a collaborator acted on suggestions before the editing pass was complete. When doing creative work, they preferred to have space to ideate and experiment in private, sharing their progress only when they are ready to do so. > With asynchronous collaboration, this is possible: a user may work in isolation on their own copy of a document for a while, without seeing other users’ real-time updates; sometime later, when they are ready to share their work, they can choose to merge it with their collaborators’ edits. Several such copies may exist side-by-side, and some might never be merged (e.g. if the user changed their mind about a set of edits). [1]: https://www.inkandswitch.com/peritext/ https://www.inkandswitch.com/peritext/
- rubymamis 3y ago
  So they mention my exact concerns and addressed them, very cool. But their solution in practice doesn't look very different from what we can already achieve with git (apart from seeing your collaborator changes in real-time, which I'm not sure how substantial it is), or am I missing something? At the end of the day, how will this look to the end user? > a user may work in isolation on their own copy of a document for a while, without seeing other users’ real-time updates; sometime later, when they are ready to share their work, they can choose to merge it with their collaborators’ edits. Several such copies may exist side-by-side, and some might never be merged (e.g. if the user changed their mind about a set of edits). Again, this sounds almost exactly like what we already achieve using git, so why do we need CRDTs for that?
  - spencerflem 3y ago
    The merging is automatic, for one
    - josephg 3y ago
      Yeah; and the merging is correct in all cases. You don't get spurious conflicts like you can with git. Also the same system can work both in realtime and offline scenarios. And CRDTs can handle a lot more than just plain text editing. That said, one thing git does that I like is that sometimes I want conflict markers to be added to my document when concurrent edits happen on the same line. CRDTs (and REGs like this) store strictly more information than git does, so theoretically it should be possible to make a CRDT which adds conflict markers too like git does. But as far as I know, nobody has built that yet! I really hope someone does, because it would be really neat. I really want a git style version control system built on top of CRDTs with optional merge conflicts.
      - bmacho 3y ago
        > Yeah; and the merging is correct in all cases. No, it's obviously not. I'd say most of the time the merging is incorrect, that is, it is in contrast to both of the parties intention. Unlike git, which actually notifies them, Loro just silently modifies both changes into something nonsense/wrong.
mikebelanger 3y ago
Git is still great for developers, who are comfortable with the command-line and are willing to learn all of Git's concepts. But let's be honest, Git isn't the most friendly tool for most computer users, many of whom have never touched any command-line interface, let alone Git. And yes, there's git-GUIs, but I find those are even more complex.
ghizmandier 3y ago
Hey, sounds like you'd love what we're building @ Ellipsus — https://ellipsus.com https://ellipsus.com We started off your same exact assumptions and built a text editor that combines the best of both worlds: you can collaborate in real-time on the same piece; or branch off of it, work on your own and then have it reviewed and merged back into the main branch. That and other features that should make writing together a pleasure. Reach out if you want to give it a spin, we'd appreciate your feedback!
- rubymamis 3y ago
  Looks awesome! Love the website. I signed up for the wait list.

NeutralForest 3y ago

Looks dope, could be nice for collaborative writing like a multi-author blog post or for docs.

jitl 3y ago

I'm curious about this line describing REG: > The REG algorithm excels with its fast local update speeds and eliminate concerns about tombstone collection in CRDTs. For instance, if an operation has been synchronized across all endpoints, no new operations will occur concurrently with it, allowing it to be safely removed from the history. If you remove these ops from history, does that remove the ability to time travel (per the home page "An antidote to regret, enabling historical edits traversal") or merge branches? How can we be sure an operation is synchronized? If dropping these ops is necessary for speed/storage optimization but disables time-travel, is it possible to put the removed historical/tombstone ops into a "cold storage" that's optional and only loaded for time-travel use?

czx111331 3y ago
> If you remove these ops from history, does that remove the ability to time travel (per the home page "An antidote to regret, enabling historical edits traversal") or merge branches? Yes. But squash can be supported. > How can we be sure an operation is synchronized? In Loro, we not only record the real-world timestamp efficiently, similar to Git, but also capture the DAG information. This approach ensures that if an operation (op) is particularly old, it will have many other ops depending on it. By utilizing both pieces of information, we can determine the operations that are likely synced across all peers. For peers like servers, it's feasible to preserve all operations. However, we can remove some operations in scenarios such as opening the document online for the first time. > If dropping these ops is necessary for speed/storage optimization but disables time-travel, is it possible to put the removed historical/tombstone ops into a "cold storage" that's optional and only loaded for time-travel use? Yes. This is not supported at the moment, but we hope to implement it before version 1.0.
josephg 3y ago
Hi! I invented replayable event graphs. I'm writing a paper at the moment about it, which hopefully should be out in a month or so. Send me a private email and I can mail you the current draft if you like. > If you remove these ops from history, does that remove the ability to time travel Yes it does. You also need the ops from history to be able to merge changes. You can only merge changes so long as you have the operations going back to the point at which the fork happened. > is it possible to put the removed historical/tombstone ops into a "cold storage" that's optional and only loaded for time-travel use? Absolutely. And this is very practically useful. For example, you could have a web page which loads the current state of a document (just a string. Unlike CRDTs, it needs no additional metadata!). Then if some merge happens while you have the document open, the browser could just fetch the operations from the server back as far as it needs to be able to merge. But in normal operation, none of the historical operations need to be loaded at all. All this said, with text documents the overhead of just keeping the historical operations is pretty tiny anyway. In my testing using diamond types (same algorithm, different library), storing the entire set of historical operations usually increases the file size by less than 50% compared to just storing the final text string. Its much more efficient on disk than git, and more efficient than other CRDTs like automerge and Yjs. So I think most of the time its easier to just keep the history around and not worry about the complexity.

lewisjoe 3y ago

It's great work improvising over Peritext using joseph's latest CRDT work. Much needed literature in "applying CRDTs for richtext" space. But I'm surprised why this one too hasn't focussed a lot on rich-text block elements (like lists, tables & sections) as much as it focussed on text attributes (like bold and italics).

injuly 3y ago
I've gone through the Peritext paper several times, and attempted to implement support for lists/tables/sections myself, but found it to be difficult owing to the limited scope of the original paper. I reached out to Martin Kleppman about this, and this is what he told me: "Thanks for your message. I've written up a document on how to extend Patreon with nested block elements such as bullet points. It's not properly published yet, but you can find a draft here: https://martinkl.notion.site/Block-elements-in-rich-text-CRDT-for-Peritext-2-a3b69f886dbc4ad1abe81cea0b3e6623?pvs=4 https://martinkl.notion.site/Block-elements-in-rich-text-CRD... My colleagues are currently in the process of integrating this algorithm into Automerge, and creating bindings to the ProseMirror rich text editor." Personally, I found the document to be very helpful.
- fnordsensei 3y ago
  I've recently implemented a rich text editor on top of Automerge and Peritext that supports blocks, inline-blocks (or whatever you call them: images, tables, etc.), unaware that this was in the works. The implementation ended up _very_ close to what's described in the linked post.
mweidner 3y ago
The demo they show uses the Quill rich-text editor, which handles block elements analogously to text attributes: a block's type is determined by the attributes on its trailing newline. E.g., for a block that is part of an ordered list, the newline's format is `{ list: "ordered" }`. https://quilljs.com/docs/delta/#line-formatting https://quilljs.com/docs/delta/#line-formatting
hwillis 3y ago

doublerabbit 3y ago

Hmm, not helpful. I'm on iOS, so there is no console. > Application error: a client-side exception has occurred (see the browser console for more information).

erlend_sh 3y ago

Would be nice to see cola included in the benchmarks: https://nomad.foo/blog/cola https://nomad.foo/blog/cola

mikebelanger 3y ago

Looks neat! Would there be a way of intercepting state and making 'snapshots' into a more traditional format, like SQL, or even a JSON file? It sounds like this defaults to the server storing the whole state in their binary format, ditto the client-side portion of it. Nothing wrong with the format, but this is an early project, and nobody wants their data in something that's potentially unstable, or something that might get corrupted.

CodeGroyper 3y ago
Supposedly you can always make a deep copy or a backup of anything you have.
czx111331 3y ago
We are carefully stabilizing our encoding format and will have a clear storage format documentation introduced in version 1.0. I agree that a more transparent format can provide users with a better sense of control, and we will try to create a human-readable format for exporting CRDT data (the kind that includes operation history). As for the application state, Loro already supports direct export in json format.
- mikebelanger 3y ago
  Ok, I appreciate you reaching back. And my bad, I didn't see that it already supports direct export of JSON.