6 ms·

Just commenting to say I (author) am here this is a good place for feedback or pointing out relevant references.

by marijn 11y ago

haches 11y ago
Thanks for writing this post. I was already curious how you implemented the collaboration feature. Very happy to get some details. Fantastic work.
anilgulecha 11y ago
Hi Marijn, I'm curious if you've taken note of substance.io, and what you think of their composer and viewer, which similarly keeps the structure and views separate. I'm not related to substance.io, just someone curious about editors on the web, and so have kept track of various options. Prosemirror looks very promising.
- marijn 11y ago
  ProseMirror is also definitely intended to keep out any considerations of the layout/look of the document. I hadn't seen substance.io yet.
thomsbg 11y ago
Hi! Awesome post. I've been working on a similar project based on the quill editor [1], which is in turn based on the rich-text OT library [2]. Initially, I thought to wire this all up using the sharejs project [3], but never could quite grok that codebase. Instead, I've been working on a similar system to one you describe, with changes applied in order based on sequential version number, and concurrent updates forced to "rebase" against earlier changes (not yet open-sourced). A question: > Because applying changes in a different order might create a different document, rebasing isn't quite as easy as transforming all of our own changes through all of the remotely made changes. Can you explain more about how you arrived this conclusion? From my understanding, a correct transform function should allow exactly this, according to transformation property 1 [4]. Perhaps your algorithm doesn't exactly satisfy this property; what characteristics does it have instead? [1] http://quilljs.com/ http://quilljs.com/ [2] https://github.com/ottypes/rich-text https://github.com/ottypes/rich-text [3] http://sharejs.org/ http://sharejs.org/ [4] https://en.wikipedia.org/wiki/Operational_transformation#Convergence_properties https://en.wikipedia.org/wiki/Operational_transformation#Con...
- marijn 11y ago
  > From my understanding, a correct transform function should allow exactly this A correct OT transform, yes. But I'm not using OT's invariants, so this is not something my transforms do. For example, in my system, if you have "insert X at pos 5" and "insert Y at pos 5", the document will contain "XY" or "YX", depending on which arrived first.
  - thomsbg 11y ago
    Interesting! I'm excited to see how this all turns out, and what we can learn from each other.
- josephg 11y ago
  (ShareJS author here) Sorry ShareJS is such a mess right now. I did a bunch of work to make it into a kind of OT-backed database to power derbyjs. Its very much straddling two worlds at the moment and I think its doing a bad job of both. It'll get cleaned up eventually.
kylemathews 11y ago
Hi, thanks for this great project. Curious if you've looked at CRDTs? (https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...)
- marijn 11y ago
  Yes, I coincidentally saw Shapiro's talk at the Curry On conference last month, and was quite inspired by that way of looking at the problem.
aburgm 11y ago
Thanks for the article, interesting read! How well does this approach scale to high-concurrency situations? The more active participants there are, the more often you'd be rejecting client changes and ask them to rebase, causing extra round-trips. It sounds like when a critical number of participants is reached, you'll create change requests more quickly than you can handle them. If each participant has a latency of 100ms to your server, and say you have 10 participants making 2 changes per second each, will you experience some sort of congestion? On a similar note, what happens if you have two active clients, one with say 100ms latency and another with 500ms? If the 100ms client will be typing continously with 4 characters per second, the changes of the 500ms client won't get through until the 100ms one stops typing, right? I realize these might not be much of a real issues in practise, but did you consider such cases? In particular, is there something in your machinery that prevents the server from doing the rebase, as opposed to sending it back to the client to do it? As for references, I went all the way and implemented distributed OT in Gobby (https://github.com/gobby/gobby https://github.com/gobby/gobby). It's somewhat doable for plain text editing, but going beyond that inflates the complexity very quickly. It gives you some nice properties like authorship tracking even across undo operations. However, the complexity goes with at least n^2 with the number of participants, so there is also an upper limit in the number of active participants that's reached rather sooner than later (even though I never actually found out where it is in practise).
- marijn 11y ago
  I did consider such cases. My current theory is that it is unlikely that clients will keep up a torrent of updates so constant that there'll be a lag of more than, say, a second before any given change gets through. But if this would turn out to be an issue, you could have the server implement some kind of fairness algorithm by slightly delaying clients that recently got their changes through and prioritizing those that made a few unsuccessful tries.
jarpineh 11y ago
This looks extremely interesting. I have looked at several JS-based text editors, and so far none have matched my requirements. ProseMirror sounds like something I have thought about as a possible approach. Well, not so much the collaboration part. I've had hard time just trying to reason about DOM vs document tree, contenteditable vs trying to match writer's intentions, and so on. This pattern matching of user's intentions and using DOM diffing to find out what actually happened feels right to me, especially using persistent data structures. Handling parts of the document as a tree and others as a list seems like decision that could probably make development much easier as you'd know, I imagine. I have been wondering about how to do structured inline element annotations, and lists inside block level elements certainly looks it could work. Also, this relates to what LightTable folk's talked about in IDE as a value [1]. I'm definitely going to back this. That said, I'd like to know more about what is actually going to (hopefully) be open sourced. Is this a stand alone app with plugins, a collection of components, or a library to build your own editor? Is it written in purely JavaScript version 5, 6, or something else? What about the persistent data structure implementation? How are user intent patterns modeled, ie. what would it entail to build your own? Have you thought about intermixing document tree with editors for other types of content, like tables and media? [1] http://www.chris-granger.com/2013/01/24/the-ide-as-data/ http://www.chris-granger.com/2013/01/24/the-ide-as-data/
- marijn 11y ago
  This is a library providing the client-side editor component, with a bunch of modules (UI, collaboration), and a well-documented API through which to connect it to surrounding code. It is written in ES6. The persistent data structures are not very involved, just objects that never get mutated after their initial construction.
erisds 11y ago
First of all, this looks truly awesome! I've spent a lot of time recently looking at the available open source editors, and with the exception of specifically 'code editors' there isn't much out there that supports markdown. I have been reading about the recently announced content-kit: http://madhatted.com/2015/7/31/announcing-content-kit-and-mobiledoc http://madhatted.com/2015/7/31/announcing-content-kit-and-mo... and am wondering if you have considered the 'card' style concept for ProseMirror? I see ProseMirror has an interface for adding images, and says that it will support different document models in future and I'm wondering how extensible that will be. What sort of APIs are going to be available, will it be possible to create custom 'blocks' or 'cards' of data - e.g. defining a block for adding a table / spreadsheet - similar to what you see in things like Quip or readme.io?
- marijn 11y ago
  Basically, yes, the document object is designed to be extended by user code, so you could add your own kinds of nodes (even allowing them to contain existing node types, as in a table), but for some nodes (like tables) the impact will probably be big enough to require some specific UI-related code in the core.
- marijn 11y ago
  Content-Kit looks extremely interesting, by the way. Funny that the announcement came just a few days before mine.

Just commenting to say I (author) am here this is a good place for feedback or pointing out relevant references.

by marijn 11y ago

Just commenting to say I (author) am here this is a good place for feedback or pointing out relevant references.

Thanks for writing this post. I was already curious how you implemented the collaboration feature. Very happy to get some details. Fantastic work.

anilgulecha 11y ago

Hi Marijn, I'm curious if you've taken note of substance.io, and what you think of their composer and viewer, which similarly keeps the structure and views separate. I'm not related to substance.io, just someone curious about editors on the web, and so have kept track of various options. Prosemirror looks very promising.

marijn 11y ago
ProseMirror is also definitely intended to keep out any considerations of the layout/look of the document. I hadn't seen substance.io yet.

thomsbg 11y ago

Hi! Awesome post. I've been working on a similar project based on the quill editor [1], which is in turn based on the rich-text OT library [2]. Initially, I thought to wire this all up using the sharejs project [3], but never could quite grok that codebase. Instead, I've been working on a similar system to one you describe, with changes applied in order based on sequential version number, and concurrent updates forced to "rebase" against earlier changes (not yet open-sourced). A question: > Because applying changes in a different order might create a different document, rebasing isn't quite as easy as transforming all of our own changes through all of the remotely made changes. Can you explain more about how you arrived this conclusion? From my understanding, a correct transform function should allow exactly this, according to transformation property 1 [4]. Perhaps your algorithm doesn't exactly satisfy this property; what characteristics does it have instead? [1] http://quilljs.com/ http://quilljs.com/ [2] https://github.com/ottypes/rich-text https://github.com/ottypes/rich-text [3] http://sharejs.org/ http://sharejs.org/ [4] https://en.wikipedia.org/wiki/Operational_transformation#Convergence_properties https://en.wikipedia.org/wiki/Operational_transformation#Con...

marijn 11y ago
> From my understanding, a correct transform function should allow exactly this A correct OT transform, yes. But I'm not using OT's invariants, so this is not something my transforms do. For example, in my system, if you have "insert X at pos 5" and "insert Y at pos 5", the document will contain "XY" or "YX", depending on which arrived first.
- thomsbg 11y ago
  Interesting! I'm excited to see how this all turns out, and what we can learn from each other.
josephg 11y ago
(ShareJS author here) Sorry ShareJS is such a mess right now. I did a bunch of work to make it into a kind of OT-backed database to power derbyjs. Its very much straddling two worlds at the moment and I think its doing a bad job of both. It'll get cleaned up eventually.

kylemathews 11y ago

Hi, thanks for this great project. Curious if you've looked at CRDTs? (https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...)

marijn 11y ago
Yes, I coincidentally saw Shapiro's talk at the Curry On conference last month, and was quite inspired by that way of looking at the problem.

aburgm 11y ago

Thanks for the article, interesting read! How well does this approach scale to high-concurrency situations? The more active participants there are, the more often you'd be rejecting client changes and ask them to rebase, causing extra round-trips. It sounds like when a critical number of participants is reached, you'll create change requests more quickly than you can handle them. If each participant has a latency of 100ms to your server, and say you have 10 participants making 2 changes per second each, will you experience some sort of congestion? On a similar note, what happens if you have two active clients, one with say 100ms latency and another with 500ms? If the 100ms client will be typing continously with 4 characters per second, the changes of the 500ms client won't get through until the 100ms one stops typing, right? I realize these might not be much of a real issues in practise, but did you consider such cases? In particular, is there something in your machinery that prevents the server from doing the rebase, as opposed to sending it back to the client to do it? As for references, I went all the way and implemented distributed OT in Gobby (https://github.com/gobby/gobby https://github.com/gobby/gobby). It's somewhat doable for plain text editing, but going beyond that inflates the complexity very quickly. It gives you some nice properties like authorship tracking even across undo operations. However, the complexity goes with at least n^2 with the number of participants, so there is also an upper limit in the number of active participants that's reached rather sooner than later (even though I never actually found out where it is in practise).

marijn 11y ago
I did consider such cases. My current theory is that it is unlikely that clients will keep up a torrent of updates so constant that there'll be a lag of more than, say, a second before any given change gets through. But if this would turn out to be an issue, you could have the server implement some kind of fairness algorithm by slightly delaying clients that recently got their changes through and prioritizing those that made a few unsuccessful tries.

jarpineh 11y ago

This looks extremely interesting. I have looked at several JS-based text editors, and so far none have matched my requirements. ProseMirror sounds like something I have thought about as a possible approach. Well, not so much the collaboration part. I've had hard time just trying to reason about DOM vs document tree, contenteditable vs trying to match writer's intentions, and so on. This pattern matching of user's intentions and using DOM diffing to find out what actually happened feels right to me, especially using persistent data structures. Handling parts of the document as a tree and others as a list seems like decision that could probably make development much easier as you'd know, I imagine. I have been wondering about how to do structured inline element annotations, and lists inside block level elements certainly looks it could work. Also, this relates to what LightTable folk's talked about in IDE as a value [1]. I'm definitely going to back this. That said, I'd like to know more about what is actually going to (hopefully) be open sourced. Is this a stand alone app with plugins, a collection of components, or a library to build your own editor? Is it written in purely JavaScript version 5, 6, or something else? What about the persistent data structure implementation? How are user intent patterns modeled, ie. what would it entail to build your own? Have you thought about intermixing document tree with editors for other types of content, like tables and media? [1] http://www.chris-granger.com/2013/01/24/the-ide-as-data/ http://www.chris-granger.com/2013/01/24/the-ide-as-data/

marijn 11y ago
This is a library providing the client-side editor component, with a bunch of modules (UI, collaboration), and a well-documented API through which to connect it to surrounding code. It is written in ES6. The persistent data structures are not very involved, just objects that never get mutated after their initial construction.

erisds 11y ago

First of all, this looks truly awesome! I've spent a lot of time recently looking at the available open source editors, and with the exception of specifically 'code editors' there isn't much out there that supports markdown. I have been reading about the recently announced content-kit: http://madhatted.com/2015/7/31/announcing-content-kit-and-mobiledoc http://madhatted.com/2015/7/31/announcing-content-kit-and-mo... and am wondering if you have considered the 'card' style concept for ProseMirror? I see ProseMirror has an interface for adding images, and says that it will support different document models in future and I'm wondering how extensible that will be. What sort of APIs are going to be available, will it be possible to create custom 'blocks' or 'cards' of data - e.g. defining a block for adding a table / spreadsheet - similar to what you see in things like Quip or readme.io?

marijn 11y ago
Basically, yes, the document object is designed to be extended by user code, so you could add your own kinds of nodes (even allowing them to contain existing node types, as in a table), but for some nodes (like tables) the impact will probably be big enough to require some specific UI-related code in the core.
marijn 11y ago
Content-Kit looks extremely interesting, by the way. Funny that the announcement came just a few days before mine.