7 ms·

Having been a dev on EtherPad, Google Wave, Coda, and other real-time collaborative apps with OT, undo, and so on... I think it's correct that, ideally, there

Having been a dev on EtherPad, Google Wave, Coda, and other real-time collaborative apps with OT, undo, and so on... I think it's correct that, ideally, there would be a framework that handles real-time collaboration, undo/redo, and offline support for you, and then you build your app with these problems already solved. I will probably create such a framework eventually. I don't see it as a database engineering problem, it's more like a framework or application architecture, which every app like Google Docs or Figma has its own version of. Writing such a framework is not too much harder than writing such an app, it just requires a little more abstraction and some documentation. If you've never written an undo manager, sync engine, etc., and you aren't writing a complex app, it's hard to arrive at the right design by thinking about pure data sync. Also, storing and querying data are solved problems; it's more a question of coming up with a generic data model for an application and defining its semantics.

kwerk 5y ago
Where would one go to follow along if you start building this framework? You’ve got one pending GH star from me :)
- BiteCode_dev 5y ago
  +1. I would even pay to watch you code this.
  - gurjeet 5y ago
    Didn’t know there was a market for coding session watchers. Let me know if someone’s interested in watch me slow-code my current long running side project: Focusly.
    - TeMPOraL 5y ago
      > Didn’t know there was a market for coding session watchers. I think this is an actual niche on Twitch.
- sambroner 5y ago
  You could start by watching the Fluid Framework github.com/microsoft/fluidframework (I'm a dev on the team). We solve a lot of these problems. This whole article speaks to the longer term vision. Even the article's Title is effectively our internal pitch: a web first database... that enables very low latency collab
  - stopachka 5y ago
    Brought a smile to read this comment. Will look deeper on fluid framework — rooting for ya’ll!
unknown_error 5y ago
Firebase?
pumanoir 5y ago
There are a few of those frameworks. The first one that comes to mind is YJS here is a getting started[1]. 1. https://www.tag1consulting.com/blog/deep-dive-real-time-collaborative-editing-solutions-tagteamtalk-001-0 https://www.tag1consulting.com/blog/deep-dive-real-time-coll...
- sambroner 5y ago
  YJS (and Kevin Jahns!) is fantastic, although I'm not sure they're interested in addressing all of these problems. In so far YJS has been very focused on replicating state from one place to another. And the parent comment may be making light of some of the big issues in the space. You need to be generic, but also performant and simple compared to the competition. Real time text editing is not trivial although there's solid prior art now. Even undo management is a whole problem space (what should a cross user undo look like if there have been dependent changes).
  - kevinjahns 5y ago
    Yjs is exactly that. It is a simple abstraction for building any kind of collaborative application. It has ready to use solutions for most problems related to this problem space. The selective UndoManager, for example, is generic&configurable and can be reused for all kinds of stuff. It supports many different editors. It supports many different (scalable) backends. It is much more than state synchronization. If there is anything missing, then let's work on it. Yjs is extensible and allows for custom features that others can reuse.
- kybernetikos 5y ago
  Do you know if any of these frameworks support different users being permissioned to see different documents or different views of collections.
  - sambroner 5y ago
    Fluid Framework (a project I contribute to/work on through Microsoft) supports this via token in our default service implementation, but we don't have integration with an ACL DB.
alfonsodev 5y ago
something like Microsoft's Fluid Framework ?
YousefED 5y ago
I’m exploring the ideas (an easy to use framework to build local-first [1] apps) in my library Reactive-CRDT (https://github.com/yousefed/reactive-crdt https://github.com/yousefed/reactive-crdt). Feedback welcome! All credit for the underlying tech to YJS, which has been amazing as mentioned by others in this thread. [1]: https://www.inkandswitch.com/local-first.html https://www.inkandswitch.com/local-first.html
stopachka 5y ago
Very excited to hear this!
mathgladiator 5y ago
The common abstraction is a shared log. I'm boiling down my current side project ( http://www.adama-lang.org/ http://www.adama-lang.org/ ) into reusable components. I'm also looking at WebAssembly as the way of doing work ( http://www.adama-lang.org/blog/micro-monoliths http://www.adama-lang.org/blog/micro-monoliths ) versus generic operations. Something interesting to consider is how important is offline use these days? If we get to an online-99.9% world, then the solution feels ... simple.
- ithkuil 5y ago
  Not long ago for work I was forced to use office365. It has collaborative editing feature that at the surface looks equivalent to Google docs, but in practice it's not: it wasn't reliable, I lost some edits a few times and even when it worked it was slow and sluggish (I knew that because I could see another person's shared screen in zoom), and it actually co-editing a file definitely felt a different experience. The relevant bit your comment is: while most people indeed testified that the experience was not as good as Google docs, it wasn't that terrible to most of the people. Turns out that it really depends how well you're connected. Being connected 99.9% sucks more if that 0.1% is a crappy connection that drops packets occasionally as opposed to just saying "I'm online most of the time, I don't care if I need to edit the doc that rare 0.1% of the time when my connection is broken for good)
  - cyanydeez 5y ago
    real time collab is a figment of coders. real business processes are async in nature. what people really want is a easy way to open someone elses work, review it or add to it, and move on without all the emailing, chatting, while important, tend to be wlso a aync need thats note mvp to the process
    - ithkuil 5y ago
      Yeah whatever, I'm not a product guy and I do not pretend to have any clue about what people really want. My point of my anecdote was that "connected" is not a boolean property.
hanspagel 5y ago
You might like what I’m working on. A Node.js backend powered by Y.js. https://www.hocuspocus.dev https://www.hocuspocus.dev Send me an email to humans@tiptap.dev if you want to take a peek.
- chirau 5y ago
  I definitely want to check it out. Also, I love the simplicity of your site. Did you use a template or design it from scratch? I have a project that I need a page for and would want something similar to yours. Happy to pay for the work too.
  - Santosh83 5y ago
    Appears to be compiled using Gridsome, from looking briefly at source (CTRL+U).
austincheney 5y ago
I have been thinking about these concepts in a person app. The primary point of collaboration in this personal app is the file system in real time. The framework I settled on for this problem is a security model. In this app there are two types of agents: personal devices and users. Collaboration between devices is unrestricted because they are only physically separated segments of the same single user. Collaboration between users requires a share which is a segment of availability. By default a share is read only. Device identity is not exposed to other users so a separate user will have no idea they are collaborating across different computers if they are owned by the same user. I have not written undo for the file system yet though. A requested feature is file/directory synchronization and I am considering adding desktop/camera video sharing.

kwerk 5y ago

Where would one go to follow along if you start building this framework? You’ve got one pending GH star from me :)

BiteCode_dev 5y ago
+1. I would even pay to watch you code this.
- gurjeet 5y ago
  Didn’t know there was a market for coding session watchers. Let me know if someone’s interested in watch me slow-code my current long running side project: Focusly.
  - TeMPOraL 5y ago
    > Didn’t know there was a market for coding session watchers. I think this is an actual niche on Twitch.
sambroner 5y ago
You could start by watching the Fluid Framework github.com/microsoft/fluidframework (I'm a dev on the team). We solve a lot of these problems. This whole article speaks to the longer term vision. Even the article's Title is effectively our internal pitch: a web first database... that enables very low latency collab
- stopachka 5y ago
  Brought a smile to read this comment. Will look deeper on fluid framework — rooting for ya’ll!

unknown_error 5y ago

Firebase?

pumanoir 5y ago

There are a few of those frameworks. The first one that comes to mind is YJS here is a getting started[1]. 1. https://www.tag1consulting.com/blog/deep-dive-real-time-collaborative-editing-solutions-tagteamtalk-001-0 https://www.tag1consulting.com/blog/deep-dive-real-time-coll...

sambroner 5y ago
YJS (and Kevin Jahns!) is fantastic, although I'm not sure they're interested in addressing all of these problems. In so far YJS has been very focused on replicating state from one place to another. And the parent comment may be making light of some of the big issues in the space. You need to be generic, but also performant and simple compared to the competition. Real time text editing is not trivial although there's solid prior art now. Even undo management is a whole problem space (what should a cross user undo look like if there have been dependent changes).
- kevinjahns 5y ago
  Yjs is exactly that. It is a simple abstraction for building any kind of collaborative application. It has ready to use solutions for most problems related to this problem space. The selective UndoManager, for example, is generic&configurable and can be reused for all kinds of stuff. It supports many different editors. It supports many different (scalable) backends. It is much more than state synchronization. If there is anything missing, then let's work on it. Yjs is extensible and allows for custom features that others can reuse.
kybernetikos 5y ago
Do you know if any of these frameworks support different users being permissioned to see different documents or different views of collections.
- sambroner 5y ago
  Fluid Framework (a project I contribute to/work on through Microsoft) supports this via token in our default service implementation, but we don't have integration with an ACL DB.

alfonsodev 5y ago

something like Microsoft's Fluid Framework ?

YousefED 5y ago

I’m exploring the ideas (an easy to use framework to build local-first [1] apps) in my library Reactive-CRDT (https://github.com/yousefed/reactive-crdt https://github.com/yousefed/reactive-crdt). Feedback welcome! All credit for the underlying tech to YJS, which has been amazing as mentioned by others in this thread. [1]: https://www.inkandswitch.com/local-first.html https://www.inkandswitch.com/local-first.html

stopachka 5y ago

Very excited to hear this!

mathgladiator 5y ago

The common abstraction is a shared log. I'm boiling down my current side project ( http://www.adama-lang.org/ http://www.adama-lang.org/ ) into reusable components. I'm also looking at WebAssembly as the way of doing work ( http://www.adama-lang.org/blog/micro-monoliths http://www.adama-lang.org/blog/micro-monoliths ) versus generic operations. Something interesting to consider is how important is offline use these days? If we get to an online-99.9% world, then the solution feels ... simple.

ithkuil 5y ago
Not long ago for work I was forced to use office365. It has collaborative editing feature that at the surface looks equivalent to Google docs, but in practice it's not: it wasn't reliable, I lost some edits a few times and even when it worked it was slow and sluggish (I knew that because I could see another person's shared screen in zoom), and it actually co-editing a file definitely felt a different experience. The relevant bit your comment is: while most people indeed testified that the experience was not as good as Google docs, it wasn't that terrible to most of the people. Turns out that it really depends how well you're connected. Being connected 99.9% sucks more if that 0.1% is a crappy connection that drops packets occasionally as opposed to just saying "I'm online most of the time, I don't care if I need to edit the doc that rare 0.1% of the time when my connection is broken for good)
- cyanydeez 5y ago
  real time collab is a figment of coders. real business processes are async in nature. what people really want is a easy way to open someone elses work, review it or add to it, and move on without all the emailing, chatting, while important, tend to be wlso a aync need thats note mvp to the process
  - ithkuil 5y ago
    Yeah whatever, I'm not a product guy and I do not pretend to have any clue about what people really want. My point of my anecdote was that "connected" is not a boolean property.

hanspagel 5y ago

You might like what I’m working on. A Node.js backend powered by Y.js. https://www.hocuspocus.dev https://www.hocuspocus.dev Send me an email to humans@tiptap.dev if you want to take a peek.

chirau 5y ago
I definitely want to check it out. Also, I love the simplicity of your site. Did you use a template or design it from scratch? I have a project that I need a page for and would want something similar to yours. Happy to pay for the work too.
- Santosh83 5y ago
  Appears to be compiled using Gridsome, from looking briefly at source (CTRL+U).

austincheney 5y ago

I have been thinking about these concepts in a person app. The primary point of collaboration in this personal app is the file system in real time. The framework I settled on for this problem is a security model. In this app there are two types of agents: personal devices and users. Collaboration between devices is unrestricted because they are only physically separated segments of the same single user. Collaboration between users requires a share which is a segment of availability. By default a share is read only. Device identity is not exposed to other users so a separate user will have no idea they are collaborating across different computers if they are owned by the same user. I have not written undo for the file system yet though. A requested feature is file/directory synchronization and I am considering adding desktop/camera video sharing.