8 ms·

Flux over the Wire using Websockets and Immutable JS

81 points by elierotenberg 12y ago

stucorbishley 12y ago
The single source of truth diagram is so great! :) Was getting tired of turning my head sideways..
sehr 12y ago
cache: http://webcache.googleusercontent.com/search?q=cache:http://blog.rotenberg.io/flux-over-the-wire-3/ http://webcache.googleusercontent.com/search?q=cache:http://...
elierotenberg 12y ago
Wow, a crashed site presenting a web architecture surely isn't classy! Sorry about that, I'm working on it.
- ac2u 12y ago
  Y'know... I quite liked jumping in and reading right away! Didn't notice it was lacking certain assets at first.
wereHamster 12y ago
I see that Remutable uses 'patches'. How does it compare with ShareJS? Does it allow collaborative editing (ie multiple people changing the same data structure concurrently, is patch application commutative)? The idea behind ShareJS (or OT for that matter) is awesome. You never have to think about what to do if two people edit the same document concurrently. It just works. You get undo for free. And you can write the code that it saves automatically in the background. Never again do users have to press a 'save' button. A while ago I wrote a library which builds on top of the same ideas as ShareJS (ie. OT) but with slightly different focus. You define object types and their properties. This allows the library to automatically parse JSON into JavaScript class instances. It also tracks changes to these objects in a very similar way as ShareJS: each change has a 'path' (where something was changed) and then the actual chang record (currently set and splice). I use Avers in an SPA which is an editor of fairly complex data structures (90 different object types, nested ~5 levels deep). For rendering I use React. The integration is really simple: Attach a change listener to the root and re-render when anything changes. https://github.com/wereHamster/avers https://github.com/wereHamster/avers https://caurea.org/2015/02/08/the-future-of-avers.html https://caurea.org/2015/02/08/the-future-of-avers.html
- elierotenberg 12y ago
  I didn't now about ShareJS, but I'll definitely check it out. At its core, the patch abstraction is commutative (and more importantly, reversible), so I should be able to implement the 'rebase'-like operation. My feeling is that it would be more efficient to perform this at a higher level though, by replaying Actions in the right order in every client to get eventual consistency.
  - wereHamster 12y ago
    ShareJS is just one implementation of OT. If you want to check something out, then start with OT (http://en.wikipedia.org/wiki/Operational_transformation http://en.wikipedia.org/wiki/Operational_transformation).
    - elierotenberg 12y ago
      Very interesting pointer, thanks. Remutable.Patch objects represent a collection of map edits, ie. a set of { key, value } pairs indicating that 'key' should be replaced by 'value' (and value can be void 0 to indicate deletion). They also embed a version and a hash for faster matching, but they can already be reverted/combined (for undo/redo stack or batching). I will read more about OTs!
      - aidos 12y ago
        OTs are the same idea along with a sort of rebase step for replaying changes that come in to get the head to the same point (regardless of the order in which the operations were received). It seems like a great model, but as I mentioned above, there's surprisingly little code around for you to draw on. I seem to recall reading a conversation from the ShareJS guys (though maybe it was someone else) about how hairy the edge cases can get when you're trying to do it for general data-structures (which was disappointing, because conceptually it seems simple).
  - dugmartin 12y ago
    You should also checkout ot.js: https://github.com/Operational-Transformation/ot.js/ https://github.com/Operational-Transformation/ot.js/ They have a nice visualization of OT here: http://operational-transformation.github.io/visualization.html http://operational-transformation.github.io/visualization.ht...
amelius 12y ago
Hmm, I don't really get it yet, but "single source of truth" sounds to me like it needs extra round-trips to this "source of truth" in order to update the display (even if the updates come from the client itself). Is that correct?
- elierotenberg 12y ago
  You're right, although its fairly easy to implement optimistic updates on top of this. Since patch objects are reversible, you can use a simple logical clock and an undo/redo stack to perform optimistic updates locally, and undo them/redo them when the remote SSoT sends updates.
  - amelius 12y ago
    Well, it could be me, but that doesn't sound very simple. Isn't that something the framework should take care of? Also, what happens when simultaneously transmitted actions/patches are in conflict?
    - elierotenberg 12y ago
      Absolutely. I plan to include it in the framework ultimately, although it requires implementing an optimistic client-side dispatcher in JS which mocks the optimistic behaviour of the actual, eventually true server-side dispatcher (which may not always be implemented in full JS, eg. if it needs to sync with a db). Conflicts are handled by the server-side dispatcher logic. Many applications can handle conflicts easily (eg. chat rooms which only append new messages), others can be more complex (eg. document editing). As mentionned in another comment, Remutable.Patch leaves room for rebase-like algorithms. Its definitely a very promising further work! :)
  - deleted 12y ago
    [deleted]
zzzaim 12y ago
Alas, I've been working on something like this for the last few weeks on and off, especially the "patching" of immutable data, my solution was to use jiff[1], which generates JSON Patch operations[2], which is then applied to the Immutable object using immpatch[3] (a lib I wrote). A cursory look at Remutable seems it to be the better and faster option :) [1] https://github.com/cujojs/jiff https://github.com/cujojs/jiff [2] https://tools.ietf.org/html/rfc6902 https://tools.ietf.org/html/rfc6902 [3] https://github.com/zaim/immpatch https://github.com/zaim/immpatch (edited formatting)
- elierotenberg 12y ago
  That was nearly exactly my initial approach! :) However I realized actual diffing was really slow, especially for large collections, as it involves scanning the entire structure for changes, when in fact all changes could simply be flagged upon mutation. Conceptually, it performs diffing (ie. it gives you a diff object), its just much, much faster!
rattray 12y ago
slightly offtopic question; what are all those `should.be.exactly(whatever)` calls in the Remutable readme?
- elierotenberg 12y ago
  Its shouldJS. An assertion lib with a fun chainable DSL. Basically whatever.should.predicate desugars to assert(predicate(whatever)). See https://github.com/shouldjs/should.js https://github.com/shouldjs/should.js :)
verroq 12y ago
Synchronising state over the wire? Surely this is a solved problem in games development already - serialise your state into a consistent binary form, send over the binary diff of the previous state (AKA delta compression), deserialise on client side.
rattray 12y ago
Remutable looks really impressive. I'm a little leery of the rest of it, though, as it means my backend is no longer a simple REST API. I'm also rather nonplussed that the full-stack Flux doesn't include optimistic display with loading icon, error state, and rollback handling out of the box. It doesn't even seem like an easy thing to do manually with this setup, though I haven't played with it enough to know. Few minor questions: it looks like a base Remutable must be an Immutable.Map(). Can child items be any of the Immutable datatypes? Do we get updateIn() and similar?
- elierotenberg 12y ago
  Thanks for your feedback. Several other comments point in the same direction: automatic (or assisted) handling of optimistic updates. I think this is actually relatively easy, since there are clear hooks in the Nexus Flux API to implement dispatch-time and update-time behaviour, but I will definitely provide an example, or even better, an implementation of this that just works out of the box.
  - rattray 12y ago
    Awesome! Excited to see how this goes, especially with a bigger example!
  - drabinowitz 12y ago
    This is a really cool implementation! I'm wondering if it wouldn't be possible to handle optimistic updates by submitting a request to the server and then submitting an optimistic response from the client to itself. Then, when the server comes back, you hook into the already present optimistic response and just confirm or deny it based on the response from the server. That way you keep as little state on the client as possible.
Rapzid 12y ago
The best example of single source of truth may be chat, but I believe chat is the best example of a replicated log. This throw me off a bit while reading the article :| Left it, came back to the comments later to see people talking about OT and realized I missed something. Just an obesevation.
mandeepj 12y ago
> We will rather record the mutations (or more accurately, the transitions between immutable states), and just replay them on the client. Few questions - How can we replay all the mutations? How about instead of replaying, if we just show the last mutation? Will it not be the same thing? I think this flux architecture will work best only in chatting applications. This approach is also making the app itself very chatter so performance concerns in case you do not have strong machines. I think on form pages this flux approach may not be required.
mikaelemtinger 12y ago
Take a look at swarm.js for a fantastic way to keep stores in sync - even works in offline. Matrix.io could also be an interesting option for this, but focuses more on communication than shared data.

81 points by elierotenberg 12y ago

The single source of truth diagram is so great! :) Was getting tired of turning my head sideways..

sehr 12y ago

cache: http://webcache.googleusercontent.com/search?q=cache:http://blog.rotenberg.io/flux-over-the-wire-3/ http://webcache.googleusercontent.com/search?q=cache:http://...

elierotenberg 12y ago

Wow, a crashed site presenting a web architecture surely isn't classy! Sorry about that, I'm working on it.

ac2u 12y ago
Y'know... I quite liked jumping in and reading right away! Didn't notice it was lacking certain assets at first.

wereHamster 12y ago

I see that Remutable uses 'patches'. How does it compare with ShareJS? Does it allow collaborative editing (ie multiple people changing the same data structure concurrently, is patch application commutative)? The idea behind ShareJS (or OT for that matter) is awesome. You never have to think about what to do if two people edit the same document concurrently. It just works. You get undo for free. And you can write the code that it saves automatically in the background. Never again do users have to press a 'save' button. A while ago I wrote a library which builds on top of the same ideas as ShareJS (ie. OT) but with slightly different focus. You define object types and their properties. This allows the library to automatically parse JSON into JavaScript class instances. It also tracks changes to these objects in a very similar way as ShareJS: each change has a 'path' (where something was changed) and then the actual chang record (currently set and splice). I use Avers in an SPA which is an editor of fairly complex data structures (90 different object types, nested ~5 levels deep). For rendering I use React. The integration is really simple: Attach a change listener to the root and re-render when anything changes. https://github.com/wereHamster/avers https://github.com/wereHamster/avers https://caurea.org/2015/02/08/the-future-of-avers.html https://caurea.org/2015/02/08/the-future-of-avers.html

elierotenberg 12y ago
I didn't now about ShareJS, but I'll definitely check it out. At its core, the patch abstraction is commutative (and more importantly, reversible), so I should be able to implement the 'rebase'-like operation. My feeling is that it would be more efficient to perform this at a higher level though, by replaying Actions in the right order in every client to get eventual consistency.
- wereHamster 12y ago
  ShareJS is just one implementation of OT. If you want to check something out, then start with OT (http://en.wikipedia.org/wiki/Operational_transformation http://en.wikipedia.org/wiki/Operational_transformation).
  - elierotenberg 12y ago
    Very interesting pointer, thanks. Remutable.Patch objects represent a collection of map edits, ie. a set of { key, value } pairs indicating that 'key' should be replaced by 'value' (and value can be void 0 to indicate deletion). They also embed a version and a hash for faster matching, but they can already be reverted/combined (for undo/redo stack or batching). I will read more about OTs!
    - aidos 12y ago
      OTs are the same idea along with a sort of rebase step for replaying changes that come in to get the head to the same point (regardless of the order in which the operations were received). It seems like a great model, but as I mentioned above, there's surprisingly little code around for you to draw on. I seem to recall reading a conversation from the ShareJS guys (though maybe it was someone else) about how hairy the edge cases can get when you're trying to do it for general data-structures (which was disappointing, because conceptually it seems simple).
- dugmartin 12y ago
  You should also checkout ot.js: https://github.com/Operational-Transformation/ot.js/ https://github.com/Operational-Transformation/ot.js/ They have a nice visualization of OT here: http://operational-transformation.github.io/visualization.html http://operational-transformation.github.io/visualization.ht...

amelius 12y ago

Hmm, I don't really get it yet, but "single source of truth" sounds to me like it needs extra round-trips to this "source of truth" in order to update the display (even if the updates come from the client itself). Is that correct?

elierotenberg 12y ago
You're right, although its fairly easy to implement optimistic updates on top of this. Since patch objects are reversible, you can use a simple logical clock and an undo/redo stack to perform optimistic updates locally, and undo them/redo them when the remote SSoT sends updates.
- amelius 12y ago
  Well, it could be me, but that doesn't sound very simple. Isn't that something the framework should take care of? Also, what happens when simultaneously transmitted actions/patches are in conflict?
  - elierotenberg 12y ago
    Absolutely. I plan to include it in the framework ultimately, although it requires implementing an optimistic client-side dispatcher in JS which mocks the optimistic behaviour of the actual, eventually true server-side dispatcher (which may not always be implemented in full JS, eg. if it needs to sync with a db). Conflicts are handled by the server-side dispatcher logic. Many applications can handle conflicts easily (eg. chat rooms which only append new messages), others can be more complex (eg. document editing). As mentionned in another comment, Remutable.Patch leaves room for rebase-like algorithms. Its definitely a very promising further work! :)
- deleted 12y ago
  [deleted]

zzzaim 12y ago

Alas, I've been working on something like this for the last few weeks on and off, especially the "patching" of immutable data, my solution was to use jiff[1], which generates JSON Patch operations[2], which is then applied to the Immutable object using immpatch[3] (a lib I wrote). A cursory look at Remutable seems it to be the better and faster option :) [1] https://github.com/cujojs/jiff https://github.com/cujojs/jiff [2] https://tools.ietf.org/html/rfc6902 https://tools.ietf.org/html/rfc6902 [3] https://github.com/zaim/immpatch https://github.com/zaim/immpatch (edited formatting)

elierotenberg 12y ago
That was nearly exactly my initial approach! :) However I realized actual diffing was really slow, especially for large collections, as it involves scanning the entire structure for changes, when in fact all changes could simply be flagged upon mutation. Conceptually, it performs diffing (ie. it gives you a diff object), its just much, much faster!

rattray 12y ago

slightly offtopic question; what are all those `should.be.exactly(whatever)` calls in the Remutable readme?

elierotenberg 12y ago
Its shouldJS. An assertion lib with a fun chainable DSL. Basically whatever.should.predicate desugars to assert(predicate(whatever)). See https://github.com/shouldjs/should.js https://github.com/shouldjs/should.js :)

verroq 12y ago

Synchronising state over the wire? Surely this is a solved problem in games development already - serialise your state into a consistent binary form, send over the binary diff of the previous state (AKA delta compression), deserialise on client side.

rattray 12y ago

Remutable looks really impressive. I'm a little leery of the rest of it, though, as it means my backend is no longer a simple REST API. I'm also rather nonplussed that the full-stack Flux doesn't include optimistic display with loading icon, error state, and rollback handling out of the box. It doesn't even seem like an easy thing to do manually with this setup, though I haven't played with it enough to know. Few minor questions: it looks like a base Remutable must be an Immutable.Map(). Can child items be any of the Immutable datatypes? Do we get updateIn() and similar?

elierotenberg 12y ago
Thanks for your feedback. Several other comments point in the same direction: automatic (or assisted) handling of optimistic updates. I think this is actually relatively easy, since there are clear hooks in the Nexus Flux API to implement dispatch-time and update-time behaviour, but I will definitely provide an example, or even better, an implementation of this that just works out of the box.
- rattray 12y ago
  Awesome! Excited to see how this goes, especially with a bigger example!
- drabinowitz 12y ago
  This is a really cool implementation! I'm wondering if it wouldn't be possible to handle optimistic updates by submitting a request to the server and then submitting an optimistic response from the client to itself. Then, when the server comes back, you hook into the already present optimistic response and just confirm or deny it based on the response from the server. That way you keep as little state on the client as possible.

Rapzid 12y ago

The best example of single source of truth may be chat, but I believe chat is the best example of a replicated log. This throw me off a bit while reading the article :| Left it, came back to the comments later to see people talking about OT and realized I missed something. Just an obesevation.

mandeepj 12y ago

> We will rather record the mutations (or more accurately, the transitions between immutable states), and just replay them on the client. Few questions - How can we replay all the mutations? How about instead of replaying, if we just show the last mutation? Will it not be the same thing? I think this flux architecture will work best only in chatting applications. This approach is also making the app itself very chatter so performance concerns in case you do not have strong machines. I think on form pages this flux approach may not be required.

mikaelemtinger 12y ago

Take a look at swarm.js for a fantastic way to keep stores in sync - even works in offline. Matrix.io could also be an interesting option for this, but focuses more on communication than shared data.