5 ms·

Eventual Consistency in Real-time Web Apps

31 points by strife25 12y ago

dahjelle 12y ago
Are there any "standard" models for treating a realtime web app as just another distributed database node (with, of course, extra security precautions and having to do server-side data re-validation)? I'm aware of CouchDB/PouchDB and Meteor's use of mini-Mongo client-side. Are there others?
- mantraxB 12y ago
  Actors.
  - seanmcdirmid 12y ago
    Actors says nothing about consistency; you have to do it all on your own. Basically a non solution in this area.
    - jburwell 12y ago
      +1. Actors are simply a unit of concurrency and means of communication between those units. Their scope and division of labor, as well as, the associated communication protocols are left to developer. They provide no inherent consistency guarantees.
- lucian1900 12y ago
  If you have a CRDT implementation you can use on both sides, you can perform operations on the client and merge with the server correctly.
  - dahjelle 12y ago
    That's exactly what I'm planning on doing, as that seems to be simultaneously correct and relatively easy to program. I'm surprised, though, that I haven't found a "nicely packaged" version of some CRDTs, a server DB, and a client-side DB. I suppose it is probably because everyone's use-cases are so different and that using CRDTs limit the data you can use, to some degree.
    - skrebbel 12y ago
      I'm just learning about CRDTs because of this thread. Which CRDT things did you find, did you experiment with any, which are you using and why?
      - dahjelle 12y ago
        (semi cross-posted from another comment) Aral Balkan gave a talk on a CRDT called WOOT for text editing[1] that I found really helpful to get the general idea of the concept. (Really only the last 8 or 9 minutes of his talk.) If you want more of the nitty-gritty on some of the different types of CRDT, there's fairly readable paper on the topic[2]. From there, you can start using Google Scholar to find the other papers that have been written. [1] https://www.youtube.com/watch?v=NSTZ4mIv_wk https://www.youtube.com/watch?v=NSTZ4mIv_wk [2] http://hal.upmc.fr/docs/00/55/55/88/PDF/techreport.pdf http://hal.upmc.fr/docs/00/55/55/88/PDF/techreport.pdf
- skrebbel 12y ago
  This seems to be like the big problem of modern (web) app development. Even for not-very-realtime apps, the problem is essentially the same, and non trivial (single page web) apps I've seen end up hand-cooking some sort of sync protocol through a mixture of REST calls and maybe some websockets here and there. I like the Couch/Pouch idea, but it's really only handy when CouchDB fits your data. You can stream all CouchDB updates to a RDBMS using the changes stream, for example when you want to easily create management information, but I don't see how you can avoid making the CouchDB bunch-of-documents-per-db concept central to the app. I haven't looked into Meteor, but my suspicion is that due to Mongo's comparably document-oriented setup, it can't be all that different.
- mathbr 12y ago
  When I had to do that for some apps, the main issues were * while being offline some notifications get lost so one needs to get a sense of the general state of the central server * I wanted to synchronize only part of the database to each client (data it has the right to see) * pushes to the server could not be generic due to business logic (again, rights of the user) Thus I started recording changes on the client, pushing them to the central server through functional endpoints and then pulling all changes from the server. To pull data from the server I found a paper[1] with an interesting way of encoding data so that the client can get a sense of the difference by getting a data structure of size proportional to the diff (no matter the whole dataset size). I developed a library for that, pretty limited for now [2]. [#2] http://conferences.sigcomm.org/sigcomm/2011/papers/sigcomm/p218.pdf http://conferences.sigcomm.org/sigcomm/2011/papers/sigcomm/p... [#1] https://github.com/3musket33rs/mathsync https://github.com/3musket33rs/mathsync
aidos 12y ago
I hadn't seen the merge / fill idea before. Worth reading to get more ideas on this subject. OT but seems like a good time to ask, does anyone have any experience with Operational Transformations for dealing with syncing in web apps? I've started throwing the idea around for a web app I'm building and it seems like a really interesting pattern. The only real implementation I can find is in sharejs [1]. I work with python on the backend and I can't really find an implementation of it (though for my constrained use-case I can generate good enough code to do it myself). It seems like a good approach but there's very little in the way of libraries implementing it, so I was wondering if it's somehow problematic. [1] http://sharejs.org/ http://sharejs.org/
- dahjelle 12y ago
  In general, I've read that OT tends to be very complicated to get right, which is why only a few libraries implement it. If you can use what's called a CRDT, your life will be easier, but there are fewer data structures that are supported. Aral Balkan gave a talk on OT vs a CRDT called WOOT for text editing[1] that I found really helpful. If you want more of the nitty-gritty on some of the different types of CRDT, there's fairly readable paper on the topic[2]. [1] https://www.youtube.com/watch?v=NSTZ4mIv_wk https://www.youtube.com/watch?v=NSTZ4mIv_wk [2] http://hal.upmc.fr/docs/00/55/55/88/PDF/techreport.pdf http://hal.upmc.fr/docs/00/55/55/88/PDF/techreport.pdf
  - aidos 12y ago
    That's great information, thank you. The paper looks really interesting - going to take a little while to get through, might start with the video :) I'd also read that it's hard to get right (I think that was in a comment on HN by one of the people who worked on wave/sharejs). In my case I've limited myself to two different flavours of small object with a limited set of fields (instead of a totally generalised system). Most of the algorithms I've seen focus on the text element of the syncing but I'm much more interested in the object / list / field part for my use case. Sharejs actually have a generic system of handling the OT on arbitrary json structures, but I'm not sure how well it works.
dgreensp 12y ago
It's a lot simpler to just have the socket, with a notification when your PUT has landed. This is what Meteor does. I don't see anything I recognize as eventual consistency in the traditional sense. It sounds like the client is just trying to receive a stream of updates, which could have been made by a single writer, and the only consistency issues are caused by the different overlapping mechanisms for getting updates.
- strife25 12y ago
  That is certainly simpler. I will say that our notifications come from a different service that is separate from the service the responds to HTTP requests, which leads to the process that is described in my post.
- macspoofing 12y ago
  The complexity depends on what constraints you put on yourself. Here's an example, suppose you have a UI panel with a number of checkboxes, radiobuttons, and sliders, and you want to synchronize the state across several users, without blocking (i.e. if a user changes state, he doesn't need to wait for server response to change change state again). Global state exists on a single server. Let's say your users start changing state furiously, what happens? ...well, the object state will live in several places (and slightly different for every client): 1) Local: In the view but before event dispatched to the model. 2) Local: In model but before it was synced to the server. 2a) Local: After model sync and in queue waiting to be sent to the server. 3) Remote: In transit to the server. 4) Remote: On the server (i.e. the one true state). 5) Remote: In transit to the clients. 6) Local: Received but before it was synced to the model. 7) Local: In model but before synced to the view. 8) In the initial state sync (e.g. when a new client joins) So at any given point, every client state is inconsistent with every other client (maybe even significantly if the latency for one or more of the clients is bad) since pieces of the eventual state are spread across all these locations. If now every user stops changing state furiously, every client should eventually be consistent. It's not a hard problem but not completely trivial. For example, you will need conflict resolution rules on the server and in your client and traditional distributed issues like (temporary) network partitioning and latency need to be taken into account. Message ordering will probably matter. >It sounds like the client is just trying to receive a stream of updates, which could have been made by a single writer Not necessarily. You have the same synchronization issues whether the updates are coming from a single source, or from multiple sources.
deoxxa 12y ago
I've had a lot of success with embedding initial content for a page as JSON in a script tag with a specific ID and a type of "application/json". This way you can rely on the content being there and sidestep the whole timing issue completely. You can potentially speed up your loading time as well, if you have a bunch of things you'd usually fire additional requests off for. I believe I got this idea from Backbone's documentation [1] originally, but I've seen it mentioned in other places as well. [1]: http://backbonejs.org/#FAQ-bootstrap http://backbonejs.org/#FAQ-bootstrap
cromwellian 12y ago
The Errai J2EE framework for GWT has an 'Operational Transform Eventually Consistent" Web Socket bus implementation for those curious. (https://github.com/errai/errai/tree/master/errai-otec https://github.com/errai/errai/tree/master/errai-otec) Demo video: https://www.youtube.com/watch?v=ChCasRr0cZc https://www.youtube.com/watch?v=ChCasRr0cZc

31 points by strife25 12y ago

Are there any "standard" models for treating a realtime web app as just another distributed database node (with, of course, extra security precautions and having to do server-side data re-validation)? I'm aware of CouchDB/PouchDB and Meteor's use of mini-Mongo client-side. Are there others?

mantraxB 12y ago
Actors.
- seanmcdirmid 12y ago
  Actors says nothing about consistency; you have to do it all on your own. Basically a non solution in this area.
  - jburwell 12y ago
    +1. Actors are simply a unit of concurrency and means of communication between those units. Their scope and division of labor, as well as, the associated communication protocols are left to developer. They provide no inherent consistency guarantees.
lucian1900 12y ago
If you have a CRDT implementation you can use on both sides, you can perform operations on the client and merge with the server correctly.
- dahjelle 12y ago
  That's exactly what I'm planning on doing, as that seems to be simultaneously correct and relatively easy to program. I'm surprised, though, that I haven't found a "nicely packaged" version of some CRDTs, a server DB, and a client-side DB. I suppose it is probably because everyone's use-cases are so different and that using CRDTs limit the data you can use, to some degree.
  - skrebbel 12y ago
    I'm just learning about CRDTs because of this thread. Which CRDT things did you find, did you experiment with any, which are you using and why?
    - dahjelle 12y ago
      (semi cross-posted from another comment) Aral Balkan gave a talk on a CRDT called WOOT for text editing[1] that I found really helpful to get the general idea of the concept. (Really only the last 8 or 9 minutes of his talk.) If you want more of the nitty-gritty on some of the different types of CRDT, there's fairly readable paper on the topic[2]. From there, you can start using Google Scholar to find the other papers that have been written. [1] https://www.youtube.com/watch?v=NSTZ4mIv_wk https://www.youtube.com/watch?v=NSTZ4mIv_wk [2] http://hal.upmc.fr/docs/00/55/55/88/PDF/techreport.pdf http://hal.upmc.fr/docs/00/55/55/88/PDF/techreport.pdf
skrebbel 12y ago
This seems to be like the big problem of modern (web) app development. Even for not-very-realtime apps, the problem is essentially the same, and non trivial (single page web) apps I've seen end up hand-cooking some sort of sync protocol through a mixture of REST calls and maybe some websockets here and there. I like the Couch/Pouch idea, but it's really only handy when CouchDB fits your data. You can stream all CouchDB updates to a RDBMS using the changes stream, for example when you want to easily create management information, but I don't see how you can avoid making the CouchDB bunch-of-documents-per-db concept central to the app. I haven't looked into Meteor, but my suspicion is that due to Mongo's comparably document-oriented setup, it can't be all that different.
mathbr 12y ago
When I had to do that for some apps, the main issues were * while being offline some notifications get lost so one needs to get a sense of the general state of the central server * I wanted to synchronize only part of the database to each client (data it has the right to see) * pushes to the server could not be generic due to business logic (again, rights of the user) Thus I started recording changes on the client, pushing them to the central server through functional endpoints and then pulling all changes from the server. To pull data from the server I found a paper[1] with an interesting way of encoding data so that the client can get a sense of the difference by getting a data structure of size proportional to the diff (no matter the whole dataset size). I developed a library for that, pretty limited for now [2]. [#2] http://conferences.sigcomm.org/sigcomm/2011/papers/sigcomm/p218.pdf http://conferences.sigcomm.org/sigcomm/2011/papers/sigcomm/p... [#1] https://github.com/3musket33rs/mathsync https://github.com/3musket33rs/mathsync

aidos 12y ago

I hadn't seen the merge / fill idea before. Worth reading to get more ideas on this subject. OT but seems like a good time to ask, does anyone have any experience with Operational Transformations for dealing with syncing in web apps? I've started throwing the idea around for a web app I'm building and it seems like a really interesting pattern. The only real implementation I can find is in sharejs [1]. I work with python on the backend and I can't really find an implementation of it (though for my constrained use-case I can generate good enough code to do it myself). It seems like a good approach but there's very little in the way of libraries implementing it, so I was wondering if it's somehow problematic. [1] http://sharejs.org/ http://sharejs.org/

dahjelle 12y ago
In general, I've read that OT tends to be very complicated to get right, which is why only a few libraries implement it. If you can use what's called a CRDT, your life will be easier, but there are fewer data structures that are supported. Aral Balkan gave a talk on OT vs a CRDT called WOOT for text editing[1] that I found really helpful. If you want more of the nitty-gritty on some of the different types of CRDT, there's fairly readable paper on the topic[2]. [1] https://www.youtube.com/watch?v=NSTZ4mIv_wk https://www.youtube.com/watch?v=NSTZ4mIv_wk [2] http://hal.upmc.fr/docs/00/55/55/88/PDF/techreport.pdf http://hal.upmc.fr/docs/00/55/55/88/PDF/techreport.pdf
- aidos 12y ago
  That's great information, thank you. The paper looks really interesting - going to take a little while to get through, might start with the video :) I'd also read that it's hard to get right (I think that was in a comment on HN by one of the people who worked on wave/sharejs). In my case I've limited myself to two different flavours of small object with a limited set of fields (instead of a totally generalised system). Most of the algorithms I've seen focus on the text element of the syncing but I'm much more interested in the object / list / field part for my use case. Sharejs actually have a generic system of handling the OT on arbitrary json structures, but I'm not sure how well it works.

dgreensp 12y ago

It's a lot simpler to just have the socket, with a notification when your PUT has landed. This is what Meteor does. I don't see anything I recognize as eventual consistency in the traditional sense. It sounds like the client is just trying to receive a stream of updates, which could have been made by a single writer, and the only consistency issues are caused by the different overlapping mechanisms for getting updates.

strife25 12y ago
That is certainly simpler. I will say that our notifications come from a different service that is separate from the service the responds to HTTP requests, which leads to the process that is described in my post.
macspoofing 12y ago
The complexity depends on what constraints you put on yourself. Here's an example, suppose you have a UI panel with a number of checkboxes, radiobuttons, and sliders, and you want to synchronize the state across several users, without blocking (i.e. if a user changes state, he doesn't need to wait for server response to change change state again). Global state exists on a single server. Let's say your users start changing state furiously, what happens? ...well, the object state will live in several places (and slightly different for every client): 1) Local: In the view but before event dispatched to the model. 2) Local: In model but before it was synced to the server. 2a) Local: After model sync and in queue waiting to be sent to the server. 3) Remote: In transit to the server. 4) Remote: On the server (i.e. the one true state). 5) Remote: In transit to the clients. 6) Local: Received but before it was synced to the model. 7) Local: In model but before synced to the view. 8) In the initial state sync (e.g. when a new client joins) So at any given point, every client state is inconsistent with every other client (maybe even significantly if the latency for one or more of the clients is bad) since pieces of the eventual state are spread across all these locations. If now every user stops changing state furiously, every client should eventually be consistent. It's not a hard problem but not completely trivial. For example, you will need conflict resolution rules on the server and in your client and traditional distributed issues like (temporary) network partitioning and latency need to be taken into account. Message ordering will probably matter. >It sounds like the client is just trying to receive a stream of updates, which could have been made by a single writer Not necessarily. You have the same synchronization issues whether the updates are coming from a single source, or from multiple sources.

deoxxa 12y ago

I've had a lot of success with embedding initial content for a page as JSON in a script tag with a specific ID and a type of "application/json". This way you can rely on the content being there and sidestep the whole timing issue completely. You can potentially speed up your loading time as well, if you have a bunch of things you'd usually fire additional requests off for. I believe I got this idea from Backbone's documentation [1] originally, but I've seen it mentioned in other places as well. [1]: http://backbonejs.org/#FAQ-bootstrap http://backbonejs.org/#FAQ-bootstrap

cromwellian 12y ago

The Errai J2EE framework for GWT has an 'Operational Transform Eventually Consistent" Web Socket bus implementation for those curious. (https://github.com/errai/errai/tree/master/errai-otec https://github.com/errai/errai/tree/master/errai-otec) Demo video: https://www.youtube.com/watch?v=ChCasRr0cZc https://www.youtube.com/watch?v=ChCasRr0cZc