8 ms·
The most useful pattern I know of for offline web apps is the command queue. Basically, the rendered state of the client is the acknowledged server state plus
by mbrock 10y ago
The most useful pattern I know of for offline web apps is the command queue.
Basically, the rendered state of the client is the acknowledged server state plus the client-side command queue.
User actions don't make a server request and then update the UI. Instead they directly append to the local command queue, which updates the UI state, and right away the client begins communicating with the server to make the local change real.
While the client's command queue is nonempty, the UI shows a spinner or equivalent. If commands cannot be realized because of network failure, the UI remains functional but with a clear warning that changes are waiting to be synchronized.
(The API for connectivity status are useful for making sure that the command queue resumes syncing when the user's internet comes back.)
- mark242 10y agoI suspect that using Promise.all() helps greatly with this.
- mbrock 10y agoThe way I've done it is to just have an array of commands and a simple queue thing that keeps retrying its head element until success, triggered by connectivity change or manual user retry. (This was also useful when our backend had random issues causing 500s sometimes.) A command has both an AJAX request and a state updating function. It's really easy with a React-like framework because you can just apply the command queue's state changes as part of the main view render, without actually modifying the main state.
- mark242 10y agoYeah, look into using Promise.all. You're probably looking at a good few hundred lines of code reduction. What you're describing as a "command" (request / state update) is just a promise with a map function applied. Your "simple queue thing" can just be an array that you fire Promise.all at.
- mbrock 10y agoCan you serialize a promise to localStorage? How do you deal with network failure in the middle of the promise array?
- mark242 10y agoIf a promise in the array is rejected, Promise.all (which returns a promise) rejects with (supposedly) the first rejected promise, but that's extremely hard to predict if you have two promises which could reject. Serializing a promise to localStorage; I get what you're trying to do, you want to have a worker pick up exactly where it left off when you left the app. This is where a service worker would help you. I suppose you could write some kind of durable mailbox a la Akka, only running in the browser.
- mbrock 10y agoYeah, these kinds of issues were why I decided to manually manage an array of serializable commands instead of using Promise.all. Also showing the state of the queue (Sims-style, or just a number of pending changes) is easier when you control the queue yourself.
- yonkshi 10y agoHaving recently built an offline first app (mobile app I'll admit), we didn't use a command queue, and I regret it deeply now. I advise everyone who's reading this to go for a command queue. Saves a lot of time debugging data sync. Our biggest challenge was to sync data with relationships, especially data with circular relationships. We couldn't come up with a generic way to sync circular relationships, so we ended up building a very special purpose buffer on both client and server side.
- CorvusCrypto 10y agois there anything more you can say about your buffer solution? Curious if it is something similar to vector clock syncing that you guys came up with.
- Pxtl 10y agoWhat was the challenge with circular relationships? I work on an offline-first system and all inter-object relationships are managed by GUIDs, so cyclical references don't present any problems - objects are sent and received as one big list keyed by guids, not as a tree.
- allannienhuis 10y ago"objects are sent and received as one big list keyed by guids, not as a tree" That's an interesting pattern I've not heard of before - any links you could recommend for more details on it? It sounds really useful for moderate sized data sets.
- kemayo 10y agoI remember coming across it fairly heavily in react/redux stuff, mostly because it makes dealing with redux's stores a lot simpler if you avoid nested structures like that. There's a library called normalizr[1], whose purpose is to take nested API responses and turn them into flat structures. Have an article[2] about it. [1]: https://github.com/paularmstrong/normalizr https://github.com/paularmstrong/normalizr [2]: https://tonyhb.gitbooks.io/redux-without-profanity/content/normalizer.html https://tonyhb.gitbooks.io/redux-without-profanity/content/n...
- blazespin 10y agoYes, This. I struggled with understanding how I'd deal with a complex offline multi user collaborative web app until I starting looking at in terms of a Kafka queue. Gives you everything, including undo/history. Any suggestions on front end frameworks?
- mbrock 10y agoI implemented such a solution in a React app and I really appreciated the ability to apply pending state changes without mutating the state itself. I don't know of any open source libraries for the command queue itself. If your state changing commands go through some kind of layer that you control then this stuff is easier. When I implemented it, I first refactored all the commands to go through the same code path, which I could then modify to implement the queue.
- wnevets 10y agoKnow of any good examples/tutorials of this?
- mbrock 10y agoUnfortunately not, but if you think of it as using the command pattern in a queue-ish way to get a similar behavior as the command queue of "The Sims", you have the basic architecture, and from there it's just a matter of coding. The thing about React-like frameworks that makes it very nice is that you can keep the command queue in a separate place and have a root rendering function like this: 1. Set the view state to a copy of the actual state 2. Update the view state according to each queued command in turn 3. Render the view state including a status bubble showing that some changes are not saved yet And separately from the rendering, you have a worker that tries (and retries) to perform the queued commands. When a command is successfully performed, it's removed from the queue and its effect on the state is saved in the actual state. Since I can't find anything on Google when searching for "react command queue" it would be cool to write a blog post with a simple example, but I don't know when I'd have time, so I encourage anyone who's implemented a similar thing to go ahead.
- ibash 10y agoThis is similar to what I wrote about here: https://blog.superhuman.com/architecting-a-web-app-to-just-work-offline-part-1-8697f316c0eb#.p5tofnlfu https://blog.superhuman.com/architecting-a-web-app-to-just-w...
- brilliantcode 10y agoDo you mean like CQRS/ES?
- mbrock 10y agoIt makes a lot of sense in an event sourcing context because the code used for applying events can be the same in the optimistic case and in the real updates. In our case, we weren't using an event sourcing architecture, but this client side pattern can be used anyway.
- thenipper 10y agoThat feedback that the commands are queued or can't be processed is key. I encountered an issue on an app I manage at work where the UI showed that a certain state changing action had taken place before it was resolved on the sever. Needless to say it caused a variety of issues as users thought the system had processed a significant action(hiring a candidate) when they hadn't.
- freditup 10y agoWhat about in cases where the commands may fail because of the actions of other users? For example, say you had a virtual market in a game and someone did a "buy from John" command while offline. they then proceed to use this purchase to beat future levels. However, once the user reconnects online, it turns out that John already sold to somebody else, and thus their purchase and everything after it is invalidated. In your command queue pattern, what sort of ways do you handle this? Have certain "chokepoints" where the user must be online to proceed? What if you have an app where that sort of chokepoint seems to occur too frequently to make the queue useful? Edit: I think this is similar to Fiahil's sibling comment, also relevant points made there. Thanks for the quality thoughts everyone!
- JoelMcCracken 10y agoThe real thing is, it is important to synchronize with the server at that point, otherwise something could be invalid. It may not be possible to make every app work while offline. You'll probably want to potentially disable certain features while offline, such as store purchases (or, at least, queue them up, but don't show them as having been successfully purchased)
- smackmybishop 10y agoThat case would probably need server synchronization to confirm, but other cases can be handled with operational transformations. https://en.wikipedia.org/wiki/Operational_transformation https://en.wikipedia.org/wiki/Operational_transformation
- mbrock 10y agoThat wasn't a big problem in our particular application, so in case of failure we would just report a potentially unsatisfactory error message along the lines of "Your changes could not be saved. Please try again." (with a list of the discarded commands' descriptions). In a situation where this kind of thing was more important, I would think about how to let the user decide how to reconciliate their changes. It could be that a choice of discarding or retrying would suffice, or something more complex. The "chokepoint" notion is also useful, and in fact our app did have a distinction between potentially offline actions and necessarily synchronous actions, but our synchronous actions were mostly queries like searches.
- ramzyo 10y agoThis makes so much sense. Would have saved me many headaches when developing an offline app a few years ago.
- Fiahil 10y agoYou still have to handle cases where the server state has been updated (possibly via another medium, event, ...) and when the user's internet comes back it's not a matter of pushing clients' commands anymore. Instead you have to merge changes (either backend or front end side) before fetching the new state. For example, I try to purchase an item on my desktop, but can't because I lost connectivity. So, I proceed to buy it on my mobile. When the net comes back, did I mean to buy this item two times or just one ? In this case, it's easy to find a workable solution (cancel the second order command), but you get the idea: It's not trivial. CQRS and ES, are wonderful tools in such situations. Past that point, your web app is far away from the little CRUD mashup it was at the beginning.
- mbrock 10y agoYeah, heavy transactions like purchases need special care, but that's true in general. At least you'd want a confirmation from the user. In our case, a simple command queue solved the vast majority of use cases, and the corner cases were not disasters but at worst somewhat confusing.
- junaidajmal 10y agoStay at home mom Kelly Richards from New York after resigning from her full time job managed to average from $6000-$8000 a month from freelancing at home… This is how she done ,,,,,,,,,,,,,,,HERE ►WWW.bizcash3.COM
- andrewstuart2 10y agoConflicts are actually not all that difficult to solve. We had to solve this exact problem with a time entry solution that needed to be mostly available offline. Taking most of our inspiration from Git, it was simple. Define the atomic unit of conflict, and the definition of a conflict that cannot be automatically resolved, and simply help the end user understand and resolve the conflict. 95% of cases were fairly easy to merge without interaction, once we had defined "conflict." And the remaining 5% just required a little extra user experience design to help surface the appropriate resolution.
- 10y ago
- sgk284 10y agoI'm a big fan of command queues, but if there is a chance of a server-side failure that can't be realized immediately on the client (e.g. an edit conflict with a different user editing a same piece of information), then I find that surfacing those errors in a meaningful way can be difficult and frustrating for the user. By the time the client has synced with the server, the client could be doing something completely unrelated in an entirely different part of the app. Explaining to them that "the thing you were editing two hours ago has an issue, go here to resolve it" can be tricky.
- mbrock 10y agoYeah, it can get tricky, but it's a tradeoff between working nicely in the majority of cases and being accurate in the corner cases. If you make sure to show the user a clear warning that they're working offline, then they might be more understanding if their changes are rejected two hours later.
- mjevans 10y agoTreating changes more like emails/outgoing order forms might be useful here. You could popup a /failure/ message and leave a little indicator icon that takes them to the queue and lets them see the items that failed. If retry is an option they can see it, if retry is not an option they can be told why.
- junaidajmal 10y agoStay at home mom Kelly Richards from New York after resigning from her full time job managed to average from $6000-$8000 a month from freelancing at home… This is how she done ,,,,,,,,,,,,,,,HERE ►WWW.bizcash3.COM
- amelius 10y agoUnfortunately, the "command queue" abstraction does not work very well with access control. Imagine that one day the requirements w.r.t. security change. What do you do when some parts of the state may not be viewed by all users? And what if that logic depends on the state itself?