4 ms·

Replicache: Easy Offline-First for Existing Applications

111 points by aboodman 7y ago

aboodman 7y ago
Hi Hacker News! I'm starting a new company attempting to solve the offline-first / mobile-sync problem once and for all. Replicache makes your mobile, web, or desktop application blazingly responsive, by buffering all reads and writes to a local cache (aka "offline-first"). This is something I've been thinking about on and off for over fifteen years, on major projects at Google as well as my previous vc-backed startup. I previously thought there was no satisfying general solution, but happy to say I think I was wrong: Replicache is pretty dang easy, and it works with any existing stack. We're looking for groups to partner with would like early access. If that sounds interesting, please reach out! - Aaron
- justjake 7y ago
  I've written my own version of this for a couple projects but it's not 100% Would love to be part of the early access and see how Replicache fares
  - aboodman 7y ago
    Great, sign up here: https://replicache.typeform.com/to/eDbYYL https://replicache.typeform.com/to/eDbYYL! We'll be going through shortly to find the next set of folks to test with.
- sroussey 7y ago
  How does it compare to CouchDB/RxDb, or Gun?
  - aboodman 7y ago
    CouchDB, see: https://news.ycombinator.com/item?id=22175530 https://news.ycombinator.com/item?id=22175530 RxDB: - Doesn't guarantee convergence. It's up to the developer to provide a feed of deltas to the client and ensure they have the correct effect when applied to the client. This is the one of the hardest parts! - Provides no help with conflict resolution. Push replication just pushes the new version of each document. The server must figure out how the document differs from the stored version and what that must mean in terms of logical operations. - Implemented in JavaScript. Difficult to use in native apps. -- I'm not up to date with Gun (it has gone through many iterations), but I think it is currently trying to be a masterless system which means the same concerns as with CRDTs apply.
    - code-is-code 7y ago
      RxDB maker here. You can do conflict-resolution by subscribing to the changeFeed and when multiple versions with the same revision-height exist, you create a new version with a 'merged' document. Depending on your dataset this can be tricky and painful. RxDB can only be used with javascript and will likely ever be.
- Semaphor 7y ago
  Sorry, no technical comments but a UI one: Please change the font/style. Depending on monitor color settings, the paragraph text in Retina Thin is really hard to read, but even on my other monitor it’s still a problem with the letter "s" [0]. I think it’s a problem with the thinness and white on dark background. It was hurting my eyes and permanently distracting me while trying to read the copy. [0]: https://i.imgur.com/Ia0roKn.png https://i.imgur.com/Ia0roKn.png
  - aboodman 7y ago
    Thank you for this feedback, I'll ask our designer for his advice. Perhaps we can detect high density monitors and only do Retina in that case or something.
lootsauce 7y ago
This looks really interesting. I have been digging deep into sync for the past few months and am very interested in how this works. I am currently trying to understand automerge and delta patching trying to implement from scratch. I have looked at Gun and couch db + pouch db but both did not feel like the right answer.
- aboodman 7y ago
  Be aware that CRDTs like automerge are solving a different (and harder) problem than Replicache. They are trying to implement convergence in an asynchronous system where there is no central authority. See the excellent article https://www.inkandswitch.com/local-first.html https://www.inkandswitch.com/local-first.html for more on this type of application. Most classic web services don't have this requirement because they do in fact have a central authority -- the service itself. Moreover, for web services, it is crucial that the central authority actually be authoritative. You don't want client and server state kind of gets smooshed together arbitrarily, but for the client's view of the state to be a mere suggestion - one which the server always overrides. So my view is that CRDTs are not really an appropriate basis for building this kind of feature in a web service. However I think the tech is awesome (Replicache actually started out as a true CRDT and moved to its current design after extensive iteration with customers). See https://www.figma.com/blog/how-figmas-multiplayer-technology-works/ https://www.figma.com/blog/how-figmas-multiplayer-technology... for how Figmas came to same conclusion wrt CRDTs for their service. -- (P|C)ouchDB: - Using couch as your backend db ends up being a nonstarter for most applications. A distributed multitenant database is a big big thing and a hugely important technical decision. Most orgs are not going to go with couch just to get sync. See https://medium.com/wandering-cto/mobile-syncing-with-couchbase-6f076d8c7e08#deca https://medium.com/wandering-cto/mobile-syncing-with-couchba... for an example of this. - The couchdb replication protocol offers no help with conflict resolution. It just tells you there was a conflict and gives you two conflicting documents. This isn't practical for most applications.
  - daleharvey 7y ago
    PouchDB author here, your project looks great good job. I certainly agree that switching backends to CouchDB has made it hard for people to adopt Pouch/Couch. I have often considered how I could make Pouch work with arbitrary data sources, but as you well know its a tricky problem. However I dont understand from your website or comment here about how conflict resolution is easier? Given the situation of having 2 clients recieving an initial data of {key: value}, the clients go offline, 1 client writes {key: foo} and the other writes {key: bar}, the clients then both reconnect. What is the new state?
dstaley 7y ago
For something that's open-source, production ready thanks to the lovely folks at Mozilla, and ready to host on your own infrastructure check out Kinto[1]. Replicache and Kinto have overlapping feature sets, but my guess is that Replicache is probably a bit more optimized (especially if the delta updates are truly as small as possible). That being said, of all the problems I've run into implementing sync for my applications with Kinto, the size of update requests hasn't been one of them. Also while Kinto can support live subscriptions, the only plugin implementing them currently relies on a third-party. The biggest difference I see between them is that Replicache sits in front of your server, whereas Kinto _is_ your server. Disclosure: I'm a contributor to Kinto, primarily working on transitioning the JavaScript libraries to TypeScript. [1] https://docs.kinto-storage.org/en/latest/ https://docs.kinto-storage.org/en/latest/
- aboodman 7y ago
  I'm sorry to say that I didn't know about Kinto. Thanks for the link! Kinto seems, is in some ways, the opposite design approach to Replicache. Replicache answers the question: I have this existing web service with a whole complex backend stack, how do I make it work offline-first? Kinto answers the question: I have this website, and I don't want a backend stack. Where can I store data and have it work offline-first? I think the problem Kinto is solving is also important, thanks for sharing it.
sizediterable 7y ago
I'm curious how well Replicache composes together with web state management libraries such as Redux and Apollo GraphQL. My closest experience with something like Replicache's offline view is optimistic responses in GraphQL[1]. Maybe some more web-specific examples can be added to the site? [1] https://www.apollographql.com/docs/react/performance/optimistic-ui/ https://www.apollographql.com/docs/react/performance/optimis...
- aboodman 7y ago
  Apollo's optimistic responses are similar Replicache's local writes. The difference is that Replicache's offline writes automatically refresh every affected subscription and ensures the entire UI is consistent with the latest changes. You don't need to manually modify affected queries: https://www.apollographql.com/docs/react/performance/optimistic-ui/#adding-to-a-list https://www.apollographql.com/docs/react/performance/optimis.... How Replicache composes with Apollo needs more thought. Off the top of my head, I think that Replicache replaces the need for Apollo's unified cache, but not with many of its other features. For example, it's easy to imagine Replicache being one of several data sources backing Apollo. In fact we have a customer who has done something like that already, and it's quite nice because the consuming code doesn't even need to know whether a GraphQL query is coming from offline data or the server. That can even change over the life of the product as more features in an application become offline-enabled.
  - sizediterable 7y ago
    Thanks for a thorough explanation!

111 points by aboodman 7y ago

Hi Hacker News! I'm starting a new company attempting to solve the offline-first / mobile-sync problem once and for all. Replicache makes your mobile, web, or desktop application blazingly responsive, by buffering all reads and writes to a local cache (aka "offline-first"). This is something I've been thinking about on and off for over fifteen years, on major projects at Google as well as my previous vc-backed startup. I previously thought there was no satisfying general solution, but happy to say I think I was wrong: Replicache is pretty dang easy, and it works with any existing stack. We're looking for groups to partner with would like early access. If that sounds interesting, please reach out! - Aaron

justjake 7y ago
I've written my own version of this for a couple projects but it's not 100% Would love to be part of the early access and see how Replicache fares
- aboodman 7y ago
  Great, sign up here: https://replicache.typeform.com/to/eDbYYL https://replicache.typeform.com/to/eDbYYL! We'll be going through shortly to find the next set of folks to test with.
sroussey 7y ago
How does it compare to CouchDB/RxDb, or Gun?
- aboodman 7y ago
  CouchDB, see: https://news.ycombinator.com/item?id=22175530 https://news.ycombinator.com/item?id=22175530 RxDB: - Doesn't guarantee convergence. It's up to the developer to provide a feed of deltas to the client and ensure they have the correct effect when applied to the client. This is the one of the hardest parts! - Provides no help with conflict resolution. Push replication just pushes the new version of each document. The server must figure out how the document differs from the stored version and what that must mean in terms of logical operations. - Implemented in JavaScript. Difficult to use in native apps. -- I'm not up to date with Gun (it has gone through many iterations), but I think it is currently trying to be a masterless system which means the same concerns as with CRDTs apply.
  - code-is-code 7y ago
    RxDB maker here. You can do conflict-resolution by subscribing to the changeFeed and when multiple versions with the same revision-height exist, you create a new version with a 'merged' document. Depending on your dataset this can be tricky and painful. RxDB can only be used with javascript and will likely ever be.
Semaphor 7y ago
Sorry, no technical comments but a UI one: Please change the font/style. Depending on monitor color settings, the paragraph text in Retina Thin is really hard to read, but even on my other monitor it’s still a problem with the letter "s" [0]. I think it’s a problem with the thinness and white on dark background. It was hurting my eyes and permanently distracting me while trying to read the copy. [0]: https://i.imgur.com/Ia0roKn.png https://i.imgur.com/Ia0roKn.png
- aboodman 7y ago
  Thank you for this feedback, I'll ask our designer for his advice. Perhaps we can detect high density monitors and only do Retina in that case or something.

lootsauce 7y ago

This looks really interesting. I have been digging deep into sync for the past few months and am very interested in how this works. I am currently trying to understand automerge and delta patching trying to implement from scratch. I have looked at Gun and couch db + pouch db but both did not feel like the right answer.

aboodman 7y ago
Be aware that CRDTs like automerge are solving a different (and harder) problem than Replicache. They are trying to implement convergence in an asynchronous system where there is no central authority. See the excellent article https://www.inkandswitch.com/local-first.html https://www.inkandswitch.com/local-first.html for more on this type of application. Most classic web services don't have this requirement because they do in fact have a central authority -- the service itself. Moreover, for web services, it is crucial that the central authority actually be authoritative. You don't want client and server state kind of gets smooshed together arbitrarily, but for the client's view of the state to be a mere suggestion - one which the server always overrides. So my view is that CRDTs are not really an appropriate basis for building this kind of feature in a web service. However I think the tech is awesome (Replicache actually started out as a true CRDT and moved to its current design after extensive iteration with customers). See https://www.figma.com/blog/how-figmas-multiplayer-technology-works/ https://www.figma.com/blog/how-figmas-multiplayer-technology... for how Figmas came to same conclusion wrt CRDTs for their service. -- (P|C)ouchDB: - Using couch as your backend db ends up being a nonstarter for most applications. A distributed multitenant database is a big big thing and a hugely important technical decision. Most orgs are not going to go with couch just to get sync. See https://medium.com/wandering-cto/mobile-syncing-with-couchbase-6f076d8c7e08#deca https://medium.com/wandering-cto/mobile-syncing-with-couchba... for an example of this. - The couchdb replication protocol offers no help with conflict resolution. It just tells you there was a conflict and gives you two conflicting documents. This isn't practical for most applications.
- daleharvey 7y ago
  PouchDB author here, your project looks great good job. I certainly agree that switching backends to CouchDB has made it hard for people to adopt Pouch/Couch. I have often considered how I could make Pouch work with arbitrary data sources, but as you well know its a tricky problem. However I dont understand from your website or comment here about how conflict resolution is easier? Given the situation of having 2 clients recieving an initial data of {key: value}, the clients go offline, 1 client writes {key: foo} and the other writes {key: bar}, the clients then both reconnect. What is the new state?

dstaley 7y ago

For something that's open-source, production ready thanks to the lovely folks at Mozilla, and ready to host on your own infrastructure check out Kinto[1]. Replicache and Kinto have overlapping feature sets, but my guess is that Replicache is probably a bit more optimized (especially if the delta updates are truly as small as possible). That being said, of all the problems I've run into implementing sync for my applications with Kinto, the size of update requests hasn't been one of them. Also while Kinto can support live subscriptions, the only plugin implementing them currently relies on a third-party. The biggest difference I see between them is that Replicache sits in front of your server, whereas Kinto _is_ your server. Disclosure: I'm a contributor to Kinto, primarily working on transitioning the JavaScript libraries to TypeScript. [1] https://docs.kinto-storage.org/en/latest/ https://docs.kinto-storage.org/en/latest/

aboodman 7y ago
I'm sorry to say that I didn't know about Kinto. Thanks for the link! Kinto seems, is in some ways, the opposite design approach to Replicache. Replicache answers the question: I have this existing web service with a whole complex backend stack, how do I make it work offline-first? Kinto answers the question: I have this website, and I don't want a backend stack. Where can I store data and have it work offline-first? I think the problem Kinto is solving is also important, thanks for sharing it.

sizediterable 7y ago

I'm curious how well Replicache composes together with web state management libraries such as Redux and Apollo GraphQL. My closest experience with something like Replicache's offline view is optimistic responses in GraphQL[1]. Maybe some more web-specific examples can be added to the site? [1] https://www.apollographql.com/docs/react/performance/optimistic-ui/ https://www.apollographql.com/docs/react/performance/optimis...

aboodman 7y ago
Apollo's optimistic responses are similar Replicache's local writes. The difference is that Replicache's offline writes automatically refresh every affected subscription and ensures the entire UI is consistent with the latest changes. You don't need to manually modify affected queries: https://www.apollographql.com/docs/react/performance/optimistic-ui/#adding-to-a-list https://www.apollographql.com/docs/react/performance/optimis.... How Replicache composes with Apollo needs more thought. Off the top of my head, I think that Replicache replaces the need for Apollo's unified cache, but not with many of its other features. For example, it's easy to imagine Replicache being one of several data sources backing Apollo. In fact we have a customer who has done something like that already, and it's quite nice because the consuming code doesn't even need to know whether a GraphQL query is coming from offline data or the server. That can even change over the life of the product as more features in an application become offline-enabled.
- sizediterable 7y ago
  Thanks for a thorough explanation!