10 ms·

Offline-First Apps: Why Should Apps Be Made to Work in an Offline State?

121 points by 8bitsrule 4y ago

jleyank 4y ago
Why must I have three separate must-work activities (your front-end, my network, your server)? Why not just let me use the supercomputer I have in my possession and run the thing without the baggage and overhead?
- tested23 4y ago
  People expect apps to work across all their devices
  - szundi 4y ago
    you can always sync, that's good practice with "Offline First"
- dfabulich 4y ago
  That's what TFA is about. > 1. Users are offline; experiencing latency issues or are in unreliable network conditions. > 2. Fetching the data over the network will be slower than fetching it from a local source. > 3. The app users should be informed about the low network conditions but it shouldn’t be a hindrance to their objective. > 4. Users’ network and battery status are taken into account, and thus only the data that has changed since the last synchronization should be synced.
  - swyx 4y ago
    what is TFA? not a familiar acronym in this context (usually it means The Force Awakens)
    - gurjeet 4y ago
      "The Fine Article". Or you can use your imagination to replace 'Fine' with any other F-word. I think the acronym used to represent the phrase "The F*ing Article", and then, for a softer touch, people changed F to represent 'Fine'.
    - deleted 4y ago
      [deleted]
    - Espressosaurus 4y ago
      The Fucking Article, at least in the circles I run in.
    - centixel 4y ago
      TFA from RTFA
      - bigiain 4y ago
        > from RTFA which itself is from RTFM (manual), which was the catch-cry of early 90s irc and usenet tech newsgroups.
    - paulclinger 4y ago
      This freaking (or a variation) article; https://www.urbandictionary.com/define.php?term=TFA https://www.urbandictionary.com/define.php?term=TFA
    - thwaway03 4y ago
      It stands for The "Fine" Article. If you want a less polite version, replace the F with another word
    - slindz 4y ago
      A school teacher might explain it to be "The Frickin Article".
  - version_five 4y ago
    > Fetching the data over the network will be slower than fetching it from a local source. A big aspect of this is data collection. Microsoft office has tons of features that could easily be local buy that it wants to be "connected services" that are subject to their own TOS so they can steal your data. Based on the lag, it seems like all spell and grammar checks go through some server somewhere, almost certainly with the primary goal of scraping as much info as they can from what you're doing. Online first is all about data harvesting, much less so ease of operation and certainty not customer experience.
- deleted 4y ago
  [deleted]
- swyx 4y ago
  this works until you lose or break your "supercomputer in your posession".
  - szundi 4y ago
    the first in "Offline First" does not mean only
    - jleyank 4y ago
      Err, email copies, backups, USB sticks, save-to-sync, ... There are lots of shared/safe alternatives when saving my data/work. Not having to have to sweat net lag, though, is a big win. And devices are pretty damn reliable unless you drop the things. And even then, I have this phone thingie that has lots of storage - why can't I sync to that directly without 3rd-party involvement? This would address you "what if it crashed" by giving a second copy (or more if there's multiple people sitting near me).
- bsder 4y ago
  > Why not just let me use the supercomputer I have in my possession and run the thing without the baggage and overhead? Because the app would be too easy to steal and too hard to monetize. At this point, from an economic perspective, apps either need to be completely open source or running in the cloud behind secured servers. Even if I dismiss piracy as a problem (and I don't), the bigger issue is that if the app does something clever/technical and is somehow successful, someone will tear it apart, clone it, and put their own copy on the app stores. Running in the cloud is how companies are dodging that race to the bottom (see: Chitubox as a prime example--good or bad example is up to you to decide).
elwesties 4y ago
Shameless plug for the banking app that I work on https://www.youtube.com/watch?v=7AsM4gwmS5Y https://www.youtube.com/watch?v=7AsM4gwmS5Y Fully offline on both Android and iOS
- theylovezmw 4y ago
  That was a really interesting talk, thanks for sharing! Have y'all given other talks? would love to learn more about your dev/release workflow
  - elwesties 4y ago
    No problem :) Only other talk so far is a very similar talk from the iOS side available here - https://youtu.be/rvfVs_29MsI?t=4578 https://youtu.be/rvfVs_29MsI?t=4578. We do intend to talk about many more things in the near future. Ill try to remember to report back here when we do.
smeagull 4y ago
Apps get deleted the moment they stop working. I don't care why - alternatives exist.
knubie 4y ago
I've adopted this approach with my own app [0], and there have definitely been some challenges, but ultimately I'm glad I made the decision. A lot of the tech in the web world is focused around the traditional client <-> server / API model. Building a local first app feels like you're going against the grain in some ways and you're often not able to take advantage of the latest and greatest tech. In my case I am using PouchDB as a local database, which sync to an instance of CouchDB on the server. I think there is room for improvement with this combo, but for the most part they are a mature, battle tested combo that works really well as an offline first sync solution. [0] https://mochi.cards/ https://mochi.cards/
- divbzero 4y ago
  I’ve tried PouchDB before but only on “toy” projects not deployed production. What challenges have there been with PouchDB or other offline-first techniques?
  - locutous 4y ago
    I like offline first app design, and have built several. However, I don't do it anymore unless there is a very clear use case and direct customer benefit. Some apps just don't work offline at all. One of the decision trees is you often have some API needs and can't just live within the pouch db database. Particularly account creation and such. It may need to plug into other services and initiate them, you could use a database as a message queue, but it's awkward compared to an API. The couch/pouch models gets somewhat more complex when you leave the one db per user model. If there are shared data buckets, the database access patterns require a good deal of thought and experience with the couch db auth model. Particularly it's limitations. Offline access also requires thought around sensitive data. If a user has had their access revoked, an offline first app may preserve it which can rub against security requirements. Initial sync can be very slow if you're dealing with a lot of data. This can be a poor experience if you make a new user wait minutes to get started. Or you can also build a backend so they can start immediately and then switch to the local copy after replication. I've build this kind of system before. There are advantages to the route. Minimal data transfer from the server can represent lower load for your services. You can also do an offline only model for data, potentially lowering cloud costs for the number of users you serve - a definite boon for free tiers. The replication is pretty slick and quick, web socket like snappy data without web sockets. Using tools that do most of the heavy lifting for you and you mostly have to just plumb correctly.
rektide 4y ago
I'd really like to see more elaboration of different architectures used for offline. Having used Apollo's GraphQL recently, it definitely has a fairly robust toolkit for handling the cached data; it's a good mini-database-like thing. It feels more than a little like having a Spring repository[1], but in the front end. My hope is, as much as address user needs, we also as a positive side-effect get a nice well defined front-end architecture. The question of how programmers represent & handle data is fascinating, and offline-first is a more general capability we should have some story for when architecting these client-server resourceful systems. [1] https://www.digitalocean.com/community/tutorials/spring-repository-annotation https://www.digitalocean.com/community/tutorials/spring-repo...
- dragonshed 4y ago
  The only framework alternatives I’m aware of are PouchDB and BreezeJS. Most everything else I’ve seen seems to tack on offline support.
  - JamesSwift 4y ago
    There is also Realm
- JamesSwift 4y ago
  We need to differentiate between storage and sync is the problem. Sure, you might have something that supports the storage part, but sync is a domain/app specific problem. When you do offline first, you need to look at every endpoint and piece of data and make a decision about how it should be fetched/refreshed (and this is just considering the read-only usecase. read-write is a much harder problem). There are good resources online that outline the different fetch strategies (e.g. cache-first, cache-then-network, network-only) but it doesn't prevent the developer from needing to go through and make the decisions around each of those strategies and to implement them.
milofeynman 4y ago
I worked on a construction app that was offline first, and let me tell you the reason why people don't do it by default is how much more expensive and difficult it is. There isn't good tooling out there that just makes it work. So you have to do everything custom and carefully. And broken sync happens and conflicts happen.
api 4y ago
It’s sad that this is a question, at least for apps that don’t do inherently online things. Pretty soon the very concept of doing your own computation with your own data will be foreign to a lot of people.
Lev1a 4y ago
Because "having access to the internet" is not a foregone conclusion for many, MANY people/regions of the world. It's as simple as that.
- Scoundreller 4y ago
  Like Canada where people regularly blow through their 4 or 5gb that they’re paying $40+/month for.
- colordrops 4y ago
  Also, you know you can block network access to the app for privacy reasons and it will still work.
- karmicthreat 4y ago
  I design for offline because my apps might be used in places where internet might not be easily available. Like in the basement of a factory, with a hostile IT department. Lots of places all over the place where you might have equipment but can't get internet service.
theonemind 4y ago
I've found even apps that support downloading content for offline usage don't work properly offline. They do stupid things like sign you out randomly, refuse to work without a connection, put un-dismissable banners about not having connections over controls, or hang in random spots trying to navigate their UI. I'd settle for offline-second apps that, when ostensibly having the ability work offline, can work with any sort of designed or considered experience (dismiss banners about no connection, don't hang randomly, etc.)
- colordrops 4y ago
  Downloaded songs on Spotify stopped working for me if I'm offline. Why even bother downloading songs???
  - sojournerc 4y ago
    You are not alone. I live in the mountains. The primary reason for downloading is to have music in low/no connectivity. It seems worst when I have a bad connection, and it just can't deal, wanting to download the whole thing again. I swear, they don't ever test this stuff in anything but ideal California reception.
    - bin_bash 4y ago
      > ideal California reception you must not spend much time in California
      - deleted 4y ago
        [deleted]
      - getcrunk 4y ago
        Big facts. I don't leave the house much these days. Recently I went on a 30+ min drive to visit a friend. From a super dense urban area to a suburb that is growing into a dense area. And I legit had no internet for like 4 minutes. Obviously it'll be worse for non densely populated area
cgrealy 4y ago
Offline mode adds significant complexity to the service, since you are essentially replicating your entire server side stack (logic, storage, etc) on the client. Offline mode adds a cost to every new feature, even if it's just deciding if you are going to support said feature offline. If you're dealing with sensitive data, it also means you need a way to secure that data on the client device. I'm not saying apps shouldn't have an offline mode, but software engineering resource isn't free and most businesses would need to justify the cost of offline (even if it's just an opportunity cost).
- hsn915 4y ago
  Offline-first forces you to reconsider your design / architecture. It does not force you to replicate your functionality. What forces you to replicate your stack is poorly designed architecture.
  - Existenceblinks 4y ago
    He said "software engineering resource isn't free" so online-only may require simpler and cheaper solution. For business point of view, that architecture is considerably great, not poor.
- amelius 4y ago
  > Offline mode adds significant complexity to the service, since you are essentially replicating your entire server side stack (logic, storage, etc) on the client. Not necessarily true if you can run the same code on the client as you do on the server.
  - cgrealy 4y ago
    That’s doable, but it’s not without its own issues. Plus, you frequently don’t want to architect your client code the same as your server code (you’re not going to run a bunch of micro services on the client side for example)
- encryptluks2 4y ago
mbalex99 4y ago
My startup built an SDK framework that’s allows companies to build offline apps that can even sync over a multihop mesh network. It’s powered by a CRDT to handle the conflict resolution. From my experience most of our customers are not mobile developers (they’re JavaScript developers asked by their bosses to build mobile apps) and don’t even want to spend time learning coredata or room for iOS or Android. So most of the time they just stick to what they know, HTTP requests. Luckily we made our SDK easy to use so that these JavaScript developers can get both the network communication and the offline first caching in one product. They absolutely love it. Most apps aren’t offline first because it’s so hard to build the infrastructure to pull it off without a lot of bugs. Most apps focus entirely on the UI code layer. If infrastructure and frameworks made this a lot easier to use, I bet offline first would be a lot more popular and users would be a lot happier.
- waydegg 4y ago
  Are there similar services to your company's (https://www.ditto.live/ https://www.ditto.live/) out there already? I haven't had the need for this yet, but I can see your product being pretty damn useful
  - logical_proof 4y ago
    I am sure it is but the good old sign up and we’ll tell you how much this magic costs is an marketing pattern that I don’t trust. If you can’t be transparent about price I don’t feel confident in giving you my info.
    - nemothekid 4y ago
      If you know little about a potential client's use case, how do you price the product?
    - mbalex99 4y ago
      Regarding pricing, we are actively working on a general available pricing for our entire system. This is going to take us a couple of quarters to operationalize everything; we have a pretty good idea of what the pricing will be for the everyone-version. Right now we have our enterprise pricing nicely figured out. This might sound a bit strange but unlike many startups, we actually went to market backwards by focusing on the enterprise first. When we started the company, we had this crazy idea that we could build a distributed, query replicated, database that could sync over a mesh network. No one had ever built anything like this and we had some skepticism that such a product could actually generate revenue to create a sustainable business. Eventually, most infrastructure or database startups survive in large part to enterprise deployments. So we sought enterprise use cases and dollars first. And I have to say that was totally worth it and we've been rapidly gaining customers here. However, in our hearts, we are all frontend web, mobile, IoT app developers that all wanted to build collaborative apps like iMessage, Figma, Miro, Trello that had offline, sync, and conflict resolution capabilities out of the box. Once we make a couple of product and ops advancements over the next could of quarters, we will have very clear pricing like any other service out there. So just hang tight! Want us to get to pricing faster? Definitely recommend some of your infrastructure, kubernetes, and rust distributed systems developers to join us. We are hiring hardcore!
Multicomp 4y ago
Somehow we lost the understanding of offline-first, though back when I learned about it in the .NET Compact Framework days, it was just assumed - you had a Pocket PC with some cellular sure, but that was expensive and unreliable, and Wi-Fi wasn't a guarantee. Hopefully at some point you could get your Pocket PC docked in a cradle for Exchange ActiveSync, but otherwise it was almost Wi-Fi, IrDA, or bust. So yes, you have a small SQL Server Express cache of data and you build around syncing data and working with it offline.
- JamesSwift 4y ago
  I think the difference is that nowadays almost all data is shared. Offline-first was way easier to reason about when you had a single user to accommodate when handling sync and conflict resolution.
brigandish 4y ago
I played around with offline-first a short while ago and building it into an app was a pain. What are the any libraries or tools that would make this easier?
hax0rbana 4y ago
If you're like me and clicked to read this article because you thought it was going to have some suggestions about apps that actually implement these principles, I give you this as a consolation prize: https://secondwind.guardianproject.info/ https://secondwind.guardianproject.info/ BTW, I reported the fact that their "Add the Second Wind repo to your F-Droid app" link is broken several months ago, but feel free to report it to encourage them to get that AWS problem taken care of. <3
- woojoo666 4y ago
  Wow very nice, thanks for the share. Though I admit I first saw the janky url and was not expecting such a pleasant website UI. They really should get own domain
adepressedthrow 4y ago
It seems that there are few options overall, but particularly few OSS options for doing offline storage and sync in native code (_without_ using JavaScript). The only thing I've encountered that really met those requirements (OSS, non-JS) was Couchbase, but it's really not a great experience, and requires complicated setup both for the binary and the server. Adding P2P or CRDT features is even more difficulty.
deleted 4y ago
[deleted]
EugeneOZ 4y ago
Some apps only. Some apps are just useless without access to the server. App for creating diagrams based on the search request, for example - on the server side it will use a huge DB. Any application where “live” data is ultimately important. This feature has a cost and might limit the functionality - that’s why it should be implemented only when necessary.
deleted 4y ago
[deleted]
danielheath 4y ago
A few jobs ago we had an offline-first webapp. The offline support was so hard to avoid breaking that we ended up writing a proxy server for the test suite so we could make assertions about what the browser would do when it couldn't reach the site.
- klysm 4y ago
  What’s the proxy do? Just cut off the connection?
  - danielheath 4y ago
    In practice, yeah - we could have also simulated slow / unstable connections but never found those features needed the same level of automated testing.
LAC-Tech 4y ago
This completely glosses over the rather gritty realities of how to sync data between devices. It's not easy and IME most people do an awful job because they think it will be. The only thing approaching an off the shelf solution to this is the CouchDB replication protocol. While technically good, it suffers from a number of issues if you want to use it in practice: - dearth of cloud providers for CouchDB - There's pouchDB for the browser but there's nothing for native mobile clients - While PouchDB is great, it has to run on-top of browsers IndexedDB implementation, all of which suffer from reliability and capacity issues
- JamesSwift 4y ago
  RealmDB is the best one I've used in the past. I haven't used them since they got bought by Mongo however.
  - LAC-Tech 4y ago
    didn't they just use last write wins?
    - JamesSwift 4y ago
      You have to handle your own conflict resolution. Ive only used it client-side because our apis have been existing REST systems.
- lytefm 4y ago
  > dearth of cloud providers for CouchDB Yeah there's IBM Cloudant and that's about it :/ but a single node CouchDB isn't hard to operate. > There's pouchDB for the browser but there's nothing for native mobile clients There are native clients but they aren't maintained anymore by IBM. > While PouchDB is great, it has to run on-top of browsers IndexedDB implementation When building an iOS/Android hybrid app with web technology, SQLite can be used as storage engine for PouchDB.
egberts1 4y ago
Some apps should be made to work offline. one shining example is GPS guidance system.
Existenceblinks 4y ago
I wish all major browsers stop arguing about security and getting proper design to implement File System Access API (including writable part).
glroyal 4y ago
JamesSwift 4y ago
Offline first isnt more prevalent because: 1. its hard 2. its a significant dev cost 3. there aren't a lot of resources to help you out if you aren't already familiar with CRDTs and the like (and that assumes your backend also already supports them) I can't help with #1 and #2, but for #3 I found Jake Archibald to be the best source of information online. Here is a great high level presentation (he really is a gem of a presenter. I always drop what I'm doing to watch anything he puts out, offline-first or otherwise): https://www.youtube.com/watch?v=cmGr0RszHc8 https://www.youtube.com/watch?v=cmGr0RszHc8 Also, Trello put out a multipart blog series which gets into some nuts and bolts which are really valuable when designing your own app: https://tech.trello.com/sync-architecture/ https://tech.trello.com/sync-architecture/ EDIT: I just noticed Jake has a full course available. I'm not sure how good it is, or how indepth, but I'm sure its decent based on the author: https://jakearchibald.com/2014/offline-cookbook/ https://jakearchibald.com/2014/offline-cookbook/
- lytefm 4y ago
  > 1. its hard From my experience, building an offline-first app is much easier than a traditional client-server app. I maintain an authentication backend for PouchDB/CouchDB-based apps and have received this feedback multiple times: It only requires a little bit of frontend web dev knowledge to store and retrieve JSONs locally + sync them up, without having to worry about authentication, API design, network connection or caching. > 2. its a significant dev cost It allows to move much faster than a traditional setup where frontend devs might need to sit in meetings with backend devs in order to argue about some API design. But early mistakes in the data model are much harder to fix because with CouchDB, your data model is your API. Having built apps with Pouch/Couch for > 4 years now, I think the perfect middle ground is to only accept changes made by a user through an API (proper data validation, no conflicts,...) and to use Pouch/Couch in a read-only manner, replicating all necessary user data to the client.

121 points by 8bitsrule 4y ago

Why must I have three separate must-work activities (your front-end, my network, your server)? Why not just let me use the supercomputer I have in my possession and run the thing without the baggage and overhead?

tested23 4y ago
People expect apps to work across all their devices
- szundi 4y ago
  you can always sync, that's good practice with "Offline First"
dfabulich 4y ago
That's what TFA is about. > 1. Users are offline; experiencing latency issues or are in unreliable network conditions. > 2. Fetching the data over the network will be slower than fetching it from a local source. > 3. The app users should be informed about the low network conditions but it shouldn’t be a hindrance to their objective. > 4. Users’ network and battery status are taken into account, and thus only the data that has changed since the last synchronization should be synced.
- swyx 4y ago
  what is TFA? not a familiar acronym in this context (usually it means The Force Awakens)
  - gurjeet 4y ago
    "The Fine Article". Or you can use your imagination to replace 'Fine' with any other F-word. I think the acronym used to represent the phrase "The F*ing Article", and then, for a softer touch, people changed F to represent 'Fine'.
  - deleted 4y ago
    [deleted]
  - Espressosaurus 4y ago
    The Fucking Article, at least in the circles I run in.
  - centixel 4y ago
    TFA from RTFA
    - bigiain 4y ago
      > from RTFA which itself is from RTFM (manual), which was the catch-cry of early 90s irc and usenet tech newsgroups.
  - paulclinger 4y ago
    This freaking (or a variation) article; https://www.urbandictionary.com/define.php?term=TFA https://www.urbandictionary.com/define.php?term=TFA
  - thwaway03 4y ago
    It stands for The "Fine" Article. If you want a less polite version, replace the F with another word
  - slindz 4y ago
    A school teacher might explain it to be "The Frickin Article".
- version_five 4y ago
  > Fetching the data over the network will be slower than fetching it from a local source. A big aspect of this is data collection. Microsoft office has tons of features that could easily be local buy that it wants to be "connected services" that are subject to their own TOS so they can steal your data. Based on the lag, it seems like all spell and grammar checks go through some server somewhere, almost certainly with the primary goal of scraping as much info as they can from what you're doing. Online first is all about data harvesting, much less so ease of operation and certainty not customer experience.
deleted 4y ago
[deleted]
swyx 4y ago
this works until you lose or break your "supercomputer in your posession".
- szundi 4y ago
  the first in "Offline First" does not mean only
  - jleyank 4y ago
    Err, email copies, backups, USB sticks, save-to-sync, ... There are lots of shared/safe alternatives when saving my data/work. Not having to have to sweat net lag, though, is a big win. And devices are pretty damn reliable unless you drop the things. And even then, I have this phone thingie that has lots of storage - why can't I sync to that directly without 3rd-party involvement? This would address you "what if it crashed" by giving a second copy (or more if there's multiple people sitting near me).
bsder 4y ago
> Why not just let me use the supercomputer I have in my possession and run the thing without the baggage and overhead? Because the app would be too easy to steal and too hard to monetize. At this point, from an economic perspective, apps either need to be completely open source or running in the cloud behind secured servers. Even if I dismiss piracy as a problem (and I don't), the bigger issue is that if the app does something clever/technical and is somehow successful, someone will tear it apart, clone it, and put their own copy on the app stores. Running in the cloud is how companies are dodging that race to the bottom (see: Chitubox as a prime example--good or bad example is up to you to decide).

elwesties 4y ago

Shameless plug for the banking app that I work on https://www.youtube.com/watch?v=7AsM4gwmS5Y https://www.youtube.com/watch?v=7AsM4gwmS5Y Fully offline on both Android and iOS

theylovezmw 4y ago
That was a really interesting talk, thanks for sharing! Have y'all given other talks? would love to learn more about your dev/release workflow
- elwesties 4y ago
  No problem :) Only other talk so far is a very similar talk from the iOS side available here - https://youtu.be/rvfVs_29MsI?t=4578 https://youtu.be/rvfVs_29MsI?t=4578. We do intend to talk about many more things in the near future. Ill try to remember to report back here when we do.

smeagull 4y ago

Apps get deleted the moment they stop working. I don't care why - alternatives exist.

knubie 4y ago

I've adopted this approach with my own app [0], and there have definitely been some challenges, but ultimately I'm glad I made the decision. A lot of the tech in the web world is focused around the traditional client <-> server / API model. Building a local first app feels like you're going against the grain in some ways and you're often not able to take advantage of the latest and greatest tech. In my case I am using PouchDB as a local database, which sync to an instance of CouchDB on the server. I think there is room for improvement with this combo, but for the most part they are a mature, battle tested combo that works really well as an offline first sync solution. [0] https://mochi.cards/ https://mochi.cards/

divbzero 4y ago
I’ve tried PouchDB before but only on “toy” projects not deployed production. What challenges have there been with PouchDB or other offline-first techniques?
- locutous 4y ago
  I like offline first app design, and have built several. However, I don't do it anymore unless there is a very clear use case and direct customer benefit. Some apps just don't work offline at all. One of the decision trees is you often have some API needs and can't just live within the pouch db database. Particularly account creation and such. It may need to plug into other services and initiate them, you could use a database as a message queue, but it's awkward compared to an API. The couch/pouch models gets somewhat more complex when you leave the one db per user model. If there are shared data buckets, the database access patterns require a good deal of thought and experience with the couch db auth model. Particularly it's limitations. Offline access also requires thought around sensitive data. If a user has had their access revoked, an offline first app may preserve it which can rub against security requirements. Initial sync can be very slow if you're dealing with a lot of data. This can be a poor experience if you make a new user wait minutes to get started. Or you can also build a backend so they can start immediately and then switch to the local copy after replication. I've build this kind of system before. There are advantages to the route. Minimal data transfer from the server can represent lower load for your services. You can also do an offline only model for data, potentially lowering cloud costs for the number of users you serve - a definite boon for free tiers. The replication is pretty slick and quick, web socket like snappy data without web sockets. Using tools that do most of the heavy lifting for you and you mostly have to just plumb correctly.

rektide 4y ago

I'd really like to see more elaboration of different architectures used for offline. Having used Apollo's GraphQL recently, it definitely has a fairly robust toolkit for handling the cached data; it's a good mini-database-like thing. It feels more than a little like having a Spring repository[1], but in the front end. My hope is, as much as address user needs, we also as a positive side-effect get a nice well defined front-end architecture. The question of how programmers represent & handle data is fascinating, and offline-first is a more general capability we should have some story for when architecting these client-server resourceful systems. [1] https://www.digitalocean.com/community/tutorials/spring-repository-annotation https://www.digitalocean.com/community/tutorials/spring-repo...

dragonshed 4y ago
The only framework alternatives I’m aware of are PouchDB and BreezeJS. Most everything else I’ve seen seems to tack on offline support.
- JamesSwift 4y ago
  There is also Realm
JamesSwift 4y ago
We need to differentiate between storage and sync is the problem. Sure, you might have something that supports the storage part, but sync is a domain/app specific problem. When you do offline first, you need to look at every endpoint and piece of data and make a decision about how it should be fetched/refreshed (and this is just considering the read-only usecase. read-write is a much harder problem). There are good resources online that outline the different fetch strategies (e.g. cache-first, cache-then-network, network-only) but it doesn't prevent the developer from needing to go through and make the decisions around each of those strategies and to implement them.

milofeynman 4y ago

I worked on a construction app that was offline first, and let me tell you the reason why people don't do it by default is how much more expensive and difficult it is. There isn't good tooling out there that just makes it work. So you have to do everything custom and carefully. And broken sync happens and conflicts happen.

api 4y ago

It’s sad that this is a question, at least for apps that don’t do inherently online things. Pretty soon the very concept of doing your own computation with your own data will be foreign to a lot of people.

Lev1a 4y ago

Because "having access to the internet" is not a foregone conclusion for many, MANY people/regions of the world. It's as simple as that.

Scoundreller 4y ago
Like Canada where people regularly blow through their 4 or 5gb that they’re paying $40+/month for.
colordrops 4y ago
Also, you know you can block network access to the app for privacy reasons and it will still work.
karmicthreat 4y ago
I design for offline because my apps might be used in places where internet might not be easily available. Like in the basement of a factory, with a hostile IT department. Lots of places all over the place where you might have equipment but can't get internet service.

theonemind 4y ago

I've found even apps that support downloading content for offline usage don't work properly offline. They do stupid things like sign you out randomly, refuse to work without a connection, put un-dismissable banners about not having connections over controls, or hang in random spots trying to navigate their UI. I'd settle for offline-second apps that, when ostensibly having the ability work offline, can work with any sort of designed or considered experience (dismiss banners about no connection, don't hang randomly, etc.)

colordrops 4y ago
Downloaded songs on Spotify stopped working for me if I'm offline. Why even bother downloading songs???
- sojournerc 4y ago
  You are not alone. I live in the mountains. The primary reason for downloading is to have music in low/no connectivity. It seems worst when I have a bad connection, and it just can't deal, wanting to download the whole thing again. I swear, they don't ever test this stuff in anything but ideal California reception.
  - bin_bash 4y ago
    > ideal California reception you must not spend much time in California
    - deleted 4y ago
      [deleted]
    - getcrunk 4y ago
      Big facts. I don't leave the house much these days. Recently I went on a 30+ min drive to visit a friend. From a super dense urban area to a suburb that is growing into a dense area. And I legit had no internet for like 4 minutes. Obviously it'll be worse for non densely populated area

cgrealy 4y ago

Offline mode adds significant complexity to the service, since you are essentially replicating your entire server side stack (logic, storage, etc) on the client. Offline mode adds a cost to every new feature, even if it's just deciding if you are going to support said feature offline. If you're dealing with sensitive data, it also means you need a way to secure that data on the client device. I'm not saying apps shouldn't have an offline mode, but software engineering resource isn't free and most businesses would need to justify the cost of offline (even if it's just an opportunity cost).

hsn915 4y ago
Offline-first forces you to reconsider your design / architecture. It does not force you to replicate your functionality. What forces you to replicate your stack is poorly designed architecture.
- Existenceblinks 4y ago
  He said "software engineering resource isn't free" so online-only may require simpler and cheaper solution. For business point of view, that architecture is considerably great, not poor.
amelius 4y ago
> Offline mode adds significant complexity to the service, since you are essentially replicating your entire server side stack (logic, storage, etc) on the client. Not necessarily true if you can run the same code on the client as you do on the server.
- cgrealy 4y ago
  That’s doable, but it’s not without its own issues. Plus, you frequently don’t want to architect your client code the same as your server code (you’re not going to run a bunch of micro services on the client side for example)
encryptluks2 4y ago

mbalex99 4y ago

My startup built an SDK framework that’s allows companies to build offline apps that can even sync over a multihop mesh network. It’s powered by a CRDT to handle the conflict resolution. From my experience most of our customers are not mobile developers (they’re JavaScript developers asked by their bosses to build mobile apps) and don’t even want to spend time learning coredata or room for iOS or Android. So most of the time they just stick to what they know, HTTP requests. Luckily we made our SDK easy to use so that these JavaScript developers can get both the network communication and the offline first caching in one product. They absolutely love it. Most apps aren’t offline first because it’s so hard to build the infrastructure to pull it off without a lot of bugs. Most apps focus entirely on the UI code layer. If infrastructure and frameworks made this a lot easier to use, I bet offline first would be a lot more popular and users would be a lot happier.

waydegg 4y ago
Are there similar services to your company's (https://www.ditto.live/ https://www.ditto.live/) out there already? I haven't had the need for this yet, but I can see your product being pretty damn useful
- logical_proof 4y ago
  I am sure it is but the good old sign up and we’ll tell you how much this magic costs is an marketing pattern that I don’t trust. If you can’t be transparent about price I don’t feel confident in giving you my info.
  - nemothekid 4y ago
    If you know little about a potential client's use case, how do you price the product?
  - mbalex99 4y ago
    Regarding pricing, we are actively working on a general available pricing for our entire system. This is going to take us a couple of quarters to operationalize everything; we have a pretty good idea of what the pricing will be for the everyone-version. Right now we have our enterprise pricing nicely figured out. This might sound a bit strange but unlike many startups, we actually went to market backwards by focusing on the enterprise first. When we started the company, we had this crazy idea that we could build a distributed, query replicated, database that could sync over a mesh network. No one had ever built anything like this and we had some skepticism that such a product could actually generate revenue to create a sustainable business. Eventually, most infrastructure or database startups survive in large part to enterprise deployments. So we sought enterprise use cases and dollars first. And I have to say that was totally worth it and we've been rapidly gaining customers here. However, in our hearts, we are all frontend web, mobile, IoT app developers that all wanted to build collaborative apps like iMessage, Figma, Miro, Trello that had offline, sync, and conflict resolution capabilities out of the box. Once we make a couple of product and ops advancements over the next could of quarters, we will have very clear pricing like any other service out there. So just hang tight! Want us to get to pricing faster? Definitely recommend some of your infrastructure, kubernetes, and rust distributed systems developers to join us. We are hiring hardcore!

Multicomp 4y ago

Somehow we lost the understanding of offline-first, though back when I learned about it in the .NET Compact Framework days, it was just assumed - you had a Pocket PC with some cellular sure, but that was expensive and unreliable, and Wi-Fi wasn't a guarantee. Hopefully at some point you could get your Pocket PC docked in a cradle for Exchange ActiveSync, but otherwise it was almost Wi-Fi, IrDA, or bust. So yes, you have a small SQL Server Express cache of data and you build around syncing data and working with it offline.

JamesSwift 4y ago
I think the difference is that nowadays almost all data is shared. Offline-first was way easier to reason about when you had a single user to accommodate when handling sync and conflict resolution.

brigandish 4y ago

I played around with offline-first a short while ago and building it into an app was a pain. What are the any libraries or tools that would make this easier?

hax0rbana 4y ago

If you're like me and clicked to read this article because you thought it was going to have some suggestions about apps that actually implement these principles, I give you this as a consolation prize: https://secondwind.guardianproject.info/ https://secondwind.guardianproject.info/ BTW, I reported the fact that their "Add the Second Wind repo to your F-Droid app" link is broken several months ago, but feel free to report it to encourage them to get that AWS problem taken care of. <3

woojoo666 4y ago
Wow very nice, thanks for the share. Though I admit I first saw the janky url and was not expecting such a pleasant website UI. They really should get own domain

adepressedthrow 4y ago

It seems that there are few options overall, but particularly few OSS options for doing offline storage and sync in native code (_without_ using JavaScript). The only thing I've encountered that really met those requirements (OSS, non-JS) was Couchbase, but it's really not a great experience, and requires complicated setup both for the binary and the server. Adding P2P or CRDT features is even more difficulty.

deleted 4y ago

[deleted]

EugeneOZ 4y ago

Some apps only. Some apps are just useless without access to the server. App for creating diagrams based on the search request, for example - on the server side it will use a huge DB. Any application where “live” data is ultimately important. This feature has a cost and might limit the functionality - that’s why it should be implemented only when necessary.

deleted 4y ago

[deleted]

danielheath 4y ago

A few jobs ago we had an offline-first webapp. The offline support was so hard to avoid breaking that we ended up writing a proxy server for the test suite so we could make assertions about what the browser would do when it couldn't reach the site.

klysm 4y ago
What’s the proxy do? Just cut off the connection?
- danielheath 4y ago
  In practice, yeah - we could have also simulated slow / unstable connections but never found those features needed the same level of automated testing.

LAC-Tech 4y ago

This completely glosses over the rather gritty realities of how to sync data between devices. It's not easy and IME most people do an awful job because they think it will be. The only thing approaching an off the shelf solution to this is the CouchDB replication protocol. While technically good, it suffers from a number of issues if you want to use it in practice: - dearth of cloud providers for CouchDB - There's pouchDB for the browser but there's nothing for native mobile clients - While PouchDB is great, it has to run on-top of browsers IndexedDB implementation, all of which suffer from reliability and capacity issues

JamesSwift 4y ago
RealmDB is the best one I've used in the past. I haven't used them since they got bought by Mongo however.
- LAC-Tech 4y ago
  didn't they just use last write wins?
  - JamesSwift 4y ago
    You have to handle your own conflict resolution. Ive only used it client-side because our apis have been existing REST systems.
lytefm 4y ago
> dearth of cloud providers for CouchDB Yeah there's IBM Cloudant and that's about it :/ but a single node CouchDB isn't hard to operate. > There's pouchDB for the browser but there's nothing for native mobile clients There are native clients but they aren't maintained anymore by IBM. > While PouchDB is great, it has to run on-top of browsers IndexedDB implementation When building an iOS/Android hybrid app with web technology, SQLite can be used as storage engine for PouchDB.

egberts1 4y ago

Some apps should be made to work offline. one shining example is GPS guidance system.

Existenceblinks 4y ago

I wish all major browsers stop arguing about security and getting proper design to implement File System Access API (including writable part).

glroyal 4y ago

JamesSwift 4y ago

Offline first isnt more prevalent because: 1. its hard 2. its a significant dev cost 3. there aren't a lot of resources to help you out if you aren't already familiar with CRDTs and the like (and that assumes your backend also already supports them) I can't help with #1 and #2, but for #3 I found Jake Archibald to be the best source of information online. Here is a great high level presentation (he really is a gem of a presenter. I always drop what I'm doing to watch anything he puts out, offline-first or otherwise): https://www.youtube.com/watch?v=cmGr0RszHc8 https://www.youtube.com/watch?v=cmGr0RszHc8 Also, Trello put out a multipart blog series which gets into some nuts and bolts which are really valuable when designing your own app: https://tech.trello.com/sync-architecture/ https://tech.trello.com/sync-architecture/ EDIT: I just noticed Jake has a full course available. I'm not sure how good it is, or how indepth, but I'm sure its decent based on the author: https://jakearchibald.com/2014/offline-cookbook/ https://jakearchibald.com/2014/offline-cookbook/

lytefm 4y ago
> 1. its hard From my experience, building an offline-first app is much easier than a traditional client-server app. I maintain an authentication backend for PouchDB/CouchDB-based apps and have received this feedback multiple times: It only requires a little bit of frontend web dev knowledge to store and retrieve JSONs locally + sync them up, without having to worry about authentication, API design, network connection or caching. > 2. its a significant dev cost It allows to move much faster than a traditional setup where frontend devs might need to sit in meetings with backend devs in order to argue about some API design. But early mistakes in the data model are much harder to fix because with CouchDB, your data model is your API. Having built apps with Pouch/Couch for > 4 years now, I think the perfect middle ground is to only accept changes made by a user through an API (proper data validation, no conflicts,...) and to use Pouch/Couch in a read-only manner, replicating all necessary user data to the client.