17 ms·

Local-first software: you own your data, in spite of the cloud

748 points by godelmachine 7y ago

ricg 7y ago
"It should support data access for all time." - This is key for me after I had to convert my notes more than once between formats after the original app(s) went into extinction (beloved Circus Ponies Notebook). That's why I'm designing any new apps around a file format that can be accessed even without the app. I have a "local-first" Kanban/Trello-style app, "Boards" (http://kitestack.com/boards/ http://kitestack.com/boards/), that uses zipped HTML files (to support rich text with images). No collaboration and cross-device support just yet, but it works without a network and saves everything locally.
- threatripper 7y ago
  Boards seems to be Mac only. "A new Mac app to boost your productivity in school, at work, and for personal projects.", "For macOS 10.14 Mojave and later"
  - ricg 7y ago
    Yes, currently Mac-only, no cross-device support yet, but an iOS app is in planning.
- jonpurdy 7y ago
  I just tried Boards out. It's really nice and exactly what I wish Trello could be for a single user. Will follow the progress on this for sure.
  - ricg 7y ago
    Thanks for your feedback! I don't have a newsletter signup just yet, but if you'd like to hear about new versions, send me a quick email through the app (Help → Contact Support) and I'll add you to the list.
rado 7y ago
Common sense.
- ptx 7y ago
  You might think so from only reading the title, but what the article/paper actually proposes is using CRDTs to create a synthesis of the advantages (for the user) of both cloud applications and traditional local applications.
quickthrower2 7y ago
My side project is a local first (local storage on web) JAMStack. For extra goodness it’s mobile first too. I really love making apps this way for some reason. I think it’s the focus on just the UI and not worrying about the back end until later. For this particular app I’d consider “smartwatch first” to have been better as its for fitness!
- randomsearch 7y ago
  This is a really good way to prototype and find the minimum feature set for a SaaS startup.
TAForObvReasons 7y ago
Today's SaaS world is largely economically opposed to the idea of data ownership. It's a lot easier to make money by renting people access to their data. The problem is not inherently technical. The solution must address the fact that the software businesses favor cloud solutions and other systems that make it difficult to stop spending money
- quickthrower2 7y ago
  Yes. I always think data freedom is more important than software freedom. For example it matters less that MS Word is not free as in freedom when you can open the file in something that is.
  - TeMPOraL 7y ago
    And cloud SaaS are enemies of both! After all, in order to have data freedom, you need to have something to open in that other program, and cloud solutions do their best to not give you proper open/save file (and even if there's an export function, and even if everything is actually included in the export, it often isn't followed by import function that could read that export). For this reason, I avoid using cloud SaaS for anything where can avoid it.
    - quickthrower2 7y ago
      Absolutely. There are some exceptions though. Github is the obvious one. Dropbox by nature has your data constantly exported. Google for its sins has Google takeout.
      - TeMPOraL 7y ago
        Yes. And the first two I use, treating them mostly as dumb pieces of infrastructure. Arguably, the functionality they provide have a crucial ops component that I'm all for paying for someone else to handle it for me. But neither Github nor Dropbox locks me into anything. Google - yes, web e-mail obviously is similar to the above; as for their office suite, I recently found a good excuse to justify shelling out for a proper Microsoft Office subscription (though I don't like that it's a subscription), and I stick to using the faster, locally-available, file-using, much more powerful (if still proprietary) software.
mwilcox 7y ago
any marginally successful "local-first" app is going to go and raise $10m in vc, switch to software as a service, and add an enterprise mode that requires user permissions and data access to be managed on the server
- LeftHandPath 7y ago
  Sure, any one company probably will - but there’s a whole market. As soon as that one company abandons the local-first model, a gap opens, which will (usually, eventually) be filled by a new company offering local-first until that new company does the same. As long as the companies don’t band together and agree to end it, there should be a company offering that model somewhere somehow.
  - mwilcox 7y ago
    Even in the situation I described, the original company would leave the free / client side version on the site as free marketing. This is the standard today for enterprise-monetised open source software. Solving this problem isn't about being local-first. It's about being local-last. You have to be able to make more money by selling a software license than you make by selling equity and chasing user acquisition and retention. Then we'll see people waking up to the fact that all this proprietary data is a liability and subscriptions are golden handcuffs and people will finally get back to making real software again
- rapsey 7y ago
  Because local-first is not a viable business model compared to the cloud. Software goes where the money is.
- toomim 7y ago
  That's why we need to add local-first features into HTTP itself. I've been doing some work on it over the last few days: https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-braid-http-01 https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-b...
- 0xCMP 7y ago
gobengo 7y ago
The blog post doesn't mention this, so I thought I'd point this out. One of the paper's authors is Martin Kleppmann, who wrote the very good https://dataintensive.net/ https://dataintensive.net/ book.
- ignoramous 7y ago
  Kleppmann co-created/major contributor to Apache Kafka along with Jay Kreps and Neha Narkhede. He also co-founded Rapportive, a YC company acquired by LinkedIn, along with Rahul Vohra, who is presently the CEO of Superhuman.
- atrilumen 7y ago
  And don't miss "Turning the database inside out with Apache Samza" <3 https://www.youtube.com/watch?v=fU9hR3kiOK0 https://www.youtube.com/watch?v=fU9hR3kiOK0
theamk 7y ago
It’s strange that Evernote is omitted from the list - it is a great example of local-first app. Their recent-ish history, when the their free tire become limited to only syncing a few devices, illustrates that even if software is fully local, and supports open formats, having the functional cloud matters, a lot.
- pqs 7y ago
  I love Evernote and I use it daily, but Evernote lacks end-to-end encryption, which is a pity. I store lots of information in that app and I would be more reassured if I knew that it is only me who can read the data. I would even pay more in order to have that kind of encryption. I think that no current feature of Evernote would be affected by encryption as text recognition in images can be done at the client level.
- thundergolfer 7y ago
  Evernote is included in one of their “what not to do” images, showing that they can’t handle conflict resolution at all. But yeah otherwise Evernote is pretty good for a single user.
deleted 7y ago
[deleted]
toolslive 7y ago
My company built this more than 10 years ago, and nobody was interested. Here's a press release from 2007: https://www.macworld.com/article/1058169/mcnucle.html https://www.macworld.com/article/1058169/mcnucle.html
- toolslive 7y ago
  found a demo movie too: https://www.youtube.com/watch?v=bvzfnAY0XiU https://www.youtube.com/watch?v=bvzfnAY0XiU
- fastball 7y ago
  Might have something to do with the fact that McNucle is a freaking terrible name.
  - toolslive 7y ago
    no, most end users did not understand (or cared about) the data ownership problem back then. Times have changed.
    - procombo 7y ago
      That name though...
      - mackrevinack 7y ago
        lol
jmathai 7y ago
I've been trying to document my "local-first" approach to managing photos. I've made it a ways through but am not sure when I'll finish. Posting here since it is relevant. A Pragmatic Photo Archiving Solution: https://docs.google.com/document/d/1JzqT-DJFlS2e8ZC00HrsQITq_ukORcH9icXTGKeWQYU/edit?usp=sharing https://docs.google.com/document/d/1JzqT-DJFlS2e8ZC00HrsQITq... It's the culmination of software I've written [1] + a workflow that's resulted from it [2, 3, 4, 5]. [1] Elodie - https://github.com/jmathai/elodie https://github.com/jmathai/elodie [2] Understanding my need for an automated photo workflow - https://medium.com/vantage/understanding-my-need-for-an-automated-photo-workflow-a2ff95b46f8f https://medium.com/vantage/understanding-my-need-for-an-auto... [3] Introducing Elodie; Your Personal EXIF-based Photo and Video Assistant - https://medium.com/vantage/understanding-my-need-for-an-automated-photo-workflow-a2ff95b46f8f https://medium.com/vantage/understanding-my-need-for-an-auto... [4] My Automated Photo Workflow using Google Photos and Elodie - https://medium.com/@jmathai/my-automated-photo-workflow-using-google-photos-and-elodie-afb753b8c724 https://medium.com/@jmathai/my-automated-photo-workflow-usin... [5] One Year of Using an Automated Photo Organization and Archiving Workflow - https://artplusmarketing.com/one-year-of-using-an-automated-photo-organization-and-archiving-workflow-89cf9ad7bddf https://artplusmarketing.com/one-year-of-using-an-automated-...
- geolgau 7y ago
  This looks great and feels a lot like beets [1] for music, only that they use a database. I'll try it when I have the time to re-organize ~20 years of photos. [1] http://beets.io/ http://beets.io/
- emilburzo 7y ago
  elodie looks amazing! I definitely need to try it out it was just the thing I was thinking of building recently as it was getting really tiring to manually organize photos one extra idea that I had: there's a cool project that would enable offline geocoding[1], which would help get rid of API limits while making the reverse geocode queries almost instant (the included dataset is pretty limited, but it's not hard to extend from an openstreetmap planet dump) [1] https://pypi.org/project/reverse_geocoder/ https://pypi.org/project/reverse_geocoder/
gosubpl 7y ago
Things are happening in that space already! I work at Actyx and we have a production ready stack for local-first real serverless (peer-to-peer) applications. Please take a look at https://actyx.com/ https://actyx.com/ .
throwGuardian 7y ago
P2P has miles to go before challenging the reliability, convenience and performance of the cloud. That said, one area undervalued is Partially homomorphic cryptosystems[1], where the cloud never ever gets to see unencrypted user data. I hope the future is fast-local compute on cached data, with the cloud holding a much larger, encrypted but permissioned data store, offering utility functions like search over encrypted data [1]: https://en.m.wikipedia.org/wiki/Homomorphic_encryption https://en.m.wikipedia.org/wiki/Homomorphic_encryption
mbalex99 7y ago
Martin Kleppmann is a major inspiration for our startup, Ditto. We take the local-first concept and p2p to the next level with CRDTs and replication. But what we really do is leverage things like AWDL, mDNS, and or Bluetooth low energy to sync mobile databases instances with each other even without internet connectivity. www.ditto.live Check it out in action! https://youtu.be/1P2bKEJjdec https://youtu.be/1P2bKEJjdec https://youtu.be/ITUrk_rjnvo https://youtu.be/ITUrk_rjnvo We found that CRDTs, local first, and eventual consistency REALLY shines in the mobile phones since they constantly experience network partitions.
- Jyaif 7y ago
  Very interesting. Will the upcoming server support be end-to-end encrypted? In other words, will the server be able to read the data?
- mamcx 7y ago
  I made "traditional" enterprise-style-apps for small business and have tried to crack the sync of data several times. Exist a resource in how leverage that tech with boring stuff like inventory, invoices, etc? Hopefully,. without a total change of stacks (I use postgresql, sqlite as dbs, and need to integrate with near 12+ different db engines)
ranjithdsm 7y ago
The data ownership is very important for business world. The reason why we built our products for IoT ( www.bevywise.com ) as a more install able version is the data privacy and ownership. We really see this largely in manufacturing industry. If not local, we should provide private servers and data security.
saurabhnanda 7y ago
Not related to the actual content, but may I please request you to increase the contrast on your website. Grey on grey is very hard to read.
- josteink 7y ago
  Firefox reader mode. Instantly fixes any crappy design on any site. Learn to love it :)
radium3d 7y ago
https://github.com/syncthing/syncthing https://github.com/syncthing/syncthing Syncthing solves a large part of syncing data between devices using your own VPS, server(s), etc. If your VPS provider goes out of business, you can then just fire up a new VPS and hook it back up to your local machine(s).
- josteink 7y ago
  Sadly Syncting still lacks iOS-support. As a result I'm running Nextcloud instead.
  - geolgau 7y ago
    I'm on Android so it works, but I also used Nextcloud for some time and had issues with it "forgetting" about the data folder, so I had to re-create it every time it happened. I wasn't happy about that. With Syncthing I don't (yet) have this problem.
- mackrevinack 7y ago
  there's also resilio sync that gives you the option have the node/folder on your vps be encrypted. hopefully syncthing will add that feature at some point in the future
  - Multicomp 7y ago
    I have been considering spending more money than I should finding someone on a freelancer site to add the untrusted node / encrypted file transfer but don't give a certain node the key to decrypt the file support, that is the number one missing feature for syncthing for me. It would unlock scenarios like share your files with me and I'll share files with you and neither of us will know the others files unless your house burns down in which case you can get your files back just by hookin the sync back up with the encryption key pulled from your password manager.
    - mappu 7y ago
      You can kind of work around this by sharing an ecryptfs folder.
jsfenfen 7y ago
The paper says pinterest meets the "collaboration" ideal but github doesn't. I'm sympathetic to the idea that nothing meets the ideal, but c'mon.
sans-serif 7y ago
iOS apps like Bear achieve this by building on iCloud storage
atoav 7y ago
When I select software these are among the list of things I am looking for generally: - file formats that won’t lock you in or are even openly hackable (allows you to automate things) - no clouds that will break the software once it is gone - local storage with custom syncing or backup options - strictly no weird data collection or “We own the rights to your data”-Type of terms So if I get the slightest feeling of a lock in or unnecessary data collection you are scaring me away, because mentally I would then already look at the time after you decide to scrap your cloud or abandon your file formats. The data collection bit shows me your users aren’t front and center but something else is which makes your product even less of a good choice. If your product runs on the web, allowing for self-hosted solutions is also a big plus.
- jwr 7y ago
  While I fully agree with your selection criteria, please consider the other side of the equation, because engineering (and the world) is all about compromises. I am the author of a SaaS app (https://partsbox.io/ https://partsbox.io/). I export in open formats (JSON), there is no lock-in, it's easy to get all of your data at any time. But the app is online and will remain so. Why? Economics. Maintaining a self-hosted solution is an enormous cost, which most people forget about. You need to create, document, maintain and support an entirely different version of your software (single-user, no billing/invoicing, different email sending, different rights system, etc). And then every time you change something in your software you have to worry about migrations, not just in your database, but in all your clients databases. I am not saying it's impossible, it's just expensive, especially for companies which are built to be sustainable in the first place (e.g. not VC-funded). Believe me, if you don't have VC money to burn, you will not be experimenting with CRDTs and synchronizing distributed data from a multitude of versions of your application. I regularly have to explain why there is no on-premises version of my app. The best part is that many people think that an on-premises version should be less expensive than the online version, and come without a subscription.
  - rgun 7y ago
    > The best part is that many people think that an on-premises version should be less expensive than the online version, and come without a subscription. Which makes sense IMHO, provided they are not expecting any updates to their on-prem installation. It can just be a fork of your current codebase with no new features or warranty. Maybe you can include some terms for critical security updates but that is about it.
miki123211 7y ago
Resilio sync is a perfect example of such app. It's basically a P2P based Dropbox with no accounts, full end-to-end encryption and no folder size limits. It's not open source, but it can work without a central server if you need it to. It's also amazingly simple to set up, much simpler than sync thing.
- mackrevinack 7y ago
  i used resilio for years and thought the same about syncthing until i switched last year and now I actually prefer how syncthing is setup. if you pay for resilio there is an option to add all your folders in one go but on some computers I don't want to add all of them anyway so that's not much use to me. with the free version you have to manually add folders one by one but to do that you need the key which means you need to copy them to a text file and add them on another computer. with syncthing, it will detect other syncthing devices on your home network so you just have to add the ones you want then accept the request from another device. once that is done you select which folders the device has access to and then a notification will show up on said device asking you to connect. so basically no fiddling with keys or having to store them somewhere secure (this is all presuming i was using resilio correctly, maybe there was an easier way I was not aware of)
chuhnk 7y ago
Interesting to see this. We've taken a similar view for the initial phase of the micro network. Locality is going to matter more and more as we move into the future. Although Cloud still has its place and we don't ignore that either. https://github.com/micro/micro https://github.com/micro/micro https://micro.mu/docs/network.html https://micro.mu/docs/network.html
tannhaeuser 7y ago
You don't have to invent entire new paradigms such as CRDT for this. Unix is all about site autonomy, no-BS tooling, simplicity, and portability. So for your next project, consider Unix/Linux as deployment target during development, and only then deploy it to a cloud-hosted Unix cluster, with a local-first but cloud-hosted DB such as PostgreSQL and standardized middleware such as AMQP/RabbitMQ/qpid rather than provider-specific solutions, or at least use de-facto standard protocols such as s3 and MongoDB (if needed) and supported by multiple clouds. Many people are prematurely committing to k8s and "microservices" but in my experience, even though k8s as such isn't intended as a lock-in strategy, it has the effect of absorbing so much energy in projects (with devs more than happy to spend their time setting up auth, load balancing, and automating things rather than on business functionality), and then still ending up with a non-portable, incomprehensible mess of configs and deploy scripts that it just isn't worth it.
- vbsteven 7y ago
  My view on this is a bit different. I see Kubernetes as the abstraction layer on top of the cloud providers. In the last few years I have set up multiple k8s clusters for clients who specifically do not want to be locked in to a certain cloud provider. Once the software is running on top of k8s it is easy to switch cloud providers without changing the software. Switching to another cloud provider this way is trivial and usually only involves changing the Terraform configuration to setup a k8s cluster on another cloud. All k8s-specific config/deploy files can be reused on the new cluster. This of course only works if (as you suggest) you stay away from cloud-specific services (SQS, aurora, ECS, S3) and run everything in-cluster, or use managed services that are available on multiple providers (Postgresql via RDS, or Digital Ocean managed Postgres, Cloud SQL on GCP)
  - tannhaeuser 7y ago
    > Switching to another cloud provider this way is trivial Based on my limited experience, I highly doubt this. Have you actually deployed cross-cloud k8s setups, or is this merely a theoretical statement on your part? Deploying to another cloud provider brings a whole new universe of failure modes and auth quirks, let alone migration and switch-over woos.
milansuk 7y ago
>It should be fast. We don’t want to make round-trips to a server to interact with the application. The cloud apps are not slow only because of moving data, but there is also a problem that an average server is fast(16cores CPU + 64GB RAM), but If it's used by let's say 100users, It means one user has only 0.16core + 0.64GB memory. So an average laptop(4cores/4GB) or phone(4cores/1GB) is way faster. Basically people buy billions of transistors to use them only as a terminal to the cloud. Not to mention privacy risks. A week ago, I did showHN for skyalt.com. It's a local accessible database(+ analytics, which is coming soon). I'm still blown away how fast it is, that you can put tens of millions of rows to single table with many columns on consumer hardware and you don't pay for scale or attachments.
- jdnenej 7y ago
  That's not a fair comparison since most of the memory usage is just loading the app in to memory and then everyone is sharing the same app already loaded. Web apps don't have to be nearly as slow as they are. It's just that it's easier to make a slow app than a fast one. Also desktop apps are becoming super slow and bloated now thanks to electron.
  - 72deluxe 7y ago
    Sad about the Electron route. I don't consider it to be "desktop" software. Native toolkits are miles faster than the extremely lazy approach of running an entire new browser instance because someone wanted to write javascript for their "app". It's terrible.
    - calferreira 7y ago
      In my opinion it's not all about lazyness. You have to consider the costs of building an app native for each platform vs a cross platform framework built with javascript. Single javascript code base for all platforms is a lot cheaper.
      - 72deluxe 7y ago
mgreenleaf 7y ago
I love this idea. Especially the end-to-end encryption for data that passes through a server to enable the ease of cloud computing without relinquishing data ownership. It also depends on _who_ owns the data. In an enterprise environment the company usually has a vital interest in the data and on-premise deployments are a good way of retaining cloud computing without giving up data ownership. I'm surprised that more SAAS products don't offer on-premise given the privacy and ownership benefits. The tricky part there is making software that is easy to deploy and maintain, which might be the reason that it isn't done more often. A product like Grammarly that allowed on-premise deployment would side-step a lot of the issues with sending all that data to a third party. I can't imagine a law firm ever being able to (legally) sign up for that.
- tarsinge 7y ago
  Maybe on-premise installation happen but are not advertised? I have experienced one case in a previous company of a deal large enough to justify one for a client with sensitive data. No need to say operations were not happy, but I don’t have much informations to judge if it was a good deal for the company in the end.
CPAhem 7y ago
Those 7 points remind me of the Syncdocs https://syncdocs.com https://syncdocs.com app which ticked most of them. Using an encryption app and Dropbox will let you do the same thing, too.
thirdsurf 7y ago
I learned of Plan-Systems.org, they’re working towards something like this. Their company is non-profit, their collaboration tools are open source and protocol and the service is built on Unity and Unreal, which makes it cross-platform.
interactivecode 7y ago
There is a longer version of this article here: https://www.inkandswitch.com/local-first.html https://www.inkandswitch.com/local-first.html
jka 7y ago
As far as possible, I'm following a local-first methodology for a recipe search, meal planner, and shopping list application: https://www.reciperadar.com https://www.reciperadar.com There's a 'collaboration' mode which allows peer-to-peer sharing of a session via CRDTs over IPFS. My partner and I select our meals for the week, and then when one of us is doing the shopping, we can mark ingredients as found -- the other person's view reflects those updates in near-real-time. If either of us lose connectivity, we can continue to use the app, and when data access is restored, those changes are synced with the shared session (with automatic conflict resolution). All data in the shared session is encrypted, and the collaboration link contains the keys. Much of this functionality is thanks to peer-base, which is an experimental but extremely useful library: https://github.com/peer-base/peer-base https://github.com/peer-base/peer-base A side-benefit of this approach is that all user data can be stored locally (in browser localStorage) - there are no cookies used in communication between the app and server.
- jimpick 7y ago
  (peer-base maintainer here) This is really cool! I found the share link, and just tried it between two Chrome browsers, and it worked great! Thanks for using peer-base! There's a lot of great work happening with js-libp2p recently that would be awesome to incorporate ... I'm hoping to get active developing on it again in the new year. I've got so many ideas for improvements.
  - jka 7y ago
    Thanks a ton for developing the library :) If & when you're looking for any more contributors and/or testing, let me know; I'd be glad to pay it back. I'm hoping to open source the reciperadar application's stack soon and happy for it to be part of any ecosystem of examples (jbenet's peer-chess was a big help to me getting started).
- Fnoord 7y ago
  Cool project! Some suggestions: * Allow "patchcables" with different hooks, so for example a different recipe database, or a different language. * Allow groups of ingredients, ie. tag ingredients. For example, if I say I don't want meat, I still get shrimp suggested which might be what I want but it might also not be what I want. * In line with that, allow to specify allergens or blacklist ingredients (in line with dietary preference). Allergens are more important than dietary preference. * Allow weighting of ingredients, such as "80% tomato and 20% bell pepper", or something like "tomato or bell pepper" or "tomato xor bell pepper". * Some kind of verification reading at store or fridge (bar codes / tags) to remove entries automatically. Been dreaming of this for a long time. The way I envisioned it stores would share an API, with price, and customers could use it to compare prices and order appropriately (including taking into account S&H). Then the smartfridge (and other smart storage) would read the incoming ingredients. It would also read outgoing ingredients, and it could detect when they don't reenter. It could read weight checking how full the package still is. I had this idea 20 years ago, but I never saw a viable business case in it. You'd need to sit between all these stores who already claim they have low margins. Plus, if you'd sit between them, how would you earn profit from such software? My take is that if you want to secure privacy, software like this would require an initial investment, and then being FOSS, or a subscription and it being partly FOSS.
longnow 7y ago
I've been following a local-first methodology without realising it for an app that I've been developing. It's a workout-tracking app called harder better faster fitter. It's designed for mobile use in the gym. https://harderbetterfasterfitter.com/ https://harderbetterfasterfitter.com/ At the moment the app is a local only service and there aren't any backups. Next year I plan to add a backend. I'll be keeping some of the ideas in this article in mind. Currently I'm using the browser's local storage api to store data locally. It mostly works, but will be bolstered significantly with a cloud backup.
touchpadder 7y ago
Abolition of ownership is a communist idea. Local first, cloud for sync and backup.
rburhum 7y ago
why would the section marked “Git+Github” have “collaboration” marked as ‘no’? Like that is what it actually does...
api 7y ago
So happy to see this! We created ZeroTier for the world this is talking about. :) https://www.zerotier.com/ https://www.zerotier.com/ /shameless but on-topic plug
krzepah 7y ago
Hi everyone, I've been working on this subject for a few months already ; Thank you OP, your work is wonderful to read and even though I've spent a few months on the idea already I haven't thought of reusing Dropbox or similar. I think exciting things are about to come :) I'd like to submit Working Group proposal to the IETF. Why would we need an RPC for Independent Apps? Independent Apps are surfacing as being a solution to the lack of control of our own data. oAuth Framework has allowed a more secure web, but even if it makes a difference between an identity provider and a resource host, it does confuse the resource and the service hosts. Independent Apps should NOT be claimed by a lone company, let's make it something that the web owns. How would it be structured? I personally believe there should be multiple subjects treated by the IWA Framework, as one being the qualities of independent apps, and second being how data is accessed. Both of these are currently Topics of Interest for the IETF : https://ietf.org/topics/ https://ietf.org/topics/ - However the way this Working Group would proceed should be discussed and decided by it's members. Why not submit a single person draft? I could propose a draft but it wouldn't have the same meaning as if it would be drafted by a Working Group. As individuals, we are motivated by our own agenda and the quality of said draft wouldn't be the same. I'm volunteering, but I'd like to allow other persons to join in as well. You can join your mails here https://forms.gle/igNdd6rH4MnPK8rb8 https://forms.gle/igNdd6rH4MnPK8rb8 , at December 6 I will send the Working Group proposal to the IETF with gathered persons, if accepted I believe it should remain open to anybody to join.
- brynb 7y ago
  You should check out the Braid project. We're already working on IETF drafts for a protocol of this nature within the HTTP working group. - https://github.com/braid-work/braid-spec https://github.com/braid-work/braid-spec - https://groups.google.com/forum/#!forum/braid-http https://groups.google.com/forum/#!forum/braid-http
dijksterhuis 7y ago
Just wanted to point out that iTunes has had a local focused set up since inception, using xml format for a library’s database. That seems to still exist with the introduction of Apple Music. So all library data (play counts, skips, file locations etc) are stored locally, but streaming files are hosted remotely. Although whether this was by accident or design I have no idea.
marknadal 7y ago
P2P & CRDTs are definitely production ready already. HackerNoon & Internet Archive are using (mine) https://github.com/amark/gun https://github.com/amark/gun already. Local-first is very much the mantra of the whole dWeb community. I'm liking this naming "local-first" as an evolution to "offline-first". Ink & Switch had a good article on this: https://www.inkandswitch.com/local-first.html https://www.inkandswitch.com/local-first.html Also, for doing End-to-End Encryption, we've built some really good tooling around this as well: https://gun.eco/docs/SEA https://gun.eco/docs/SEA , wraps WebCrypto and works across Browsers, NodeJS, and React Native, so you can do some really cool cross-environment/platform apps now.
thawaway1837 7y ago
Isn’t Office 365 the platonic ideal of a local first software (suite) by this definition? High quality desktop apps, data saved in discrete documented file formats, optional ability to save in the cloud, the presence of collaborative editing, privacy is protected if you’re using it locally only, etc.
ninkendo 7y ago
Nearly all of Apple’s first party software works this way. Notes, Reminders, Pages, Numbers, Keynote, Voice memos, etc All using iCloud APIs to synchronize what is essentially local-first software. You could even count Mail, Contacts, and Calendar, although they rely on more established protocols to sync.
- davecap1 7y ago
  Apple Health and Activity also seem to work this way, although they also sync to iCloud.
johnmarcus 7y ago
I've been working on SPA's that launch from keybase.io public folders and can talk to my local KBFS storage, which is encrypted locally and then and distributed to the cloud. This way I can access my own data anywhere I have keybase installed, using apps I don't need a server to host. It's still all just prototype work for myself, but im excited about owning my own data while still having the safety of cloud distribution combined with the security of local encryption.
Scirra_Tom 7y ago
Our software Construct 3 (https://editor.construct.net/ https://editor.construct.net/) I think meets most of these points. It runs offline, and we never have access to the users project files. You can save/load locally, and it runs in the browser. Game project files are zips with JSON + raw asset files. No syncing with the server needed so it is fast, and a design mistake that is severely hampering some of our competitors! I'm not entirely sure how supporting collaboration in real time belongs on this list. Seems like a nice to have that isn't really related to the rest of the list.
StavrosK 7y ago
So I got fed up with current image hosting solutions the other day, because I realized free image hosting is unsustainable and Imgur has turned into a social network, which is the opposite of what I want. So, I figured I'd create my own paid one, and am working on https://imgz.org https://imgz.org. However, I want to add a free tier for people who are willing to host their own images, and was thinking of writing a daemon that would run on the user's computer and store all their images on a directory there. It would have to be mostly-on, but not always-on, since I'm going to be using a caching CDN. Is this a good idea? I don't know how many people would know how/want to run this, but it feels empowering from a data ownership perspective. What does everyone here think?
- imhoguy 7y ago
  Sounds like IPFS and pinning.
  - StavrosK 7y ago
    Kind of, except it will work.
    - jimpick 7y ago
      IPFS works fine. I'm on the IPFS team and I use it everyday. Depending on the app, it may or may not be a good fit. The performance you get out of it depends a lot on what features you are using. There's a whole ecosystem of projects (eg. IPFS Companion, IPFS Desktop, IPFS Cluster) and 3rd-party services which are important to consider when deploying a production-ready app. There's a lot of work ongoing to solve some of the biggest pain points (eg. content discovery across NATs), so expect the performance profile to improve dramatically in the short term and for it to become an option for many more apps.
      - StavrosK 7y ago
        > IPFS works fine. I'm on the IPFS team and I use it everyday. I run Eternum.io and IPFS has been such a pain that I am considering just shutting the service down. The node has been consuming so much RAM and CPU (even though it's behind a caching proxy and the gateway should get minimal traffic) that it was disrupting everything else on the server. The memory leaks have been off the charts for ages, so now I just restart the node every day. I set up IPFS cluster on that and another machine with the intention of moving node to the second machine, I waited for weeks for files to be pinned between the two nodes but the queued count was going up and down. In the end I set the pinning timeout to ten seconds and it finished faster, and still a bunch of files didn't manage to pin, even though the two nodes were directly connected (`ipfs swarm connect`). I shut the first node down anyway because I couldn't deal with it any more. At least now the rest of the stuff on the server isn't flapping every day. And this is on top of the atrocious pin handling that requires you to have a connection open to the IPFS daemon when you want to pin a CID for the entire duration of the pin. I opened a ticket years ago to get a sort of download manager in the node so we could have pins happen asynchronously, but there has been no movement on that at all. I'm glad it works well for you, but it has been nothing but pain for me. Hell, most of the time the gateway doesn't manage to discover files I have pinned on my local computer. I really want something like IPFS to succeed, because it has immense potential to literally change the world, but I can't even recommend that people run a node locally because I know it's going to eat their battery and slow their computer down. I don't know why these problems haven't gone away after years of work and millions in funding.
gioscarab 7y ago
I'm applying the local-first methodology in PJON, a new networking open standard : https://github.com/gioblu/PJON https://github.com/gioblu/PJON
amitport 7y ago
Hmm... homomorphic encryption.
davidkuennen 7y ago
Very cool. Let me add to all of these great examples in the comments. I published this app with focusing on exactly these points just two days ago. https://stockevents.app/ https://stockevents.app/ All tracked stocks stay within the app. You only pull information from the servers and store that information locally for offline use.
daleharvey 7y ago
I may be biased as the maintainer of PouchDB but you can do all this today (and for the last 5+ years) with PouchDB. The comment about CouchDB and the "difficulty of getting application-level conflict resolution right" I am not really certain how it applies, You dont have to handle conflicts in Pouch/CouchDB if you dont want to, there is a default model of last write (actually most edits) wins but you can handle them if needed
- knubie 7y ago
  I've been using a combo of PouchDB/CouchDB in my app[0] for the past few months, and I find it a hard combo to beat at the moment. I just haven't found anything else that works as seamlessly. While going through the article I found I was able to tick most of the boxes thanks to PouchDB. [0] https://mochi.cards/ https://mochi.cards/
- deleted 7y ago
  [deleted]
- adamwiggins 7y ago
  Hi Dale, I'm one of the local-first paper coauthors. I'm a fan of PouchDB (thanks for that) and the whole CouchDB lineage--the CouchDB book[1] was an early inspiration in my exploration of next-gen storage and collab models. I've been down the CouchDB/PouchDB path several times with several different engineering teams. Every time we were hopeful and every time we just couldn't get it to work. As one example, I worked with a small team of engineers to implement CouchDB syncing for the Clue Android and iOS mobile apps a few years back. Some of my experience is written down here[2]. After investing many months of engineering time, including some onsite help from Jan Lehnardt[3] we abandoned this architecture and went with a classic HTTPS/REST API. Other times and with different teams we've tried variations of Couch including PouchDB with web stack technology including Electron and PWA-ish HTML apps. None of these panned out either. Wish I could give better insights on why--we just can't get it to be reliable, or fast, or find a good way to share data between two users (the collaboration thing is kind of the whole point). [1]: https://guide.couchdb.org/ https://guide.couchdb.org/ [2]: https://medium.com/wandering-cto/mobile-syncing-with-couchbase-6f076d8c7e08 https://medium.com/wandering-cto/mobile-syncing-with-couchba... [3]: https://neighbourhood.ie/couchdb-support/ https://neighbourhood.ie/couchdb-support/
FpUser 7y ago
Article mentions the following tweet: What was the last app you used on your laptop that wasn't: Terminal VS Code Web Browser OS settings That guy must be living in a very limited / imaginary world world. I use boatload of local software
- habnds 7y ago
  Which are your favorites?
  - FpUser 7y ago
    I do not have favorites. I use various tools for various activities. Mechanical design, electronics, software development, video, photo, data analysis etc. etc. Way too much to list all. Some apps are being used very rarely but I still need those when the time comes
uber-geek 7y ago
A more recent issue, this one with youtube banning google accounts for posting too many emoji's during a live stream,what if all of your business data was on your google account? You would be done unless you could get unbanned. There is something to be said for using local apps. The engineers I work with are often out of range of any network, so having apps that run on their laptops is crucial.
have_faith 7y ago
> you OWN YOUR data Like most people here I'm fairly hard line when it comes to personal data abuses but I still struggle with the concepts of owning data about yourself. It's a confusion I see amongst less technically literate people when a well meaning person explains to them the importance of some latest data breach and they try to understand the concept that they owned this data, it was theirs but now it has been "stolen" or abused in some way. I would consider going as far to say that framing the data as owned by you is a bad approach, but maybe I'm just being pedantic about the language. Company A does have data about me, but I don't own it, and they have responsibilities to protect it (or delete it if requested), but I don't see any ownership in the equation, especially when the nature of the data can become quite abstract while still maintaining some reference to you. Not to take away from the intention or sentiment of framing it that way though, I'm just musing.
- gtirloni 7y ago
  You could argue in the past that PII data was just name, birthday, SSN, etc. However, nowadays pretty much anyone can be identified by their metadata so that's where I think this broad sense of ownership comes from.
- mrkeen 7y ago
  Ownership is precisely the right word to use. Companies should possess your data, not own it.
- SeanMacConMara 7y ago
  consider the concept that lots of businesses operate perfectly fine whilst temporarily having full physical control over physcial objects owned by their customers why not treat data the same way ? yes it will be very disruptive to some businesses. i hope.
- cgriswald 7y ago
  I don't entirely agree. People may be confused about data, but they are not confused about ownership. They understand ownership quite well. By sticking with that framing, you only have to get them to understand this ephemeral thing. Once you do, they can immediately apply all their concepts of ownership to it. To replace ownership with something else, well, now you've still got to get them to grasp the data, but also this something else, which might necessarily be somewhat abstract. I'm not even quite convinced they can't grasp the concept. People think they 'own' their digital copy of The Rescuers Down Under, when technically they've leased it. They generally understand they shouldn't post their credit card numbers online. I think it's more likely that they just don't understand the consequences, and even if they did, generally feel dis-empowered to do anything about it. That's a dynamic that has existed since before the internet.
herf 7y ago
I spent a lot of time with photos (Picasa) trying to do this peer to peer - this is what we built in the 2002 era. Here are a few issues: 1. Identity is hard to do on the LAN, so any sharing ends up using the cloud to figure out who has access. Similarly, identity is hard to move around, so representing your Facebook comments feed outside Facebook is difficult to do. 2. Any time you have a "special" server that handles identity, merging, or any other task, it ends up in the effective role of master, even if the rest of the parts of your system are peers. You want all your collaborations in Dropbox to survive their infrastructure being offline? It's tough to do. 3. p2p stalled a bit in the mid-2000s when storage and bandwidth got much cheaper--in a period of just two years (2002-2004), it became 100x cheaper to run a cloud service. But what continued to stall p2p was mobile. Uploading and syncing needs to run in the background, and if you're on a limited bandwidth client or a battery-limited device like iOS, sync can effectively diverge for months because the applications can't run in the background. So changes you "thought" you made don't show up on other devices. 4. For avoiding mass surveillance, what we are missing from this older time is the ability to make point to point connections (between peers) and encrypt them with forward secrecy, without data at rest in the cloud. Even systems that try to do some encryption for data at rest (e.g., iMessage) keep keys essentially forever, so data can be decrypted if you recover a private key later on. A system that only makes direct connections between peers does not have this issue. 5. Anytime you have multiple peers, you have to worry about old versions (bugs), or even attackers in the network, so it's fundamentally harder than having a shared server that's always up to date and running the latest code.
- koheripbal 7y ago
  With regards to identity management, maybe there should be a formalized integration between browsers and password managers such that the concept of "registration" goes away and new logins just automatically create user accounts with default permissions, according to email address.
  - mceachen 7y ago
    Centralized identity management was the grand promise of OAuth. It seems that somehow ended up, in most practical applications, as a "login with Facebook" button. There are a number of startups working on simplified or "passwordless" auth, but it seems that none have substantive traction. I'd love to be proven wrong, here, though!
wazoox 7y ago
I'm using NextCloud for that. Nextcloud can be basically used in local-first mode (mimicking dropbox). My phone automatically uploads all pictures to the NextCloud. Then there are apps. For instance I use Nextcloud with the Music app to stream my own mp3s from my Nextcloud to my phone running Ampache. There are also collaborative editing tools, and various options to edit all sort of documents in a web UI, and always the local editing fallback (or the opposite way, as you see fit).
Jareddd 7y ago
Awesome write-up! This brings me back to early 2000's when we we typically owned most of our graphic design software. Yes it was expensive but, there was barley any "cloud" features added and if you wanted a new version you simply had to download the update or simply keep the software you were currently using. My team and I have taken the initiative to offer a e mail design tool that is considered first-class software to the OS (https://bairmail.com https://bairmail.com). The last thing I would say is developing desktop apps vs a web app is considerably harder thus most companies are aware they are saving by controlling software updates, versioning etc.
lilactown 7y ago
I recently started using Actual[0], a local-first budgeting app. It's encrypted (soon to be end-to-end), local-first and can be synced across my devices. Would recommend, and I was inspired to use it after following its author, James Long[1] on twitter for awhile. [0] https://actualbudget.com/ https://actualbudget.com/ [1]https://twitter.com/jlongster?lang=en https://twitter.com/jlongster?lang=en
spieglt 7y ago
If you want an easy way to make a file (or .zip of files) more private before storing it in the cloud for backup/availability purposes, please check out Cloaker: github.com/spieglt/cloaker. It's simple, password-based, drag-and-drop file encryption.
pjkundert 7y ago
The Holo / Holochain project was founded with this principle as a primary goal: https://holo.host https://holo.host We’ll be implementing CRDTs soon, but the concept of local control of all data, authenticated and encrypted communications, etc. is implemented. One fundamental difference between apps that support this and those that cannot: agent-centric vs. data-centric design. Strangely, many “distributed” applications (eg. Bitcoin) didn’t make the “leap” to agent-centricity, and thus missed out on some key enabling optimizations. As a result — they are forced to implement “global consensus” (expensively), when they didn’t need to, to achieve their goals: a valid global ledger, in Bitcoin’s case. It turns out that, to implement things like cryptocurrencies, you don’t need everyone, everywhere to agree on a “total order” for every transaction in the entire network! Agent-centricity, FTW!
ph0rque 7y ago
That is my vision for https://edible.estate https://edible.estate, version 2 (version 1.0 should be done this year).
josefx 7y ago
I wrote documentation for configuration files in xsd with the idea that I could use xslt to display them in a browser on any system and use the same files for validation. This worked a year ago. Now browsers consider local files that access other local files suspect and will refuse to load anything unless beaten. So I now use a python script to run a simple local http server to view my local files from a single "origin". However http itself is already considered suspect and many claim it should be deprecated for https. In the future I will have to provide a lets encrypt signed https server with valid domain so anyone can view those files on their browser without having to mess with about:config settings or their own certificate cache. The cloud is the future, do not dare to build something that runs locally.
- bouk 7y ago
  http over localhost is considered secure by browsers
  - josefx 7y ago
    Is there a documented guarantee of that somewhere? Because I am half tempted to make a browser based UI for a few tools, but would rather use Java if there is even a tiny risk that I get the rug pulled from under my feet again.
alleycat5000 7y ago
Reminds me a little of the Upspin project: https://upspin.io/ https://upspin.io/ At least some of the motivations are the same.
at-fates-hands 7y ago
Is this something like Blockstack could help with? https://blockstack.org/ https://blockstack.org/ I'm having a hard time understanding the differences between "Local-first technology" and something like Blockstack. I'm not saying BS completely solves the issues pointed out in the blog post, but it seems to me its pretty close. What do you think? here is a list of the current apps available: https://app.co/blockstack https://app.co/blockstack
brynb 7y ago
I've been working for a few months on a database called Redwood that's intended to make it easier to build this kind of software. Having spent much of the past couple of years working with libp2p, IPFS, Dat, and similar technologies, I was curious to see what would result if I started from the ground up. https://github.com/brynbellomy/redwood https://github.com/brynbellomy/redwood So far, the model seems promising. It's fully peer-to-peer, and supports decentralized identity, configurable conflict resolution, read/write access, asset storage, and currently is running across 3 different transports: - HTTP (augmented with extensions proposed by the Braid project [1][2]) - Libp2p - WebRTC I've included two simple demos, a collaborative document editor (well, it's just a textarea at the moment), and a chat app. Would appreciate any feedback or participation that folks are willing to give. [1] https://github.com/braid-work/braid-spec https://github.com/braid-work/braid-spec [2] https://groups.google.com/forum/#!forum/braid-http https://groups.google.com/forum/#!forum/braid-http
oblib 7y ago
I've been building apps that meet this criteria for several years now. It's nice to see the concept getting some attention here. The only thing I'll point out is that CouchDB getting rated "partially meets the idea" seems pretty weak to me. They reference v2.1 but the latest version is 2.3.1 and here's link to the docs on how conflict resolution is dealt with: https://docs.couchdb.org/en/latest/replication/conflicts.html https://docs.couchdb.org/en/latest/replication/conflicts.htm... If finer grained control is needed it would be up to the developer to implement it, and it really shouldn't be difficult to do that. In my case, I use PouchDB to perform a "live sync" with all connected users so they all get the latest updates to a document. If a conflict arises it's easy for any one of the users to fix and push it to everyone connected.
hwwang2005 7y ago
I'm creator of BLOON (https://www.bloon.io https://www.bloon.io). If BLOON had been put in the Table 1 in the paper, the values could be: O O - - O - O I think BLOON is closer than Git+GitHub. And great article & paper! They do give us many inspiration about improving BLOON. Thank you!
garfieldnate 7y ago
This. I want this. It happens to me quite often that I feel demotivated or uneasy about using some software because I know I'm producing valuable data specific to me that I could use in the future, but the product stipulates that I don't get to keep the data. iOS apps are often the worst at this. There's no filesystem, so all data has to be kept with an app. I loaded all of the books I wanted to read into iBooks, and then it turned out that the iOS backup skips books that you didn't buy from Apple. Bye bye book collection T_T. Some of this could also be alleviated by Tim Berners-Lee's pod idea: https://solid.inrupt.com/ https://solid.inrupt.com/. But local-first is better. I just want files on my machine.
jefurii 7y ago
I'm surprised nobody has mentioned using git-annex (and Git of course) to manage data (full disclosure: I develop an archival management app which keeps its data in git/git-annex repositories). Of the seven key properties, git-annex gives you 1,2,3,4 and supports 5 and 7; 6 depends on how you store things. Git-annex supports identity in the sense that each clone of the repository has a UUID. You can choose to have a central hub if you like but you don't have to (surprise! you don't really need GitHub!). It comes with caveats of course: Binaries can be synced but only textual data is actually versioned, and once you put something in it will always be there unless you use something like git-filter-branch.
shreyshrey 7y ago
Tonido (https://www.tonido.com https://www.tonido.com) is a good local-first personal cloud. It allows you to access, share files stored in your PC from anywhere. We are giving it free for almost 10 years. The longevity is higher than many cloud services :). You own your data.
chaz6 7y ago
IPv6 will make it easier to access your data anywhere. I am putting this here because so far nobody has mentioned it.

748 points by godelmachine 7y ago

ricg 7y ago

"It should support data access for all time." - This is key for me after I had to convert my notes more than once between formats after the original app(s) went into extinction (beloved Circus Ponies Notebook). That's why I'm designing any new apps around a file format that can be accessed even without the app. I have a "local-first" Kanban/Trello-style app, "Boards" (http://kitestack.com/boards/ http://kitestack.com/boards/), that uses zipped HTML files (to support rich text with images). No collaboration and cross-device support just yet, but it works without a network and saves everything locally.

threatripper 7y ago
Boards seems to be Mac only. "A new Mac app to boost your productivity in school, at work, and for personal projects.", "For macOS 10.14 Mojave and later"
- ricg 7y ago
  Yes, currently Mac-only, no cross-device support yet, but an iOS app is in planning.
jonpurdy 7y ago
I just tried Boards out. It's really nice and exactly what I wish Trello could be for a single user. Will follow the progress on this for sure.
- ricg 7y ago
  Thanks for your feedback! I don't have a newsletter signup just yet, but if you'd like to hear about new versions, send me a quick email through the app (Help → Contact Support) and I'll add you to the list.

rado 7y ago

Common sense.

ptx 7y ago
You might think so from only reading the title, but what the article/paper actually proposes is using CRDTs to create a synthesis of the advantages (for the user) of both cloud applications and traditional local applications.

quickthrower2 7y ago

My side project is a local first (local storage on web) JAMStack. For extra goodness it’s mobile first too. I really love making apps this way for some reason. I think it’s the focus on just the UI and not worrying about the back end until later. For this particular app I’d consider “smartwatch first” to have been better as its for fitness!

randomsearch 7y ago
This is a really good way to prototype and find the minimum feature set for a SaaS startup.

TAForObvReasons 7y ago

Today's SaaS world is largely economically opposed to the idea of data ownership. It's a lot easier to make money by renting people access to their data. The problem is not inherently technical. The solution must address the fact that the software businesses favor cloud solutions and other systems that make it difficult to stop spending money

quickthrower2 7y ago
Yes. I always think data freedom is more important than software freedom. For example it matters less that MS Word is not free as in freedom when you can open the file in something that is.
- TeMPOraL 7y ago
  And cloud SaaS are enemies of both! After all, in order to have data freedom, you need to have something to open in that other program, and cloud solutions do their best to not give you proper open/save file (and even if there's an export function, and even if everything is actually included in the export, it often isn't followed by import function that could read that export). For this reason, I avoid using cloud SaaS for anything where can avoid it.
  - quickthrower2 7y ago
    Absolutely. There are some exceptions though. Github is the obvious one. Dropbox by nature has your data constantly exported. Google for its sins has Google takeout.
    - TeMPOraL 7y ago
      Yes. And the first two I use, treating them mostly as dumb pieces of infrastructure. Arguably, the functionality they provide have a crucial ops component that I'm all for paying for someone else to handle it for me. But neither Github nor Dropbox locks me into anything. Google - yes, web e-mail obviously is similar to the above; as for their office suite, I recently found a good excuse to justify shelling out for a proper Microsoft Office subscription (though I don't like that it's a subscription), and I stick to using the faster, locally-available, file-using, much more powerful (if still proprietary) software.

mwilcox 7y ago

any marginally successful "local-first" app is going to go and raise $10m in vc, switch to software as a service, and add an enterprise mode that requires user permissions and data access to be managed on the server

LeftHandPath 7y ago
Sure, any one company probably will - but there’s a whole market. As soon as that one company abandons the local-first model, a gap opens, which will (usually, eventually) be filled by a new company offering local-first until that new company does the same. As long as the companies don’t band together and agree to end it, there should be a company offering that model somewhere somehow.
- mwilcox 7y ago
  Even in the situation I described, the original company would leave the free / client side version on the site as free marketing. This is the standard today for enterprise-monetised open source software. Solving this problem isn't about being local-first. It's about being local-last. You have to be able to make more money by selling a software license than you make by selling equity and chasing user acquisition and retention. Then we'll see people waking up to the fact that all this proprietary data is a liability and subscriptions are golden handcuffs and people will finally get back to making real software again
rapsey 7y ago
Because local-first is not a viable business model compared to the cloud. Software goes where the money is.
toomim 7y ago
That's why we need to add local-first features into HTTP itself. I've been doing some work on it over the last few days: https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-braid-http-01 https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-b...
0xCMP 7y ago

gobengo 7y ago

The blog post doesn't mention this, so I thought I'd point this out. One of the paper's authors is Martin Kleppmann, who wrote the very good https://dataintensive.net/ https://dataintensive.net/ book.

ignoramous 7y ago
Kleppmann co-created/major contributor to Apache Kafka along with Jay Kreps and Neha Narkhede. He also co-founded Rapportive, a YC company acquired by LinkedIn, along with Rahul Vohra, who is presently the CEO of Superhuman.
atrilumen 7y ago
And don't miss "Turning the database inside out with Apache Samza" <3 https://www.youtube.com/watch?v=fU9hR3kiOK0 https://www.youtube.com/watch?v=fU9hR3kiOK0

theamk 7y ago

It’s strange that Evernote is omitted from the list - it is a great example of local-first app. Their recent-ish history, when the their free tire become limited to only syncing a few devices, illustrates that even if software is fully local, and supports open formats, having the functional cloud matters, a lot.

pqs 7y ago
I love Evernote and I use it daily, but Evernote lacks end-to-end encryption, which is a pity. I store lots of information in that app and I would be more reassured if I knew that it is only me who can read the data. I would even pay more in order to have that kind of encryption. I think that no current feature of Evernote would be affected by encryption as text recognition in images can be done at the client level.
thundergolfer 7y ago
Evernote is included in one of their “what not to do” images, showing that they can’t handle conflict resolution at all. But yeah otherwise Evernote is pretty good for a single user.

deleted 7y ago

[deleted]

toolslive 7y ago

My company built this more than 10 years ago, and nobody was interested. Here's a press release from 2007: https://www.macworld.com/article/1058169/mcnucle.html https://www.macworld.com/article/1058169/mcnucle.html

toolslive 7y ago
found a demo movie too: https://www.youtube.com/watch?v=bvzfnAY0XiU https://www.youtube.com/watch?v=bvzfnAY0XiU
fastball 7y ago
Might have something to do with the fact that McNucle is a freaking terrible name.
- toolslive 7y ago
  no, most end users did not understand (or cared about) the data ownership problem back then. Times have changed.
  - procombo 7y ago
    That name though...
    - mackrevinack 7y ago
      lol

jmathai 7y ago

I've been trying to document my "local-first" approach to managing photos. I've made it a ways through but am not sure when I'll finish. Posting here since it is relevant. A Pragmatic Photo Archiving Solution: https://docs.google.com/document/d/1JzqT-DJFlS2e8ZC00HrsQITq_ukORcH9icXTGKeWQYU/edit?usp=sharing https://docs.google.com/document/d/1JzqT-DJFlS2e8ZC00HrsQITq... It's the culmination of software I've written [1] + a workflow that's resulted from it [2, 3, 4, 5]. [1] Elodie - https://github.com/jmathai/elodie https://github.com/jmathai/elodie [2] Understanding my need for an automated photo workflow - https://medium.com/vantage/understanding-my-need-for-an-automated-photo-workflow-a2ff95b46f8f https://medium.com/vantage/understanding-my-need-for-an-auto... [3] Introducing Elodie; Your Personal EXIF-based Photo and Video Assistant - https://medium.com/vantage/understanding-my-need-for-an-automated-photo-workflow-a2ff95b46f8f https://medium.com/vantage/understanding-my-need-for-an-auto... [4] My Automated Photo Workflow using Google Photos and Elodie - https://medium.com/@jmathai/my-automated-photo-workflow-using-google-photos-and-elodie-afb753b8c724 https://medium.com/@jmathai/my-automated-photo-workflow-usin... [5] One Year of Using an Automated Photo Organization and Archiving Workflow - https://artplusmarketing.com/one-year-of-using-an-automated-photo-organization-and-archiving-workflow-89cf9ad7bddf https://artplusmarketing.com/one-year-of-using-an-automated-...

geolgau 7y ago
This looks great and feels a lot like beets [1] for music, only that they use a database. I'll try it when I have the time to re-organize ~20 years of photos. [1] http://beets.io/ http://beets.io/
emilburzo 7y ago
elodie looks amazing! I definitely need to try it out it was just the thing I was thinking of building recently as it was getting really tiring to manually organize photos one extra idea that I had: there's a cool project that would enable offline geocoding[1], which would help get rid of API limits while making the reverse geocode queries almost instant (the included dataset is pretty limited, but it's not hard to extend from an openstreetmap planet dump) [1] https://pypi.org/project/reverse_geocoder/ https://pypi.org/project/reverse_geocoder/

gosubpl 7y ago

Things are happening in that space already! I work at Actyx and we have a production ready stack for local-first real serverless (peer-to-peer) applications. Please take a look at https://actyx.com/ https://actyx.com/ .

throwGuardian 7y ago

P2P has miles to go before challenging the reliability, convenience and performance of the cloud. That said, one area undervalued is Partially homomorphic cryptosystems[1], where the cloud never ever gets to see unencrypted user data. I hope the future is fast-local compute on cached data, with the cloud holding a much larger, encrypted but permissioned data store, offering utility functions like search over encrypted data [1]: https://en.m.wikipedia.org/wiki/Homomorphic_encryption https://en.m.wikipedia.org/wiki/Homomorphic_encryption

mbalex99 7y ago

Martin Kleppmann is a major inspiration for our startup, Ditto. We take the local-first concept and p2p to the next level with CRDTs and replication. But what we really do is leverage things like AWDL, mDNS, and or Bluetooth low energy to sync mobile databases instances with each other even without internet connectivity. www.ditto.live Check it out in action! https://youtu.be/1P2bKEJjdec https://youtu.be/1P2bKEJjdec https://youtu.be/ITUrk_rjnvo https://youtu.be/ITUrk_rjnvo We found that CRDTs, local first, and eventual consistency REALLY shines in the mobile phones since they constantly experience network partitions.

Jyaif 7y ago
Very interesting. Will the upcoming server support be end-to-end encrypted? In other words, will the server be able to read the data?
mamcx 7y ago
I made "traditional" enterprise-style-apps for small business and have tried to crack the sync of data several times. Exist a resource in how leverage that tech with boring stuff like inventory, invoices, etc? Hopefully,. without a total change of stacks (I use postgresql, sqlite as dbs, and need to integrate with near 12+ different db engines)

ranjithdsm 7y ago

The data ownership is very important for business world. The reason why we built our products for IoT ( www.bevywise.com ) as a more install able version is the data privacy and ownership. We really see this largely in manufacturing industry. If not local, we should provide private servers and data security.

saurabhnanda 7y ago

Not related to the actual content, but may I please request you to increase the contrast on your website. Grey on grey is very hard to read.

josteink 7y ago
Firefox reader mode. Instantly fixes any crappy design on any site. Learn to love it :)

radium3d 7y ago

https://github.com/syncthing/syncthing https://github.com/syncthing/syncthing Syncthing solves a large part of syncing data between devices using your own VPS, server(s), etc. If your VPS provider goes out of business, you can then just fire up a new VPS and hook it back up to your local machine(s).

josteink 7y ago
Sadly Syncting still lacks iOS-support. As a result I'm running Nextcloud instead.
- geolgau 7y ago
  I'm on Android so it works, but I also used Nextcloud for some time and had issues with it "forgetting" about the data folder, so I had to re-create it every time it happened. I wasn't happy about that. With Syncthing I don't (yet) have this problem.
mackrevinack 7y ago
there's also resilio sync that gives you the option have the node/folder on your vps be encrypted. hopefully syncthing will add that feature at some point in the future
- Multicomp 7y ago
  I have been considering spending more money than I should finding someone on a freelancer site to add the untrusted node / encrypted file transfer but don't give a certain node the key to decrypt the file support, that is the number one missing feature for syncthing for me. It would unlock scenarios like share your files with me and I'll share files with you and neither of us will know the others files unless your house burns down in which case you can get your files back just by hookin the sync back up with the encryption key pulled from your password manager.
  - mappu 7y ago
    You can kind of work around this by sharing an ecryptfs folder.

jsfenfen 7y ago

The paper says pinterest meets the "collaboration" ideal but github doesn't. I'm sympathetic to the idea that nothing meets the ideal, but c'mon.

sans-serif 7y ago

iOS apps like Bear achieve this by building on iCloud storage

atoav 7y ago

When I select software these are among the list of things I am looking for generally: - file formats that won’t lock you in or are even openly hackable (allows you to automate things) - no clouds that will break the software once it is gone - local storage with custom syncing or backup options - strictly no weird data collection or “We own the rights to your data”-Type of terms So if I get the slightest feeling of a lock in or unnecessary data collection you are scaring me away, because mentally I would then already look at the time after you decide to scrap your cloud or abandon your file formats. The data collection bit shows me your users aren’t front and center but something else is which makes your product even less of a good choice. If your product runs on the web, allowing for self-hosted solutions is also a big plus.

jwr 7y ago
While I fully agree with your selection criteria, please consider the other side of the equation, because engineering (and the world) is all about compromises. I am the author of a SaaS app (https://partsbox.io/ https://partsbox.io/). I export in open formats (JSON), there is no lock-in, it's easy to get all of your data at any time. But the app is online and will remain so. Why? Economics. Maintaining a self-hosted solution is an enormous cost, which most people forget about. You need to create, document, maintain and support an entirely different version of your software (single-user, no billing/invoicing, different email sending, different rights system, etc). And then every time you change something in your software you have to worry about migrations, not just in your database, but in all your clients databases. I am not saying it's impossible, it's just expensive, especially for companies which are built to be sustainable in the first place (e.g. not VC-funded). Believe me, if you don't have VC money to burn, you will not be experimenting with CRDTs and synchronizing distributed data from a multitude of versions of your application. I regularly have to explain why there is no on-premises version of my app. The best part is that many people think that an on-premises version should be less expensive than the online version, and come without a subscription.
- rgun 7y ago
  > The best part is that many people think that an on-premises version should be less expensive than the online version, and come without a subscription. Which makes sense IMHO, provided they are not expecting any updates to their on-prem installation. It can just be a fork of your current codebase with no new features or warranty. Maybe you can include some terms for critical security updates but that is about it.

miki123211 7y ago

Resilio sync is a perfect example of such app. It's basically a P2P based Dropbox with no accounts, full end-to-end encryption and no folder size limits. It's not open source, but it can work without a central server if you need it to. It's also amazingly simple to set up, much simpler than sync thing.

mackrevinack 7y ago
i used resilio for years and thought the same about syncthing until i switched last year and now I actually prefer how syncthing is setup. if you pay for resilio there is an option to add all your folders in one go but on some computers I don't want to add all of them anyway so that's not much use to me. with the free version you have to manually add folders one by one but to do that you need the key which means you need to copy them to a text file and add them on another computer. with syncthing, it will detect other syncthing devices on your home network so you just have to add the ones you want then accept the request from another device. once that is done you select which folders the device has access to and then a notification will show up on said device asking you to connect. so basically no fiddling with keys or having to store them somewhere secure (this is all presuming i was using resilio correctly, maybe there was an easier way I was not aware of)

chuhnk 7y ago

Interesting to see this. We've taken a similar view for the initial phase of the micro network. Locality is going to matter more and more as we move into the future. Although Cloud still has its place and we don't ignore that either. https://github.com/micro/micro https://github.com/micro/micro https://micro.mu/docs/network.html https://micro.mu/docs/network.html

tannhaeuser 7y ago

You don't have to invent entire new paradigms such as CRDT for this. Unix is all about site autonomy, no-BS tooling, simplicity, and portability. So for your next project, consider Unix/Linux as deployment target during development, and only then deploy it to a cloud-hosted Unix cluster, with a local-first but cloud-hosted DB such as PostgreSQL and standardized middleware such as AMQP/RabbitMQ/qpid rather than provider-specific solutions, or at least use de-facto standard protocols such as s3 and MongoDB (if needed) and supported by multiple clouds. Many people are prematurely committing to k8s and "microservices" but in my experience, even though k8s as such isn't intended as a lock-in strategy, it has the effect of absorbing so much energy in projects (with devs more than happy to spend their time setting up auth, load balancing, and automating things rather than on business functionality), and then still ending up with a non-portable, incomprehensible mess of configs and deploy scripts that it just isn't worth it.

vbsteven 7y ago
My view on this is a bit different. I see Kubernetes as the abstraction layer on top of the cloud providers. In the last few years I have set up multiple k8s clusters for clients who specifically do not want to be locked in to a certain cloud provider. Once the software is running on top of k8s it is easy to switch cloud providers without changing the software. Switching to another cloud provider this way is trivial and usually only involves changing the Terraform configuration to setup a k8s cluster on another cloud. All k8s-specific config/deploy files can be reused on the new cluster. This of course only works if (as you suggest) you stay away from cloud-specific services (SQS, aurora, ECS, S3) and run everything in-cluster, or use managed services that are available on multiple providers (Postgresql via RDS, or Digital Ocean managed Postgres, Cloud SQL on GCP)
- tannhaeuser 7y ago
  > Switching to another cloud provider this way is trivial Based on my limited experience, I highly doubt this. Have you actually deployed cross-cloud k8s setups, or is this merely a theoretical statement on your part? Deploying to another cloud provider brings a whole new universe of failure modes and auth quirks, let alone migration and switch-over woos.

milansuk 7y ago

>It should be fast. We don’t want to make round-trips to a server to interact with the application. The cloud apps are not slow only because of moving data, but there is also a problem that an average server is fast(16cores CPU + 64GB RAM), but If it's used by let's say 100users, It means one user has only 0.16core + 0.64GB memory. So an average laptop(4cores/4GB) or phone(4cores/1GB) is way faster. Basically people buy billions of transistors to use them only as a terminal to the cloud. Not to mention privacy risks. A week ago, I did showHN for skyalt.com. It's a local accessible database(+ analytics, which is coming soon). I'm still blown away how fast it is, that you can put tens of millions of rows to single table with many columns on consumer hardware and you don't pay for scale or attachments.

jdnenej 7y ago
That's not a fair comparison since most of the memory usage is just loading the app in to memory and then everyone is sharing the same app already loaded. Web apps don't have to be nearly as slow as they are. It's just that it's easier to make a slow app than a fast one. Also desktop apps are becoming super slow and bloated now thanks to electron.
- 72deluxe 7y ago
  Sad about the Electron route. I don't consider it to be "desktop" software. Native toolkits are miles faster than the extremely lazy approach of running an entire new browser instance because someone wanted to write javascript for their "app". It's terrible.
  - calferreira 7y ago
    In my opinion it's not all about lazyness. You have to consider the costs of building an app native for each platform vs a cross platform framework built with javascript. Single javascript code base for all platforms is a lot cheaper.
    - 72deluxe 7y ago

mgreenleaf 7y ago

I love this idea. Especially the end-to-end encryption for data that passes through a server to enable the ease of cloud computing without relinquishing data ownership. It also depends on _who_ owns the data. In an enterprise environment the company usually has a vital interest in the data and on-premise deployments are a good way of retaining cloud computing without giving up data ownership. I'm surprised that more SAAS products don't offer on-premise given the privacy and ownership benefits. The tricky part there is making software that is easy to deploy and maintain, which might be the reason that it isn't done more often. A product like Grammarly that allowed on-premise deployment would side-step a lot of the issues with sending all that data to a third party. I can't imagine a law firm ever being able to (legally) sign up for that.

tarsinge 7y ago
Maybe on-premise installation happen but are not advertised? I have experienced one case in a previous company of a deal large enough to justify one for a client with sensitive data. No need to say operations were not happy, but I don’t have much informations to judge if it was a good deal for the company in the end.

CPAhem 7y ago

Those 7 points remind me of the Syncdocs https://syncdocs.com https://syncdocs.com app which ticked most of them. Using an encryption app and Dropbox will let you do the same thing, too.

thirdsurf 7y ago

I learned of Plan-Systems.org, they’re working towards something like this. Their company is non-profit, their collaboration tools are open source and protocol and the service is built on Unity and Unreal, which makes it cross-platform.

interactivecode 7y ago

There is a longer version of this article here: https://www.inkandswitch.com/local-first.html https://www.inkandswitch.com/local-first.html

jka 7y ago

As far as possible, I'm following a local-first methodology for a recipe search, meal planner, and shopping list application: https://www.reciperadar.com https://www.reciperadar.com There's a 'collaboration' mode which allows peer-to-peer sharing of a session via CRDTs over IPFS. My partner and I select our meals for the week, and then when one of us is doing the shopping, we can mark ingredients as found -- the other person's view reflects those updates in near-real-time. If either of us lose connectivity, we can continue to use the app, and when data access is restored, those changes are synced with the shared session (with automatic conflict resolution). All data in the shared session is encrypted, and the collaboration link contains the keys. Much of this functionality is thanks to peer-base, which is an experimental but extremely useful library: https://github.com/peer-base/peer-base https://github.com/peer-base/peer-base A side-benefit of this approach is that all user data can be stored locally (in browser localStorage) - there are no cookies used in communication between the app and server.

jimpick 7y ago
(peer-base maintainer here) This is really cool! I found the share link, and just tried it between two Chrome browsers, and it worked great! Thanks for using peer-base! There's a lot of great work happening with js-libp2p recently that would be awesome to incorporate ... I'm hoping to get active developing on it again in the new year. I've got so many ideas for improvements.
- jka 7y ago
  Thanks a ton for developing the library :) If & when you're looking for any more contributors and/or testing, let me know; I'd be glad to pay it back. I'm hoping to open source the reciperadar application's stack soon and happy for it to be part of any ecosystem of examples (jbenet's peer-chess was a big help to me getting started).
Fnoord 7y ago
Cool project! Some suggestions: * Allow "patchcables" with different hooks, so for example a different recipe database, or a different language. * Allow groups of ingredients, ie. tag ingredients. For example, if I say I don't want meat, I still get shrimp suggested which might be what I want but it might also not be what I want. * In line with that, allow to specify allergens or blacklist ingredients (in line with dietary preference). Allergens are more important than dietary preference. * Allow weighting of ingredients, such as "80% tomato and 20% bell pepper", or something like "tomato or bell pepper" or "tomato xor bell pepper". * Some kind of verification reading at store or fridge (bar codes / tags) to remove entries automatically. Been dreaming of this for a long time. The way I envisioned it stores would share an API, with price, and customers could use it to compare prices and order appropriately (including taking into account S&H). Then the smartfridge (and other smart storage) would read the incoming ingredients. It would also read outgoing ingredients, and it could detect when they don't reenter. It could read weight checking how full the package still is. I had this idea 20 years ago, but I never saw a viable business case in it. You'd need to sit between all these stores who already claim they have low margins. Plus, if you'd sit between them, how would you earn profit from such software? My take is that if you want to secure privacy, software like this would require an initial investment, and then being FOSS, or a subscription and it being partly FOSS.

longnow 7y ago

I've been following a local-first methodology without realising it for an app that I've been developing. It's a workout-tracking app called harder better faster fitter. It's designed for mobile use in the gym. https://harderbetterfasterfitter.com/ https://harderbetterfasterfitter.com/ At the moment the app is a local only service and there aren't any backups. Next year I plan to add a backend. I'll be keeping some of the ideas in this article in mind. Currently I'm using the browser's local storage api to store data locally. It mostly works, but will be bolstered significantly with a cloud backup.

touchpadder 7y ago

Abolition of ownership is a communist idea. Local first, cloud for sync and backup.

rburhum 7y ago

why would the section marked “Git+Github” have “collaboration” marked as ‘no’? Like that is what it actually does...

api 7y ago

So happy to see this! We created ZeroTier for the world this is talking about. :) https://www.zerotier.com/ https://www.zerotier.com/ /shameless but on-topic plug

krzepah 7y ago

Hi everyone, I've been working on this subject for a few months already ; Thank you OP, your work is wonderful to read and even though I've spent a few months on the idea already I haven't thought of reusing Dropbox or similar. I think exciting things are about to come :) I'd like to submit Working Group proposal to the IETF. Why would we need an RPC for Independent Apps? Independent Apps are surfacing as being a solution to the lack of control of our own data. oAuth Framework has allowed a more secure web, but even if it makes a difference between an identity provider and a resource host, it does confuse the resource and the service hosts. Independent Apps should NOT be claimed by a lone company, let's make it something that the web owns. How would it be structured? I personally believe there should be multiple subjects treated by the IWA Framework, as one being the qualities of independent apps, and second being how data is accessed. Both of these are currently Topics of Interest for the IETF : https://ietf.org/topics/ https://ietf.org/topics/ - However the way this Working Group would proceed should be discussed and decided by it's members. Why not submit a single person draft? I could propose a draft but it wouldn't have the same meaning as if it would be drafted by a Working Group. As individuals, we are motivated by our own agenda and the quality of said draft wouldn't be the same. I'm volunteering, but I'd like to allow other persons to join in as well. You can join your mails here https://forms.gle/igNdd6rH4MnPK8rb8 https://forms.gle/igNdd6rH4MnPK8rb8 , at December 6 I will send the Working Group proposal to the IETF with gathered persons, if accepted I believe it should remain open to anybody to join.

brynb 7y ago
You should check out the Braid project. We're already working on IETF drafts for a protocol of this nature within the HTTP working group. - https://github.com/braid-work/braid-spec https://github.com/braid-work/braid-spec - https://groups.google.com/forum/#!forum/braid-http https://groups.google.com/forum/#!forum/braid-http

dijksterhuis 7y ago

Just wanted to point out that iTunes has had a local focused set up since inception, using xml format for a library’s database. That seems to still exist with the introduction of Apple Music. So all library data (play counts, skips, file locations etc) are stored locally, but streaming files are hosted remotely. Although whether this was by accident or design I have no idea.

marknadal 7y ago

P2P & CRDTs are definitely production ready already. HackerNoon & Internet Archive are using (mine) https://github.com/amark/gun https://github.com/amark/gun already. Local-first is very much the mantra of the whole dWeb community. I'm liking this naming "local-first" as an evolution to "offline-first". Ink & Switch had a good article on this: https://www.inkandswitch.com/local-first.html https://www.inkandswitch.com/local-first.html Also, for doing End-to-End Encryption, we've built some really good tooling around this as well: https://gun.eco/docs/SEA https://gun.eco/docs/SEA , wraps WebCrypto and works across Browsers, NodeJS, and React Native, so you can do some really cool cross-environment/platform apps now.

thawaway1837 7y ago

Isn’t Office 365 the platonic ideal of a local first software (suite) by this definition? High quality desktop apps, data saved in discrete documented file formats, optional ability to save in the cloud, the presence of collaborative editing, privacy is protected if you’re using it locally only, etc.

ninkendo 7y ago

Nearly all of Apple’s first party software works this way. Notes, Reminders, Pages, Numbers, Keynote, Voice memos, etc All using iCloud APIs to synchronize what is essentially local-first software. You could even count Mail, Contacts, and Calendar, although they rely on more established protocols to sync.

davecap1 7y ago
Apple Health and Activity also seem to work this way, although they also sync to iCloud.

johnmarcus 7y ago

I've been working on SPA's that launch from keybase.io public folders and can talk to my local KBFS storage, which is encrypted locally and then and distributed to the cloud. This way I can access my own data anywhere I have keybase installed, using apps I don't need a server to host. It's still all just prototype work for myself, but im excited about owning my own data while still having the safety of cloud distribution combined with the security of local encryption.

Scirra_Tom 7y ago

Our software Construct 3 (https://editor.construct.net/ https://editor.construct.net/) I think meets most of these points. It runs offline, and we never have access to the users project files. You can save/load locally, and it runs in the browser. Game project files are zips with JSON + raw asset files. No syncing with the server needed so it is fast, and a design mistake that is severely hampering some of our competitors! I'm not entirely sure how supporting collaboration in real time belongs on this list. Seems like a nice to have that isn't really related to the rest of the list.

StavrosK 7y ago

So I got fed up with current image hosting solutions the other day, because I realized free image hosting is unsustainable and Imgur has turned into a social network, which is the opposite of what I want. So, I figured I'd create my own paid one, and am working on https://imgz.org https://imgz.org. However, I want to add a free tier for people who are willing to host their own images, and was thinking of writing a daemon that would run on the user's computer and store all their images on a directory there. It would have to be mostly-on, but not always-on, since I'm going to be using a caching CDN. Is this a good idea? I don't know how many people would know how/want to run this, but it feels empowering from a data ownership perspective. What does everyone here think?

imhoguy 7y ago
Sounds like IPFS and pinning.
- StavrosK 7y ago
  Kind of, except it will work.
  - jimpick 7y ago
    IPFS works fine. I'm on the IPFS team and I use it everyday. Depending on the app, it may or may not be a good fit. The performance you get out of it depends a lot on what features you are using. There's a whole ecosystem of projects (eg. IPFS Companion, IPFS Desktop, IPFS Cluster) and 3rd-party services which are important to consider when deploying a production-ready app. There's a lot of work ongoing to solve some of the biggest pain points (eg. content discovery across NATs), so expect the performance profile to improve dramatically in the short term and for it to become an option for many more apps.
    - StavrosK 7y ago
      > IPFS works fine. I'm on the IPFS team and I use it everyday. I run Eternum.io and IPFS has been such a pain that I am considering just shutting the service down. The node has been consuming so much RAM and CPU (even though it's behind a caching proxy and the gateway should get minimal traffic) that it was disrupting everything else on the server. The memory leaks have been off the charts for ages, so now I just restart the node every day. I set up IPFS cluster on that and another machine with the intention of moving node to the second machine, I waited for weeks for files to be pinned between the two nodes but the queued count was going up and down. In the end I set the pinning timeout to ten seconds and it finished faster, and still a bunch of files didn't manage to pin, even though the two nodes were directly connected (`ipfs swarm connect`). I shut the first node down anyway because I couldn't deal with it any more. At least now the rest of the stuff on the server isn't flapping every day. And this is on top of the atrocious pin handling that requires you to have a connection open to the IPFS daemon when you want to pin a CID for the entire duration of the pin. I opened a ticket years ago to get a sort of download manager in the node so we could have pins happen asynchronously, but there has been no movement on that at all. I'm glad it works well for you, but it has been nothing but pain for me. Hell, most of the time the gateway doesn't manage to discover files I have pinned on my local computer. I really want something like IPFS to succeed, because it has immense potential to literally change the world, but I can't even recommend that people run a node locally because I know it's going to eat their battery and slow their computer down. I don't know why these problems haven't gone away after years of work and millions in funding.

gioscarab 7y ago

I'm applying the local-first methodology in PJON, a new networking open standard : https://github.com/gioblu/PJON https://github.com/gioblu/PJON

amitport 7y ago

Hmm... homomorphic encryption.

davidkuennen 7y ago

Very cool. Let me add to all of these great examples in the comments. I published this app with focusing on exactly these points just two days ago. https://stockevents.app/ https://stockevents.app/ All tracked stocks stay within the app. You only pull information from the servers and store that information locally for offline use.

daleharvey 7y ago

I may be biased as the maintainer of PouchDB but you can do all this today (and for the last 5+ years) with PouchDB. The comment about CouchDB and the "difficulty of getting application-level conflict resolution right" I am not really certain how it applies, You dont have to handle conflicts in Pouch/CouchDB if you dont want to, there is a default model of last write (actually most edits) wins but you can handle them if needed

knubie 7y ago
I've been using a combo of PouchDB/CouchDB in my app[0] for the past few months, and I find it a hard combo to beat at the moment. I just haven't found anything else that works as seamlessly. While going through the article I found I was able to tick most of the boxes thanks to PouchDB. [0] https://mochi.cards/ https://mochi.cards/
deleted 7y ago
[deleted]
adamwiggins 7y ago
Hi Dale, I'm one of the local-first paper coauthors. I'm a fan of PouchDB (thanks for that) and the whole CouchDB lineage--the CouchDB book[1] was an early inspiration in my exploration of next-gen storage and collab models. I've been down the CouchDB/PouchDB path several times with several different engineering teams. Every time we were hopeful and every time we just couldn't get it to work. As one example, I worked with a small team of engineers to implement CouchDB syncing for the Clue Android and iOS mobile apps a few years back. Some of my experience is written down here[2]. After investing many months of engineering time, including some onsite help from Jan Lehnardt[3] we abandoned this architecture and went with a classic HTTPS/REST API. Other times and with different teams we've tried variations of Couch including PouchDB with web stack technology including Electron and PWA-ish HTML apps. None of these panned out either. Wish I could give better insights on why--we just can't get it to be reliable, or fast, or find a good way to share data between two users (the collaboration thing is kind of the whole point). [1]: https://guide.couchdb.org/ https://guide.couchdb.org/ [2]: https://medium.com/wandering-cto/mobile-syncing-with-couchbase-6f076d8c7e08 https://medium.com/wandering-cto/mobile-syncing-with-couchba... [3]: https://neighbourhood.ie/couchdb-support/ https://neighbourhood.ie/couchdb-support/

FpUser 7y ago

Article mentions the following tweet: What was the last app you used on your laptop that wasn't: Terminal VS Code Web Browser OS settings That guy must be living in a very limited / imaginary world world. I use boatload of local software

habnds 7y ago
Which are your favorites?
- FpUser 7y ago
  I do not have favorites. I use various tools for various activities. Mechanical design, electronics, software development, video, photo, data analysis etc. etc. Way too much to list all. Some apps are being used very rarely but I still need those when the time comes

uber-geek 7y ago

A more recent issue, this one with youtube banning google accounts for posting too many emoji's during a live stream,what if all of your business data was on your google account? You would be done unless you could get unbanned. There is something to be said for using local apps. The engineers I work with are often out of range of any network, so having apps that run on their laptops is crucial.

have_faith 7y ago

> you OWN YOUR data Like most people here I'm fairly hard line when it comes to personal data abuses but I still struggle with the concepts of owning data about yourself. It's a confusion I see amongst less technically literate people when a well meaning person explains to them the importance of some latest data breach and they try to understand the concept that they owned this data, it was theirs but now it has been "stolen" or abused in some way. I would consider going as far to say that framing the data as owned by you is a bad approach, but maybe I'm just being pedantic about the language. Company A does have data about me, but I don't own it, and they have responsibilities to protect it (or delete it if requested), but I don't see any ownership in the equation, especially when the nature of the data can become quite abstract while still maintaining some reference to you. Not to take away from the intention or sentiment of framing it that way though, I'm just musing.

gtirloni 7y ago
You could argue in the past that PII data was just name, birthday, SSN, etc. However, nowadays pretty much anyone can be identified by their metadata so that's where I think this broad sense of ownership comes from.
mrkeen 7y ago
Ownership is precisely the right word to use. Companies should possess your data, not own it.
SeanMacConMara 7y ago
consider the concept that lots of businesses operate perfectly fine whilst temporarily having full physical control over physcial objects owned by their customers why not treat data the same way ? yes it will be very disruptive to some businesses. i hope.
cgriswald 7y ago
I don't entirely agree. People may be confused about data, but they are not confused about ownership. They understand ownership quite well. By sticking with that framing, you only have to get them to understand this ephemeral thing. Once you do, they can immediately apply all their concepts of ownership to it. To replace ownership with something else, well, now you've still got to get them to grasp the data, but also this something else, which might necessarily be somewhat abstract. I'm not even quite convinced they can't grasp the concept. People think they 'own' their digital copy of The Rescuers Down Under, when technically they've leased it. They generally understand they shouldn't post their credit card numbers online. I think it's more likely that they just don't understand the consequences, and even if they did, generally feel dis-empowered to do anything about it. That's a dynamic that has existed since before the internet.

herf 7y ago

I spent a lot of time with photos (Picasa) trying to do this peer to peer - this is what we built in the 2002 era. Here are a few issues: 1. Identity is hard to do on the LAN, so any sharing ends up using the cloud to figure out who has access. Similarly, identity is hard to move around, so representing your Facebook comments feed outside Facebook is difficult to do. 2. Any time you have a "special" server that handles identity, merging, or any other task, it ends up in the effective role of master, even if the rest of the parts of your system are peers. You want all your collaborations in Dropbox to survive their infrastructure being offline? It's tough to do. 3. p2p stalled a bit in the mid-2000s when storage and bandwidth got much cheaper--in a period of just two years (2002-2004), it became 100x cheaper to run a cloud service. But what continued to stall p2p was mobile. Uploading and syncing needs to run in the background, and if you're on a limited bandwidth client or a battery-limited device like iOS, sync can effectively diverge for months because the applications can't run in the background. So changes you "thought" you made don't show up on other devices. 4. For avoiding mass surveillance, what we are missing from this older time is the ability to make point to point connections (between peers) and encrypt them with forward secrecy, without data at rest in the cloud. Even systems that try to do some encryption for data at rest (e.g., iMessage) keep keys essentially forever, so data can be decrypted if you recover a private key later on. A system that only makes direct connections between peers does not have this issue. 5. Anytime you have multiple peers, you have to worry about old versions (bugs), or even attackers in the network, so it's fundamentally harder than having a shared server that's always up to date and running the latest code.

koheripbal 7y ago
With regards to identity management, maybe there should be a formalized integration between browsers and password managers such that the concept of "registration" goes away and new logins just automatically create user accounts with default permissions, according to email address.
- mceachen 7y ago
  Centralized identity management was the grand promise of OAuth. It seems that somehow ended up, in most practical applications, as a "login with Facebook" button. There are a number of startups working on simplified or "passwordless" auth, but it seems that none have substantive traction. I'd love to be proven wrong, here, though!

wazoox 7y ago

I'm using NextCloud for that. Nextcloud can be basically used in local-first mode (mimicking dropbox). My phone automatically uploads all pictures to the NextCloud. Then there are apps. For instance I use Nextcloud with the Music app to stream my own mp3s from my Nextcloud to my phone running Ampache. There are also collaborative editing tools, and various options to edit all sort of documents in a web UI, and always the local editing fallback (or the opposite way, as you see fit).

Jareddd 7y ago

Awesome write-up! This brings me back to early 2000's when we we typically owned most of our graphic design software. Yes it was expensive but, there was barley any "cloud" features added and if you wanted a new version you simply had to download the update or simply keep the software you were currently using. My team and I have taken the initiative to offer a e mail design tool that is considered first-class software to the OS (https://bairmail.com https://bairmail.com). The last thing I would say is developing desktop apps vs a web app is considerably harder thus most companies are aware they are saving by controlling software updates, versioning etc.

lilactown 7y ago

I recently started using Actual[0], a local-first budgeting app. It's encrypted (soon to be end-to-end), local-first and can be synced across my devices. Would recommend, and I was inspired to use it after following its author, James Long[1] on twitter for awhile. [0] https://actualbudget.com/ https://actualbudget.com/ [1]https://twitter.com/jlongster?lang=en https://twitter.com/jlongster?lang=en

spieglt 7y ago

If you want an easy way to make a file (or .zip of files) more private before storing it in the cloud for backup/availability purposes, please check out Cloaker: github.com/spieglt/cloaker. It's simple, password-based, drag-and-drop file encryption.

pjkundert 7y ago

The Holo / Holochain project was founded with this principle as a primary goal: https://holo.host https://holo.host We’ll be implementing CRDTs soon, but the concept of local control of all data, authenticated and encrypted communications, etc. is implemented. One fundamental difference between apps that support this and those that cannot: agent-centric vs. data-centric design. Strangely, many “distributed” applications (eg. Bitcoin) didn’t make the “leap” to agent-centricity, and thus missed out on some key enabling optimizations. As a result — they are forced to implement “global consensus” (expensively), when they didn’t need to, to achieve their goals: a valid global ledger, in Bitcoin’s case. It turns out that, to implement things like cryptocurrencies, you don’t need everyone, everywhere to agree on a “total order” for every transaction in the entire network! Agent-centricity, FTW!

ph0rque 7y ago

That is my vision for https://edible.estate https://edible.estate, version 2 (version 1.0 should be done this year).

josefx 7y ago

I wrote documentation for configuration files in xsd with the idea that I could use xslt to display them in a browser on any system and use the same files for validation. This worked a year ago. Now browsers consider local files that access other local files suspect and will refuse to load anything unless beaten. So I now use a python script to run a simple local http server to view my local files from a single "origin". However http itself is already considered suspect and many claim it should be deprecated for https. In the future I will have to provide a lets encrypt signed https server with valid domain so anyone can view those files on their browser without having to mess with about:config settings or their own certificate cache. The cloud is the future, do not dare to build something that runs locally.

bouk 7y ago
http over localhost is considered secure by browsers
- josefx 7y ago
  Is there a documented guarantee of that somewhere? Because I am half tempted to make a browser based UI for a few tools, but would rather use Java if there is even a tiny risk that I get the rug pulled from under my feet again.

alleycat5000 7y ago

Reminds me a little of the Upspin project: https://upspin.io/ https://upspin.io/ At least some of the motivations are the same.

at-fates-hands 7y ago

Is this something like Blockstack could help with? https://blockstack.org/ https://blockstack.org/ I'm having a hard time understanding the differences between "Local-first technology" and something like Blockstack. I'm not saying BS completely solves the issues pointed out in the blog post, but it seems to me its pretty close. What do you think? here is a list of the current apps available: https://app.co/blockstack https://app.co/blockstack

brynb 7y ago

I've been working for a few months on a database called Redwood that's intended to make it easier to build this kind of software. Having spent much of the past couple of years working with libp2p, IPFS, Dat, and similar technologies, I was curious to see what would result if I started from the ground up. https://github.com/brynbellomy/redwood https://github.com/brynbellomy/redwood So far, the model seems promising. It's fully peer-to-peer, and supports decentralized identity, configurable conflict resolution, read/write access, asset storage, and currently is running across 3 different transports: - HTTP (augmented with extensions proposed by the Braid project [1][2]) - Libp2p - WebRTC I've included two simple demos, a collaborative document editor (well, it's just a textarea at the moment), and a chat app. Would appreciate any feedback or participation that folks are willing to give. [1] https://github.com/braid-work/braid-spec https://github.com/braid-work/braid-spec [2] https://groups.google.com/forum/#!forum/braid-http https://groups.google.com/forum/#!forum/braid-http

oblib 7y ago

I've been building apps that meet this criteria for several years now. It's nice to see the concept getting some attention here. The only thing I'll point out is that CouchDB getting rated "partially meets the idea" seems pretty weak to me. They reference v2.1 but the latest version is 2.3.1 and here's link to the docs on how conflict resolution is dealt with: https://docs.couchdb.org/en/latest/replication/conflicts.html https://docs.couchdb.org/en/latest/replication/conflicts.htm... If finer grained control is needed it would be up to the developer to implement it, and it really shouldn't be difficult to do that. In my case, I use PouchDB to perform a "live sync" with all connected users so they all get the latest updates to a document. If a conflict arises it's easy for any one of the users to fix and push it to everyone connected.

hwwang2005 7y ago

I'm creator of BLOON (https://www.bloon.io https://www.bloon.io). If BLOON had been put in the Table 1 in the paper, the values could be: O O - - O - O I think BLOON is closer than Git+GitHub. And great article & paper! They do give us many inspiration about improving BLOON. Thank you!

garfieldnate 7y ago

This. I want this. It happens to me quite often that I feel demotivated or uneasy about using some software because I know I'm producing valuable data specific to me that I could use in the future, but the product stipulates that I don't get to keep the data. iOS apps are often the worst at this. There's no filesystem, so all data has to be kept with an app. I loaded all of the books I wanted to read into iBooks, and then it turned out that the iOS backup skips books that you didn't buy from Apple. Bye bye book collection T_T. Some of this could also be alleviated by Tim Berners-Lee's pod idea: https://solid.inrupt.com/ https://solid.inrupt.com/. But local-first is better. I just want files on my machine.

jefurii 7y ago

I'm surprised nobody has mentioned using git-annex (and Git of course) to manage data (full disclosure: I develop an archival management app which keeps its data in git/git-annex repositories). Of the seven key properties, git-annex gives you 1,2,3,4 and supports 5 and 7; 6 depends on how you store things. Git-annex supports identity in the sense that each clone of the repository has a UUID. You can choose to have a central hub if you like but you don't have to (surprise! you don't really need GitHub!). It comes with caveats of course: Binaries can be synced but only textual data is actually versioned, and once you put something in it will always be there unless you use something like git-filter-branch.

shreyshrey 7y ago

Tonido (https://www.tonido.com https://www.tonido.com) is a good local-first personal cloud. It allows you to access, share files stored in your PC from anywhere. We are giving it free for almost 10 years. The longevity is higher than many cloud services :). You own your data.

chaz6 7y ago

IPv6 will make it easier to access your data anywhere. I am putting this here because so far nobody has mentioned it.