12 ms·

A Local-First Case Study

201 points by paulgb 2y ago

com 2y ago
This is great! I was quite excited to see Ink & Switch’s Embark and now this… Jake makes creating a local-first multiplayer app seem so simple.
xrd 2y ago
Ink & Switch is like the Medici family of the Internet era. Medici's funded the piano, and I&S is funding local-first. I love what they do.
- ilrwbwrkhv 2y ago
  Who is funding them?
  - xrd 2y ago
    Lots of ex-twitter and heroku people. They are really thoughtful and community based and are not in silicon valley physically so they have broken out of the groupthink. I met quite a few at the last Strange Loop and it's a really terrific group.
    - ilrwbwrkhv 2y ago
      Interesting. We definitely need more groups like them. This would be the right way of moving tech forward.
er4hn 2y ago
> Architecturally, Y-Sweet acts as a bus: clients connect to the Y-Sweet server rather than directly to each other. Whenever a client connects or makes changes, it syncs its local document with the Y-Sweet server. Y-Sweet merges the client’s document into its own copy, saves it to S3 and broadcasts updates to other clients. Since CRDTs are guaranteed to eventually converge on the same state, at the end of this process all clients have the same document. I had thought that the advantage of CRDTs was you do not need a centralized server and that if you do have a central server Operational Transforms are easier. Am I missing why CRDTs are used here?
- paulgb 2y ago
  (One of the authors of Y-Sweet) You’re right, that is one of the advantages of CRDTs, but it turns out to be hard to realize on the web — aside from RTC (which has its own dragons), you still need a server in the mix. The other thing an authoritative server solves is persisting the data. Because one server is the authority for a document at a time, you can use S3 or R2 for persistence without worrying about different servers with different versions of the document colliding and erasing each other’s changes.
  - er4hn 2y ago
    Oh, this is interesting. Can you elaborate (or link to some post) about what sort of issues you run into? I find the concept of CRDTs to be very interesting, but if you still need a centralized server I question the value of them over OTs. I'd love to understand more about if this is a connectivity issue, a CRDT issue, or what.
    - paulgb 2y ago
      The main issue you run into is that the web is designed around client-server communication; you can do peer-to-peer communication in theory but because of NAT and firewalls, many of those end up not being peer to peer in the end. Plus, if you want the data to persist so that two people can collaborate even if they are never online at the same time, you need a server anyway. CRDTs as data structures support peer-to-peer, it’s just that in many use cases that aspect of CRDTs is not needed.
      - er4hn 2y ago
        Okay that all makes a lot of sense thank you.
- com 2y ago
  I think that using a bus decomplicates the connectivity story - having a “cloud peer” removes quite a bit of coding and testing from implementing true peer to peer discovery and communications functionality. Bonus points: you could potentially rip out the bus and replace it with something that involves peer to peer connectivity without changing client data structures.
er4hn 2y ago
On an unrelated note, I'm getting local-first case studies everytime I leave a wifi space since my cell phone is still suffering from the Verizon outage (Los Angeles area). My conclusion has been I can read stuff like calendar invites (Google), I can save notes on stuff to do in Trello, but I cannot queue up IMs to send in GHC.
- raybb 2y ago
  I kinda doubt GHC stands for GitHub Comments but if it does I just gotta say thank goodness GitHub started saving comments in local cache so when you accidentally pull to refresh you don't delete your whole comment. Idk when they added this but must have been in the past 6 months or so. Same applies if you try to submit a comment and then are offline.
NeutralForest 2y ago
I've been hanging local-first circles but haven't made to switch to write anything with it yet. Is there a typical stack people recommend to get started? I'm not really sure where to start, especially in terms of backend.
- paulgb 2y ago
  I’m biased as one of the authors of Y-Sweet, but I think Jake’s choice of Yjs + Y-Sweet is as good as any to start with. We’ve worked a lot at making the localhost environment easy to get started with, and Yjs is the de facto CRDT with the most written about it. https://github.com/jamsocket/y-sweet https://github.com/jamsocket/y-sweet
  - NeutralForest 2y ago
    I'd like to start with something that's more DB oriented like SQLite both locally and in the browser for example but I understand a documented-oriented approach might make more sense.
    - paulgb 2y ago
      In that case, there are a number of recent options. https://www.instantdb.com/ https://www.instantdb.com/ and https://electric-sql.com/ https://electric-sql.com/ come to mind. I wrote about a number of them here: https://digest.browsertech.com/archive/browsertech-digest-trip-report-local-first-conf/ https://digest.browsertech.com/archive/browsertech-digest-tr...
      - NeutralForest 2y ago
        Thank you!
    - ochiba 2y ago
      There is a fairly comprehensive list of frameworks/tools here: https://localfirstweb.dev/ https://localfirstweb.dev/ I work on PowerSync and we did a Show HN last year: https://news.ycombinator.com/item?id=38473743 https://news.ycombinator.com/item?id=38473743 Also see InstantDB Show HN: https://news.ycombinator.com/item?id=41322281 https://news.ycombinator.com/item?id=41322281
xnx 2y ago
Is there some version of local-first that doesn't require a webserver, but does seamlessly sync state to a consumer cloud service like Google Drive? I'd love to write apps that have all the speed and portability of local apps, but the data isn't tied to a specific device. It seems like it would be feasible to have a large JSON blog background synced to a cloud file service after some threshold of accumulated change or time.
- michaelmure 2y ago
  Not a file storage but https://github.com/git-bug/git-bug https://github.com/git-bug/git-bug push and sync with any git remote. There is a generic data structure you can use to build your conflict-free type.
- ochiba 2y ago
  This made the rounds on HN recently: https://tonsky.me/blog/crdt-filesync/ https://tonsky.me/blog/crdt-filesync/ A PoC of using Dropbox for a local-first app
- chris_pie 2y ago
  https://remotestorage.io https://remotestorage.io while it's a specific protocol for storing user data on a compatible server, their library also provides Google Drive integration
- felipefar 2y ago
  There's already been some talk of using consumer cloud providers. One shortcoming however is that to work well you have to duplicate the users' data for each client app, so you'll consume x times more storage space of the user. This is fine for apps with little data, but is impractical for other apps.
- yunusefendi52 2y ago
  I built my habit tool app using Google Drive (app-spesific directory), it can sync between device, though it has a bug (in my code) sometimes cannot sync some data so I have to resync everything from scratch. No server needed. Currently android only, don't have ios dev subscription https://play.google.com/store/apps/details?id=com.yedev.habittrackertool https://play.google.com/store/apps/details?id=com.yedev.habi...
wonger_ 2y ago
I love how the map automatically updates based on the places typed in the editor. A great visual aid to a text-based workflow. I got confused by this comment though: > To determine when to re-render, “reactive” frameworks like Svelte and Solid track property access using Proxies, whereas “immutable” frameworks like React rely on object identity. I thought React was just as reactive as all the other JS frameworks, and that the state/setState code would look similar.
- jakelazaroff 2y ago
  Author here! Maybe I could have worded that better — basically, when you call setState in React, it compares the identities of the new state and old state to determine whether to re-render. Svelte and Solid use “signals” to automatically determine when to re-render, with the drawback that you’re no longer interacting with the “raw” value but a Proxy. But neither way would be able to detect Yjs mutating its internal state; you’re correct that even in React you would need some sort of wrapper code “tricking” it into re-rendering when appropriate.
  - jayunit 2y ago
    I’ve enjoyed using https://syncedstore.org/docs/ https://syncedstore.org/docs/ as such wrapper code for Yjs in React. They have a Svelte lib too, haven’t tried it. Great article, congrats on releasing!
- IggleSniggle 2y ago
  I haven't worked frontend in a long time but just from reading that snippet I would assume React does a `obj===obj` (identity) comparison for state update under the hood, while Svelte/Solid are doing a Proxy trap on the accessors like `parent.obj =` would result in an intercept fn being called when you hit that 'obj' access. Proxy being a little more complicated to reason about and setup in totality but a lot more powerful and flexible.
- jazzypants 2y ago
  React is not "reactive" according to the classic definition, but even the creator of Solid.js thinks that doesn't really matter. https://dev.to/this-is-learning/how-react-isn-t-reactive-and-why-you-shouldn-t-care-152m https://dev.to/this-is-learning/how-react-isn-t-reactive-and...
mentalgear 2y ago
The future is local-first. IF you haven't yet learned about it and how to break-out of the expensive cloud/serverless cage, this is a good start: https://localfirstweb.dev/ https://localfirstweb.dev/
k__ 2y ago
Awesome stuff. Reminds me of the offline-first movement from ~10 years ago. I'm currently looking into TinyBase to make working with high latency decentralised services more bearable. Would be cool if there were a better comparison of the different solutions for storage and sync, with features, good use-cases, etc.
tobyhinloopen 2y ago
That’s a nice font
RobCodeSlayer 2y ago
This is awesome! Since you’re using CRDTs, do you have any plans to make it collaborative? I would find it useful to build an itinerary with multiple people
- jakelazaroff 2y ago
  It is collaborative! Start a document and send someone else the link :)
catchmeifyoucan 2y ago
Great write-up, I've been looking for a solution like this for adding syncing to my local-app! I love that it's document stored in S3, and it's probably going to be way cheaper than if hosted elsewhere in a database. Can't wait to try it out soon
tkiolp4 2y ago
Would be perfect is somehow it could work without S3. Would be awesome if the internet could just work in p2p mode out of the box, just some JS and HTML and you have 2 computers talking to each other collaborating on a doc without the need of a server (or S3)
- josephg 2y ago
  You often still want a server somewhere to relay messages. Imagine I type something in my computer at home and turn the device off before leaving the house. Then on the road I pull up the document on my phone. I should see the latest version of the document on my phone. However, there were no moments where both devices were online at the same time. For this to work, my home computer needs to upload the changes somewhere my phone can access them. For example, a home server or a dumb box in the cloud. It’s very difficult to make this work without a server kicking around somewhere. So long as the server is fungible (it can easily be replaced for basically any other server), I don’t really see the problem with keeping a server around to relay messages.
braden-lk 2y ago
We built LegendKeeper using Yjs! (It's a mapping app as well, but fantasy). Ended up rolling our own sync server to handle large scale multiplexing, as we have D&D game-masters with 25,000+ documents to manage. (I don't know how they do it, tbh!) We opt for the central server as a super-peer and use the Yjs differential update system to avoid loading docs in memory for too long. While there are many things about local-first that are a huge pain in the ass, the UX benefits are pretty huge. The DX can be nice too! Getting to focus on product and not on data transit (once you've got a robust sync system) is pretty sweet. The first 4 weeks of launching our Yjs-based system was rough though; lots of bugs that virally replicated between peers. It requires a really paranoid eye for defensive coding; after several years, we have multiple layers of self-healing and validation on the client.
- jakelazaroff 2y ago
  I think a lot of people — including myself — would be very interested in a longer write-up of that system if you're ever interested in sharing more :)
- necrodome 2y ago
  I feel like we’re on the verge of seeing the start of a whole new wave of local-first apps (and "personal software" as mentioned here https://x.com/rauchg/status/1840293374059839726 https://x.com/rauchg/status/1840293374059839726), but we’re really missing Rails-like frameworks that offer a complete package for development. Something that gives you all the tools you need—from syncing, conflict resolution, state management, authorization, background jobs in the context of local-first to deployment—without having to reinvent the wheel. I built a simple SaaS [1] to get a sense of what's missing and while React Router + a syncing local first database [2] + $5/month Cloudflare gets you pretty far, I still found myself needing to think through a lot of pieces [1] https://usequickcheck.com/ https://usequickcheck.com/ [2] https://fireproof.storage/ https://fireproof.storage/
  - halfcat 2y ago
    Zero [0] is coming, from the people that made Replicache [0] https://zerosync.dev/ https://zerosync.dev/
dtkav 2y ago
For those interested in this stack,I have been working on an Obsidian.md plugin called Relay that makes it fully collaborative using yjs and y-sweet. We also use a hub and spoke model, but we still rely on a central server (pocketbase) for management user flows like authorization and billing. Obsidian is such a fantastic editor, and it fits so naturally with local-first collaboration.
felipefar 2y ago
I'm excited by these explorations of dynamic components in rich text. Notion has popularized the idea of documents with rich blocks, where the blocks can provide dynamic behavior to traditional documents. And now we're seeing types of inline elements that also provide more structure to rich text. Those location routes seem something that I'd use myself.
swiftcoder 2y ago
This is a gorgeous little app, and I'm really happy to see someone working on rich linking of map data and content - it feels like a really under-developed area of the whole modern maps ecosystem
dustingetz 2y ago
Inability to scale with sustained usage (1+ person*year of data) is the fatal problem with this category in existing approaches. Root of this is primarily the “partial sync problem” - when dataset outgrows both the memory and compute resources available in the client device (which is unreliable, not under your control to make reliable, and resource constrained - and not everybody has the latest giga device), you have to somehow decide which working set to replicate. If the app structure is a graph and involves relational queries, this is undecidable without actually running the queries! If the app structure is constrained to topic documents, you still have to choose which topics to cache and it still falls over at document sizes easily seen in prolonged solo use let alone use at an enterprise which market is necessary to justify VC investment. All this in an increasingly connected world where the cloud is 10ms away from city/enterprise users (the $$$) and any offline periods (subway etc) resolve quickly, so a degraded offline mode (letting you edit whatever you were last looking at w/o global relational query etc) is often acceptable. Oh, and does your app need AI? Change my view!
- ForHackernews 2y ago
  Extremely curious what kinds of data your app would produce such that it outgrows the memory available on a single client device. I have about 2 person-decades' worth of writing, art, and coding that occupies less than 10 GB (and could probably be made smaller).
  - dustingetz 2y ago
    I'm not talking about media content, rather database records. PKM apps host in-process relational queries, the records must fit in working memory and query engine must traverse the working set indexes to return answers "instantly"
    - ForHackernews 2y ago
      Again, I guess I'm struggling to imagine what kind of database your app would need that doesn't fit in a 20 Mb sqlite file? What are all these jillions of records? You're talking about full-text indexing?
foobarbecue 2y ago
I've been pretty impressed with https://roadtrippers.com https://roadtrippers.com for this purpose. (Not local-first, I guess)
gspencley 2y ago
It's funny how this newfangled "local first" thing is what us old fogies used to call just "applications." I get that there is networking and integration that a modern application will typically need to do (depending on its core purpose), and syncing state to and from servers is a special concern (especially if conflict management is necessary) that native desktop applications rarely had to do in years past. But at the end of the day, it sure does feel like we've come full circle. For a long time, every single application was "local first" by default. And now we're writing research papers and doing side POCs (I'm speaking generally, nothing to do with the author or their article) trying to figure out how to implement these things.
- matthiaswh 2y ago
  I feel this way at times too, but the ability to sync state really is key. Traditional native desktop apps rarely had to consider this because users would only ever interact on one device. Now users are moving between a computer and a phone, if not more devices. If I were to frame it succinctly I would say applications = local-only which is distinct from local-first.
  - gspencley 2y ago
    Agreed but even data sync wasn't unheard of before everything became a "web application." I mean not to get really low level or abstract, but there's a reason that operating systems have the concept of a virtual file system. Where and how your data is persisted is something that can be abstracted from the rest of the system. Add CRDT or another conflict resolution solution to that layer and, I don't want to pretend that it's simple, far from it ... but it's not new tech is all I'm really saying. Distributed systems were a very hot topic in the 90s. We even went through one very awkward and short-lived fad where we toyed with the idea of having a single application distributed across a network of computers ... not to persist data across multiple systems per se but for computation performance. This entire concept got abandoned when people realized how unnecessarily complicated it was (all cost / very little reward). But it was a thing for a while. Even RAID distributes data persistence across physical devices and needs to care about data integrity as a result. It just seems like the longer I'm in the industry, the more I realize that there is very little that is actually new ... despite the fact that we have a large number of enthusiastic young software developers who are looking at these concepts with starry eyes and youthful ignorance because they weren't around decades ago when this stuff was being researched, developed and experimented with.

201 points by paulgb 2y ago

com 2y ago

This is great! I was quite excited to see Ink & Switch’s Embark and now this… Jake makes creating a local-first multiplayer app seem so simple.

xrd 2y ago

Ink & Switch is like the Medici family of the Internet era. Medici's funded the piano, and I&S is funding local-first. I love what they do.

ilrwbwrkhv 2y ago
Who is funding them?
- xrd 2y ago
  Lots of ex-twitter and heroku people. They are really thoughtful and community based and are not in silicon valley physically so they have broken out of the groupthink. I met quite a few at the last Strange Loop and it's a really terrific group.
  - ilrwbwrkhv 2y ago
    Interesting. We definitely need more groups like them. This would be the right way of moving tech forward.

er4hn 2y ago

> Architecturally, Y-Sweet acts as a bus: clients connect to the Y-Sweet server rather than directly to each other. Whenever a client connects or makes changes, it syncs its local document with the Y-Sweet server. Y-Sweet merges the client’s document into its own copy, saves it to S3 and broadcasts updates to other clients. Since CRDTs are guaranteed to eventually converge on the same state, at the end of this process all clients have the same document. I had thought that the advantage of CRDTs was you do not need a centralized server and that if you do have a central server Operational Transforms are easier. Am I missing why CRDTs are used here?

paulgb 2y ago
(One of the authors of Y-Sweet) You’re right, that is one of the advantages of CRDTs, but it turns out to be hard to realize on the web — aside from RTC (which has its own dragons), you still need a server in the mix. The other thing an authoritative server solves is persisting the data. Because one server is the authority for a document at a time, you can use S3 or R2 for persistence without worrying about different servers with different versions of the document colliding and erasing each other’s changes.
- er4hn 2y ago
  Oh, this is interesting. Can you elaborate (or link to some post) about what sort of issues you run into? I find the concept of CRDTs to be very interesting, but if you still need a centralized server I question the value of them over OTs. I'd love to understand more about if this is a connectivity issue, a CRDT issue, or what.
  - paulgb 2y ago
    The main issue you run into is that the web is designed around client-server communication; you can do peer-to-peer communication in theory but because of NAT and firewalls, many of those end up not being peer to peer in the end. Plus, if you want the data to persist so that two people can collaborate even if they are never online at the same time, you need a server anyway. CRDTs as data structures support peer-to-peer, it’s just that in many use cases that aspect of CRDTs is not needed.
    - er4hn 2y ago
      Okay that all makes a lot of sense thank you.
com 2y ago
I think that using a bus decomplicates the connectivity story - having a “cloud peer” removes quite a bit of coding and testing from implementing true peer to peer discovery and communications functionality. Bonus points: you could potentially rip out the bus and replace it with something that involves peer to peer connectivity without changing client data structures.

er4hn 2y ago

On an unrelated note, I'm getting local-first case studies everytime I leave a wifi space since my cell phone is still suffering from the Verizon outage (Los Angeles area). My conclusion has been I can read stuff like calendar invites (Google), I can save notes on stuff to do in Trello, but I cannot queue up IMs to send in GHC.

raybb 2y ago
I kinda doubt GHC stands for GitHub Comments but if it does I just gotta say thank goodness GitHub started saving comments in local cache so when you accidentally pull to refresh you don't delete your whole comment. Idk when they added this but must have been in the past 6 months or so. Same applies if you try to submit a comment and then are offline.

NeutralForest 2y ago

I've been hanging local-first circles but haven't made to switch to write anything with it yet. Is there a typical stack people recommend to get started? I'm not really sure where to start, especially in terms of backend.

paulgb 2y ago
I’m biased as one of the authors of Y-Sweet, but I think Jake’s choice of Yjs + Y-Sweet is as good as any to start with. We’ve worked a lot at making the localhost environment easy to get started with, and Yjs is the de facto CRDT with the most written about it. https://github.com/jamsocket/y-sweet https://github.com/jamsocket/y-sweet
- NeutralForest 2y ago
  I'd like to start with something that's more DB oriented like SQLite both locally and in the browser for example but I understand a documented-oriented approach might make more sense.
  - paulgb 2y ago
    In that case, there are a number of recent options. https://www.instantdb.com/ https://www.instantdb.com/ and https://electric-sql.com/ https://electric-sql.com/ come to mind. I wrote about a number of them here: https://digest.browsertech.com/archive/browsertech-digest-trip-report-local-first-conf/ https://digest.browsertech.com/archive/browsertech-digest-tr...
    - NeutralForest 2y ago
      Thank you!
  - ochiba 2y ago
    There is a fairly comprehensive list of frameworks/tools here: https://localfirstweb.dev/ https://localfirstweb.dev/ I work on PowerSync and we did a Show HN last year: https://news.ycombinator.com/item?id=38473743 https://news.ycombinator.com/item?id=38473743 Also see InstantDB Show HN: https://news.ycombinator.com/item?id=41322281 https://news.ycombinator.com/item?id=41322281

xnx 2y ago

Is there some version of local-first that doesn't require a webserver, but does seamlessly sync state to a consumer cloud service like Google Drive? I'd love to write apps that have all the speed and portability of local apps, but the data isn't tied to a specific device. It seems like it would be feasible to have a large JSON blog background synced to a cloud file service after some threshold of accumulated change or time.

michaelmure 2y ago
Not a file storage but https://github.com/git-bug/git-bug https://github.com/git-bug/git-bug push and sync with any git remote. There is a generic data structure you can use to build your conflict-free type.
ochiba 2y ago
This made the rounds on HN recently: https://tonsky.me/blog/crdt-filesync/ https://tonsky.me/blog/crdt-filesync/ A PoC of using Dropbox for a local-first app
chris_pie 2y ago
https://remotestorage.io https://remotestorage.io while it's a specific protocol for storing user data on a compatible server, their library also provides Google Drive integration
felipefar 2y ago
There's already been some talk of using consumer cloud providers. One shortcoming however is that to work well you have to duplicate the users' data for each client app, so you'll consume x times more storage space of the user. This is fine for apps with little data, but is impractical for other apps.
yunusefendi52 2y ago
I built my habit tool app using Google Drive (app-spesific directory), it can sync between device, though it has a bug (in my code) sometimes cannot sync some data so I have to resync everything from scratch. No server needed. Currently android only, don't have ios dev subscription https://play.google.com/store/apps/details?id=com.yedev.habittrackertool https://play.google.com/store/apps/details?id=com.yedev.habi...

wonger_ 2y ago

I love how the map automatically updates based on the places typed in the editor. A great visual aid to a text-based workflow. I got confused by this comment though: > To determine when to re-render, “reactive” frameworks like Svelte and Solid track property access using Proxies, whereas “immutable” frameworks like React rely on object identity. I thought React was just as reactive as all the other JS frameworks, and that the state/setState code would look similar.

jakelazaroff 2y ago
Author here! Maybe I could have worded that better — basically, when you call setState in React, it compares the identities of the new state and old state to determine whether to re-render. Svelte and Solid use “signals” to automatically determine when to re-render, with the drawback that you’re no longer interacting with the “raw” value but a Proxy. But neither way would be able to detect Yjs mutating its internal state; you’re correct that even in React you would need some sort of wrapper code “tricking” it into re-rendering when appropriate.
- jayunit 2y ago
  I’ve enjoyed using https://syncedstore.org/docs/ https://syncedstore.org/docs/ as such wrapper code for Yjs in React. They have a Svelte lib too, haven’t tried it. Great article, congrats on releasing!
IggleSniggle 2y ago
I haven't worked frontend in a long time but just from reading that snippet I would assume React does a `obj===obj` (identity) comparison for state update under the hood, while Svelte/Solid are doing a Proxy trap on the accessors like `parent.obj =` would result in an intercept fn being called when you hit that 'obj' access. Proxy being a little more complicated to reason about and setup in totality but a lot more powerful and flexible.
jazzypants 2y ago
React is not "reactive" according to the classic definition, but even the creator of Solid.js thinks that doesn't really matter. https://dev.to/this-is-learning/how-react-isn-t-reactive-and-why-you-shouldn-t-care-152m https://dev.to/this-is-learning/how-react-isn-t-reactive-and...

mentalgear 2y ago

The future is local-first. IF you haven't yet learned about it and how to break-out of the expensive cloud/serverless cage, this is a good start: https://localfirstweb.dev/ https://localfirstweb.dev/

k__ 2y ago

Awesome stuff. Reminds me of the offline-first movement from ~10 years ago. I'm currently looking into TinyBase to make working with high latency decentralised services more bearable. Would be cool if there were a better comparison of the different solutions for storage and sync, with features, good use-cases, etc.

tobyhinloopen 2y ago

That’s a nice font

RobCodeSlayer 2y ago

This is awesome! Since you’re using CRDTs, do you have any plans to make it collaborative? I would find it useful to build an itinerary with multiple people

jakelazaroff 2y ago
It is collaborative! Start a document and send someone else the link :)

catchmeifyoucan 2y ago

Great write-up, I've been looking for a solution like this for adding syncing to my local-app! I love that it's document stored in S3, and it's probably going to be way cheaper than if hosted elsewhere in a database. Can't wait to try it out soon

tkiolp4 2y ago

Would be perfect is somehow it could work without S3. Would be awesome if the internet could just work in p2p mode out of the box, just some JS and HTML and you have 2 computers talking to each other collaborating on a doc without the need of a server (or S3)

josephg 2y ago
You often still want a server somewhere to relay messages. Imagine I type something in my computer at home and turn the device off before leaving the house. Then on the road I pull up the document on my phone. I should see the latest version of the document on my phone. However, there were no moments where both devices were online at the same time. For this to work, my home computer needs to upload the changes somewhere my phone can access them. For example, a home server or a dumb box in the cloud. It’s very difficult to make this work without a server kicking around somewhere. So long as the server is fungible (it can easily be replaced for basically any other server), I don’t really see the problem with keeping a server around to relay messages.

braden-lk 2y ago

We built LegendKeeper using Yjs! (It's a mapping app as well, but fantasy). Ended up rolling our own sync server to handle large scale multiplexing, as we have D&D game-masters with 25,000+ documents to manage. (I don't know how they do it, tbh!) We opt for the central server as a super-peer and use the Yjs differential update system to avoid loading docs in memory for too long. While there are many things about local-first that are a huge pain in the ass, the UX benefits are pretty huge. The DX can be nice too! Getting to focus on product and not on data transit (once you've got a robust sync system) is pretty sweet. The first 4 weeks of launching our Yjs-based system was rough though; lots of bugs that virally replicated between peers. It requires a really paranoid eye for defensive coding; after several years, we have multiple layers of self-healing and validation on the client.

jakelazaroff 2y ago
I think a lot of people — including myself — would be very interested in a longer write-up of that system if you're ever interested in sharing more :)
necrodome 2y ago
I feel like we’re on the verge of seeing the start of a whole new wave of local-first apps (and "personal software" as mentioned here https://x.com/rauchg/status/1840293374059839726 https://x.com/rauchg/status/1840293374059839726), but we’re really missing Rails-like frameworks that offer a complete package for development. Something that gives you all the tools you need—from syncing, conflict resolution, state management, authorization, background jobs in the context of local-first to deployment—without having to reinvent the wheel. I built a simple SaaS [1] to get a sense of what's missing and while React Router + a syncing local first database [2] + $5/month Cloudflare gets you pretty far, I still found myself needing to think through a lot of pieces [1] https://usequickcheck.com/ https://usequickcheck.com/ [2] https://fireproof.storage/ https://fireproof.storage/
- halfcat 2y ago
  Zero [0] is coming, from the people that made Replicache [0] https://zerosync.dev/ https://zerosync.dev/

dtkav 2y ago

For those interested in this stack,I have been working on an Obsidian.md plugin called Relay that makes it fully collaborative using yjs and y-sweet. We also use a hub and spoke model, but we still rely on a central server (pocketbase) for management user flows like authorization and billing. Obsidian is such a fantastic editor, and it fits so naturally with local-first collaboration.

felipefar 2y ago

I'm excited by these explorations of dynamic components in rich text. Notion has popularized the idea of documents with rich blocks, where the blocks can provide dynamic behavior to traditional documents. And now we're seeing types of inline elements that also provide more structure to rich text. Those location routes seem something that I'd use myself.

swiftcoder 2y ago

This is a gorgeous little app, and I'm really happy to see someone working on rich linking of map data and content - it feels like a really under-developed area of the whole modern maps ecosystem

dustingetz 2y ago

Inability to scale with sustained usage (1+ person*year of data) is the fatal problem with this category in existing approaches. Root of this is primarily the “partial sync problem” - when dataset outgrows both the memory and compute resources available in the client device (which is unreliable, not under your control to make reliable, and resource constrained - and not everybody has the latest giga device), you have to somehow decide which working set to replicate. If the app structure is a graph and involves relational queries, this is undecidable without actually running the queries! If the app structure is constrained to topic documents, you still have to choose which topics to cache and it still falls over at document sizes easily seen in prolonged solo use let alone use at an enterprise which market is necessary to justify VC investment. All this in an increasingly connected world where the cloud is 10ms away from city/enterprise users (the $$$) and any offline periods (subway etc) resolve quickly, so a degraded offline mode (letting you edit whatever you were last looking at w/o global relational query etc) is often acceptable. Oh, and does your app need AI? Change my view!

ForHackernews 2y ago
Extremely curious what kinds of data your app would produce such that it outgrows the memory available on a single client device. I have about 2 person-decades' worth of writing, art, and coding that occupies less than 10 GB (and could probably be made smaller).
- dustingetz 2y ago
  I'm not talking about media content, rather database records. PKM apps host in-process relational queries, the records must fit in working memory and query engine must traverse the working set indexes to return answers "instantly"
  - ForHackernews 2y ago
    Again, I guess I'm struggling to imagine what kind of database your app would need that doesn't fit in a 20 Mb sqlite file? What are all these jillions of records? You're talking about full-text indexing?

foobarbecue 2y ago

I've been pretty impressed with https://roadtrippers.com https://roadtrippers.com for this purpose. (Not local-first, I guess)

gspencley 2y ago

It's funny how this newfangled "local first" thing is what us old fogies used to call just "applications." I get that there is networking and integration that a modern application will typically need to do (depending on its core purpose), and syncing state to and from servers is a special concern (especially if conflict management is necessary) that native desktop applications rarely had to do in years past. But at the end of the day, it sure does feel like we've come full circle. For a long time, every single application was "local first" by default. And now we're writing research papers and doing side POCs (I'm speaking generally, nothing to do with the author or their article) trying to figure out how to implement these things.

matthiaswh 2y ago
I feel this way at times too, but the ability to sync state really is key. Traditional native desktop apps rarely had to consider this because users would only ever interact on one device. Now users are moving between a computer and a phone, if not more devices. If I were to frame it succinctly I would say applications = local-only which is distinct from local-first.
- gspencley 2y ago
  Agreed but even data sync wasn't unheard of before everything became a "web application." I mean not to get really low level or abstract, but there's a reason that operating systems have the concept of a virtual file system. Where and how your data is persisted is something that can be abstracted from the rest of the system. Add CRDT or another conflict resolution solution to that layer and, I don't want to pretend that it's simple, far from it ... but it's not new tech is all I'm really saying. Distributed systems were a very hot topic in the 90s. We even went through one very awkward and short-lived fad where we toyed with the idea of having a single application distributed across a network of computers ... not to persist data across multiple systems per se but for computation performance. This entire concept got abandoned when people realized how unnecessarily complicated it was (all cost / very little reward). But it was a thing for a while. Even RAID distributes data persistence across physical devices and needs to care about data integrity as a result. It just seems like the longer I'm in the industry, the more I realize that there is very little that is actually new ... despite the fact that we have a large number of enthusiastic young software developers who are looking at these concepts with starry eyes and youthful ignorance because they weren't around decades ago when this stuff was being researched, developed and experimented with.