4 ms·

I'm not talking about media content, rather database records. PKM apps host in-process relational queries, the records must fit in working memory and query engi

I'm not talking about media content, rather database records. PKM apps host in-process relational queries, the records must fit in working memory and query engine must traverse the working set indexes to return answers "instantly"

ForHackernews 2y ago
Again, I guess I'm struggling to imagine what kind of database your app would need that doesn't fit in a 20 Mb sqlite file? What are all these jillions of records? You're talking about full-text indexing?
- dustingetz 2y ago
  PKM apps are trees of strings! It's fast until its not. Even if you can sync the global dataset to the device storage, the query engine needs data in process memory and still has to traverse it with device levels of compute not cloud compute, "instantly" i.e. without making the UI feel sluggish. If you feel otherwise, use Roam or Tana for a year, even in single-player mode. The entire category is bottlenecked on this scale problem. And now add team support, because you want to sell this to teams and make money, right? Designing for casual, personal-sized datasets is a viable architecture in very few apps. Google Maps is one shining counterpoint, because the content has a natural locality to it – you only need to sync content near where you are geographically!
  - paulgb 2y ago
    I think you’re misunderstanding the overall architecture here. Instead of syncing the whole tree of strings, the way you would generally represent a PKM with Yjs is to make each logical document a Yjs document (especially given the assumption that offline periods are short.) You could still build a server-side search index over those documents, which never needs to be sent to the client.
    - dustingetz 2y ago
      this gives up relational query, i.e. you knowledge graph is no longer a graph. Notion, Roam and Tana all require relational query. What real world app category are you attempting to model that matches document structure? If the domain can be modeled as topic documents and the documents are small, like an individual google doc, sure this set of constraints may be useful. But that does not match PKM!
      - paulgb 2y ago
        It doesn’t give it up, it just moves that aspect to the server. The server can still build an index over the documents in whatever way it likes, perform expensive queries, and only send the results to the client. An example that matches that document structure is Figma; each document is individually small enough to be synced with the client, document metadata is indexed on the server, and queries over documents take place on the server.
        dustingetz 2y ago
        This (moving relational ops to server) does not match the definition of local first provided in the article and gives up most of the value prop the article enumerates in the conclusion. I agree that Figma is a candidate to be implemented primarily as a document CRDT, same as the google doc example i already provided.
        paulgb 2y ago
        Well, the problem an index over multiple documents solves for is also not present in the application presented by the article. The plan for an individual trip (or even a lifetime of trips for most people) is not going to exceed a size that can be handled and indexed on the client.
        dustingetz 2y ago
        You're right, the approach totally works for PKM apps with less than, say, 1 person*year of data, which is literally the first sentence I wrote at the top of this thread. (But – there are a lot of architectures that work with casual datasets! Like, store everything in a text file. Or fork bitcoin and run a full node on device, for that matter. What are we trying to accomplish here?)
  - ForHackernews 2y ago
    > And now add team support, because you want to sell this to teams and make money, right? I mean, I don't, personally. I'm writing a couple small apps to scratch my own itches and I might sell them to anyone else who wants an individual copy for personal use. Remember when you could just buy a copy of a program and use it on your own computer? And it would never get updated to remove functionality or break because some servers were shut down? That's the experience I'm seeking from local-first software. I think designing for casual, personal-sized data is extremely easy if you give up the idea that every program needs to be some bloated Enterprise-Ready junkware.
    - dustingetz 2y ago
      Ok, then this constraint (casual, personal-sized data) should be the headline as the entire architecture is downstream of it
      - ForHackernews 2y ago
        Sure, if you are so committed to the quantified self that you are producing hundreds of megabytes of valuable data every day, then maybe it's impractical for you to keep it all on devices with mere terabytes of local storage and only 32 GB of RAM. You are Google's dream user. :)

ForHackernews 2y ago

Again, I guess I'm struggling to imagine what kind of database your app would need that doesn't fit in a 20 Mb sqlite file? What are all these jillions of records? You're talking about full-text indexing?

dustingetz 2y ago
PKM apps are trees of strings! It's fast until its not. Even if you can sync the global dataset to the device storage, the query engine needs data in process memory and still has to traverse it with device levels of compute not cloud compute, "instantly" i.e. without making the UI feel sluggish. If you feel otherwise, use Roam or Tana for a year, even in single-player mode. The entire category is bottlenecked on this scale problem. And now add team support, because you want to sell this to teams and make money, right? Designing for casual, personal-sized datasets is a viable architecture in very few apps. Google Maps is one shining counterpoint, because the content has a natural locality to it – you only need to sync content near where you are geographically!
- paulgb 2y ago
  I think you’re misunderstanding the overall architecture here. Instead of syncing the whole tree of strings, the way you would generally represent a PKM with Yjs is to make each logical document a Yjs document (especially given the assumption that offline periods are short.) You could still build a server-side search index over those documents, which never needs to be sent to the client.
  - dustingetz 2y ago
    this gives up relational query, i.e. you knowledge graph is no longer a graph. Notion, Roam and Tana all require relational query. What real world app category are you attempting to model that matches document structure? If the domain can be modeled as topic documents and the documents are small, like an individual google doc, sure this set of constraints may be useful. But that does not match PKM!
    - paulgb 2y ago
      It doesn’t give it up, it just moves that aspect to the server. The server can still build an index over the documents in whatever way it likes, perform expensive queries, and only send the results to the client. An example that matches that document structure is Figma; each document is individually small enough to be synced with the client, document metadata is indexed on the server, and queries over documents take place on the server.
      - dustingetz 2y ago
        This (moving relational ops to server) does not match the definition of local first provided in the article and gives up most of the value prop the article enumerates in the conclusion. I agree that Figma is a candidate to be implemented primarily as a document CRDT, same as the google doc example i already provided.
        paulgb 2y ago
        Well, the problem an index over multiple documents solves for is also not present in the application presented by the article. The plan for an individual trip (or even a lifetime of trips for most people) is not going to exceed a size that can be handled and indexed on the client.
        dustingetz 2y ago
        You're right, the approach totally works for PKM apps with less than, say, 1 person*year of data, which is literally the first sentence I wrote at the top of this thread. (But – there are a lot of architectures that work with casual datasets! Like, store everything in a text file. Or fork bitcoin and run a full node on device, for that matter. What are we trying to accomplish here?)
- ForHackernews 2y ago
  > And now add team support, because you want to sell this to teams and make money, right? I mean, I don't, personally. I'm writing a couple small apps to scratch my own itches and I might sell them to anyone else who wants an individual copy for personal use. Remember when you could just buy a copy of a program and use it on your own computer? And it would never get updated to remove functionality or break because some servers were shut down? That's the experience I'm seeking from local-first software. I think designing for casual, personal-sized data is extremely easy if you give up the idea that every program needs to be some bloated Enterprise-Ready junkware.
  - dustingetz 2y ago
    Ok, then this constraint (casual, personal-sized data) should be the headline as the entire architecture is downstream of it
    - ForHackernews 2y ago
      Sure, if you are so committed to the quantified self that you are producing hundreds of megabytes of valuable data every day, then maybe it's impractical for you to keep it all on devices with mere terabytes of local storage and only 32 GB of RAM. You are Google's dream user. :)