6 ms·

A Graph-Based Firebase

177 points by stopachka 4y ago

revskill 4y ago
I use Hasura for this purpose. With some hooks, you can achive offline mode, too. You need a bit tricky hacks to use Hasura permission system the way you want, though.
- stopachka 4y ago
  Thanks for taking a look at the essay. As you mentioned, indeed it's possible if you store all commands, and sync with IndexedDB. The edge cases get quite complicated though. We're optimistic that a local layer that handles transactions and sync could makes things a lot easier.
- kennymeyers 4y ago
  Tell me more about this!
- alexashka 4y ago
  Hasura does not offer syncing. Syncing is hard :)
  - revskill 4y ago
    Aha, yup. My way of offline, is just store all commands offline and sync the command only. Syncing the view is hard.
    - lf-non 4y ago
      We used to do this for an app, but moved away from that eventually, because: 1. You need to support old command (and payload schema) indefinitely. Or need migrations for those instead. 2. In highly interactive apps, this needs a lot of chatter. A peer that is offline for a few weeks may need to sync many many commands before they get to the latest version, and people generally don't like waiting on sync to complete. And syncing while users are interacting with the app gets complex pretty fast. We tried to alleviate 2 using things like compaction and merging of commands, but eventually it ended up being much more complex than syncing the latest ui state which was a much smaller payload for our case. It also eliminated many of corner case bugs in our command history compaction logic where some clients could end up in an invalid or unexpected states in very specific scenarios which were very hard to reproduce. Doing aggressive compaction while retaining effective order can get tricky if the object model is complex and deeply interlinked. Not saying it is not a valid approach, but Ymmv.
kevmo314 4y ago
> These triples say that the Layer with id 1 has a fontSize 20 and backgroundColor blue. Since they are different rows, there’s no conflict. This sounds a lot like Bigtable (https://cloud.google.com/bigtable https://cloud.google.com/bigtable), which also does last-write-wins conflict resolution layer. So this is adding a GraphQL + frontend layer to it?
- stopachka 4y ago
  I'm not as familiar with Bigtable's data model. Afaik they don't use a triple-store like system, but their data model does look interesting. [^1] The concept you have is on point though. You can think that we've moved a graph-database over to the frontend, and introduces a GraphQL-like language for it. [^1]: https://static.googleusercontent.com/media/research.google.com/en//archive/bigtable-osdi06.pdf https://static.googleusercontent.com/media/research.google.c...
  - kevmo314 4y ago
    Very cool, super exciting to see a clean frontend to it. As a big Firebase fan, I'm looking forward to what you build :)
fhur 4y ago
Very interesting. I came to many similar conclusions completely independently, even attempted to build a typesafe in memory datalog in typescript. I also came to the conclusion that just exposing datalog triples as a query language would never feel right and tried to expose a graphql like language that generated the datalog triples. IMO react relay offers a great similar offering with their normalized cache. Relay has great DX too and can be totally type safe. To my knowledge datalog is way too dynamic for static analysis. That being said, I would love to try Instant out. I'm really happy to see innovation in this area.
- stopachka 4y ago
  Thank you for the kind words! Both Joe and I are around if you any feedback. In terms of type-safety, we're thinking of it as an added layer. I'm optimistic, that since InstaQL provides objects as the interface, we could have more idiomatic types down the road.
  - eurasiantiger 4y ago
    This comment thread looks like it could benefit from Dgraph, which is an open source GraphQL-layer database built on top of a RDF n-quad store (badger).
fernandohur 4y ago
Another project with similar goals in mind: https://paulbutler.org/2020/the-webassembly-app-gap/ https://paulbutler.org/2020/the-webassembly-app-gap/ Also backed by YC.
- jitl 4y ago
  Is this blog post a company? What company?
  - dang 4y ago
    https://news.ycombinator.com/item?id=30502978 https://news.ycombinator.com/item?id=30502978
- paulgb 4y ago
  Thanks for the mention! For some context, that blog post discusses what I saw as missing pieces of the stack circa 2020. The startup I’m building (https://driftingin.space https://driftingin.space) is roughly the “lambda for websockets” part, whereas Instant is closer to the “generalized CRDT data layer part”.
pezo1919 4y ago
Are transactions supported in that solution or in the mentioned Datalog related ecosystem? I have built my own similar reactive inmemory triple store with typescripts type safety, but then I realized I still need transactions which are a bit of a pain, because transactions are bundling the otherwise separete triplets, so the atomic, independent logic of triplets and related effects breaks a bit. (I am sure it's solvable.)
- jitl 4y ago
  The example code uses a `transact` function. But it really depends on what you mean by “transaction”. We don’t use a triple store at Notion, but we do use an abstraction that ensures a collection of operations either all succeed or all fail. We don’t support “interactive” transaction, where you can read, modify, write as an atomic group. This just isn’t desirable in a multiplayer or offline system - in cases we need that kind of consistency we use a normal HTTP API which is online-only.
  - pezo1919 4y ago
    Cool, thanks! Now a question: how a dev picks up related knowledge? I have a BSc and extra 7 years in development, but this area was totally gray to me. My motivation was only to come up a scalable solution for offline first apps with some kind of automatic persistence support, so at the end of the day my design goals were quite similar. How do you come up with stuff at notion? Are there must have books or just going with gut, experience and existing solutions you are aware about?
- stopachka 4y ago
  As jitl points out, in multiplayer settings, what you need is some way to commit a series of transactions all-together or not at all. We support this. In the case where you want to read the database inside your transaction, we take inspiration from Datomic. Datomic runs all mutations in one high-memory box. You can provide functions that run in that box. This way, you can guarantee that the reads inside your transaction have the latest value. There's a lot of UX to figure out there, and this would be something to try to avoid in an offline-available setting.
mizzao 4y ago
This looks like an evolution of Meteor, doesn't it?
- stopachka 4y ago
  Meteor certainly inspires us!
  - eyeswideopen 4y ago
    What would you say are your biggest differences?
    - stopachka 4y ago
      Relations, and less vertical integration
antidnan 4y ago
Very cool! What're the differences between this and something like Replicache?
- stopachka 4y ago
  I haven't looked too deeply at Replicache, but the two differences as I understand it: 1. Replicache is a layer you add over an existing backend. Instant handles the backend. 2. The expose a key-value store. We expose a graph store. Both 1&2 have pros and cons each way. I think we're inspired by the same problems, and their docs are really well done.
joe_fishfish 4y ago
Does Instant have any plans for iOS and Android SDKs? One of the biggest wins with Firebase is that it's really easy to integrate across platforms.
- stopachka 4y ago
  Yes! Right now one of our users implemented an RN integration. We'd love to support iOS and Android soon.
waveywaves 4y ago
Are there any plans to open source instant in the future ?
- stopachka 4y ago
  We like the open-core mode. We have definite plans to open-source the client. Still think the best way to do this for the server.
habosa 4y ago
I used to work on Firebase. I also use Firebase heavily in my work and personal projects. I am normally quite skeptical of “Firebase but X” projects because they often seem to be running head first into the issues that the Firebase team so skillfully avoided. This isn’t one of those projects. The author of this essay clearly gets what you need to actually build a backend for a modern app that’s also simple enough to get started with. I’m very excited about this project.
- stopachka 4y ago
  Thank you for the kind words.
jinjin2 4y ago
Another product that does all this (relations, reactivity, offline, permissions, sync, etc…) really well, is Realm. It is beyond awesome for building mobile apps, but mobile only. Annoyingly they have never released a web version. No idea why. I would still look into it for ideas. It is super powerful and a joy to work with. I would love to see something like that working in the browser.
ar7hur 4y ago
I would really use this for many of my projects! For project A, I used Firebase which was great initially but eventually ran into the limitations well described in Stopa's essay. For project B, I decided to write everything from scratch but the amount of work it required is crazy, and of course the result in quite brittle.
- stopachka 4y ago
  Thrilled to hear this, thank you :).
tlarkworthy 4y ago
At Firebase we sometimes pondered the feasibility of a SQL version, but the semantics of SQL seem littered with semantic footguns that don't lend themselves to offline, secure, scalable and event driven distributed applications. We know everybody wanted better query expressivity but delivering that in a mobile friendly, clienside, secure package was very difficult to see the path to. tripletstore/datalog actually seems like a decent compromise between SQL and no-SQL that could actually work out! Awesome idea!
- stopachka 4y ago
  I'm encouraged reading this. Thank you.
j-pb 4y ago
The basic intuitions they have are correct, but if feels like there is a serious lack of or disregard for the theoretical background needed to talk about this stuff properly. > (pull db '[* {:team/task [* {:task/owner [*]}]}] team-id) That's not Datalog. It's kind of a mix between a conjunctive query and a regular path query. Datalog isn't even really a query language. It's a class of query languages with a specific expressive power. Whereas SQL is essentially conjunctive queries with non-stratified Negation, Datalog is recursive conjunctive queries with no or only stratified negation. Datalog also has nothing to do with triples, they are two completely orthogonal concepts. I've been slaving away in this very space for years and this post heavily reminds my of my initial hubris. Building something truly foundational that can be universally implemented and perform in all potential target languages from Javascript, to WASM via e.g. Zig and Rust, while at the same time being both conceptually simple, and easy to implement, is really really really difficult with a lot of pain lurking in the details.
- blain_the_train 4y ago
  do you write about your findings and ideas? i would be interested to read more.
  - j-pb 4y ago
    I regularly regret not writing a blog... But if you want to talk about some ideas feel free to drop by at discord.gg/tribles
- deleted 4y ago
  [deleted]
dustingetz 4y ago
One possible solution to the "sync" layer (how do you sync the local db shard with the global consensus) looks like this: https://www.researchgate.net/publication/359578461_Continuous_Integration_of_Data_Histories_into_Consistent_Namespaces https://www.researchgate.net/publication/359578461_Continuou... (2022) also need to fit whole view updates into 16ms frame budget (not just one query but every query on the page that is impacted by a change as well as downstream reactive views). At some tipping point it can be faster to move relational queries to the cloud (sacrificing local first), and treat local first as an edge case (not all page components need to be live in offline mode - depending on the app, you may just need document edits not relations)
- stopachka 4y ago
  The research paper looks intriguing. I'll look into it. I am not as convinced about moving queries to the cloud, when it comes to the north stars of Figma / Linear / Notion. It would be hard to get the same kind of UX.

177 points by stopachka 4y ago

I use Hasura for this purpose. With some hooks, you can achive offline mode, too. You need a bit tricky hacks to use Hasura permission system the way you want, though.

stopachka 4y ago
Thanks for taking a look at the essay. As you mentioned, indeed it's possible if you store all commands, and sync with IndexedDB. The edge cases get quite complicated though. We're optimistic that a local layer that handles transactions and sync could makes things a lot easier.
kennymeyers 4y ago
Tell me more about this!
alexashka 4y ago
Hasura does not offer syncing. Syncing is hard :)
- revskill 4y ago
  Aha, yup. My way of offline, is just store all commands offline and sync the command only. Syncing the view is hard.
  - lf-non 4y ago
    We used to do this for an app, but moved away from that eventually, because: 1. You need to support old command (and payload schema) indefinitely. Or need migrations for those instead. 2. In highly interactive apps, this needs a lot of chatter. A peer that is offline for a few weeks may need to sync many many commands before they get to the latest version, and people generally don't like waiting on sync to complete. And syncing while users are interacting with the app gets complex pretty fast. We tried to alleviate 2 using things like compaction and merging of commands, but eventually it ended up being much more complex than syncing the latest ui state which was a much smaller payload for our case. It also eliminated many of corner case bugs in our command history compaction logic where some clients could end up in an invalid or unexpected states in very specific scenarios which were very hard to reproduce. Doing aggressive compaction while retaining effective order can get tricky if the object model is complex and deeply interlinked. Not saying it is not a valid approach, but Ymmv.

kevmo314 4y ago

> These triples say that the Layer with id 1 has a fontSize 20 and backgroundColor blue. Since they are different rows, there’s no conflict. This sounds a lot like Bigtable (https://cloud.google.com/bigtable https://cloud.google.com/bigtable), which also does last-write-wins conflict resolution layer. So this is adding a GraphQL + frontend layer to it?

stopachka 4y ago
I'm not as familiar with Bigtable's data model. Afaik they don't use a triple-store like system, but their data model does look interesting. [^1] The concept you have is on point though. You can think that we've moved a graph-database over to the frontend, and introduces a GraphQL-like language for it. [^1]: https://static.googleusercontent.com/media/research.google.com/en//archive/bigtable-osdi06.pdf https://static.googleusercontent.com/media/research.google.c...
- kevmo314 4y ago
  Very cool, super exciting to see a clean frontend to it. As a big Firebase fan, I'm looking forward to what you build :)

fhur 4y ago

Very interesting. I came to many similar conclusions completely independently, even attempted to build a typesafe in memory datalog in typescript. I also came to the conclusion that just exposing datalog triples as a query language would never feel right and tried to expose a graphql like language that generated the datalog triples. IMO react relay offers a great similar offering with their normalized cache. Relay has great DX too and can be totally type safe. To my knowledge datalog is way too dynamic for static analysis. That being said, I would love to try Instant out. I'm really happy to see innovation in this area.

stopachka 4y ago
Thank you for the kind words! Both Joe and I are around if you any feedback. In terms of type-safety, we're thinking of it as an added layer. I'm optimistic, that since InstaQL provides objects as the interface, we could have more idiomatic types down the road.
- eurasiantiger 4y ago
  This comment thread looks like it could benefit from Dgraph, which is an open source GraphQL-layer database built on top of a RDF n-quad store (badger).

fernandohur 4y ago

Another project with similar goals in mind: https://paulbutler.org/2020/the-webassembly-app-gap/ https://paulbutler.org/2020/the-webassembly-app-gap/ Also backed by YC.

jitl 4y ago
Is this blog post a company? What company?
- dang 4y ago
  https://news.ycombinator.com/item?id=30502978 https://news.ycombinator.com/item?id=30502978
paulgb 4y ago
Thanks for the mention! For some context, that blog post discusses what I saw as missing pieces of the stack circa 2020. The startup I’m building (https://driftingin.space https://driftingin.space) is roughly the “lambda for websockets” part, whereas Instant is closer to the “generalized CRDT data layer part”.

pezo1919 4y ago

Are transactions supported in that solution or in the mentioned Datalog related ecosystem? I have built my own similar reactive inmemory triple store with typescripts type safety, but then I realized I still need transactions which are a bit of a pain, because transactions are bundling the otherwise separete triplets, so the atomic, independent logic of triplets and related effects breaks a bit. (I am sure it's solvable.)

jitl 4y ago
The example code uses a `transact` function. But it really depends on what you mean by “transaction”. We don’t use a triple store at Notion, but we do use an abstraction that ensures a collection of operations either all succeed or all fail. We don’t support “interactive” transaction, where you can read, modify, write as an atomic group. This just isn’t desirable in a multiplayer or offline system - in cases we need that kind of consistency we use a normal HTTP API which is online-only.
- pezo1919 4y ago
  Cool, thanks! Now a question: how a dev picks up related knowledge? I have a BSc and extra 7 years in development, but this area was totally gray to me. My motivation was only to come up a scalable solution for offline first apps with some kind of automatic persistence support, so at the end of the day my design goals were quite similar. How do you come up with stuff at notion? Are there must have books or just going with gut, experience and existing solutions you are aware about?
stopachka 4y ago
As jitl points out, in multiplayer settings, what you need is some way to commit a series of transactions all-together or not at all. We support this. In the case where you want to read the database inside your transaction, we take inspiration from Datomic. Datomic runs all mutations in one high-memory box. You can provide functions that run in that box. This way, you can guarantee that the reads inside your transaction have the latest value. There's a lot of UX to figure out there, and this would be something to try to avoid in an offline-available setting.

mizzao 4y ago

This looks like an evolution of Meteor, doesn't it?

stopachka 4y ago
Meteor certainly inspires us!
- eyeswideopen 4y ago
  What would you say are your biggest differences?
  - stopachka 4y ago
    Relations, and less vertical integration

antidnan 4y ago

Very cool! What're the differences between this and something like Replicache?

stopachka 4y ago
I haven't looked too deeply at Replicache, but the two differences as I understand it: 1. Replicache is a layer you add over an existing backend. Instant handles the backend. 2. The expose a key-value store. We expose a graph store. Both 1&2 have pros and cons each way. I think we're inspired by the same problems, and their docs are really well done.

joe_fishfish 4y ago

Does Instant have any plans for iOS and Android SDKs? One of the biggest wins with Firebase is that it's really easy to integrate across platforms.

stopachka 4y ago
Yes! Right now one of our users implemented an RN integration. We'd love to support iOS and Android soon.

waveywaves 4y ago

Are there any plans to open source instant in the future ?

stopachka 4y ago
We like the open-core mode. We have definite plans to open-source the client. Still think the best way to do this for the server.

habosa 4y ago

I used to work on Firebase. I also use Firebase heavily in my work and personal projects. I am normally quite skeptical of “Firebase but X” projects because they often seem to be running head first into the issues that the Firebase team so skillfully avoided. This isn’t one of those projects. The author of this essay clearly gets what you need to actually build a backend for a modern app that’s also simple enough to get started with. I’m very excited about this project.

stopachka 4y ago
Thank you for the kind words.

jinjin2 4y ago

Another product that does all this (relations, reactivity, offline, permissions, sync, etc…) really well, is Realm. It is beyond awesome for building mobile apps, but mobile only. Annoyingly they have never released a web version. No idea why. I would still look into it for ideas. It is super powerful and a joy to work with. I would love to see something like that working in the browser.

ar7hur 4y ago

I would really use this for many of my projects! For project A, I used Firebase which was great initially but eventually ran into the limitations well described in Stopa's essay. For project B, I decided to write everything from scratch but the amount of work it required is crazy, and of course the result in quite brittle.

stopachka 4y ago
Thrilled to hear this, thank you :).

tlarkworthy 4y ago

At Firebase we sometimes pondered the feasibility of a SQL version, but the semantics of SQL seem littered with semantic footguns that don't lend themselves to offline, secure, scalable and event driven distributed applications. We know everybody wanted better query expressivity but delivering that in a mobile friendly, clienside, secure package was very difficult to see the path to. tripletstore/datalog actually seems like a decent compromise between SQL and no-SQL that could actually work out! Awesome idea!

stopachka 4y ago
I'm encouraged reading this. Thank you.

j-pb 4y ago

The basic intuitions they have are correct, but if feels like there is a serious lack of or disregard for the theoretical background needed to talk about this stuff properly. > (pull db '[* {:team/task [* {:task/owner [*]}]}] team-id) That's not Datalog. It's kind of a mix between a conjunctive query and a regular path query. Datalog isn't even really a query language. It's a class of query languages with a specific expressive power. Whereas SQL is essentially conjunctive queries with non-stratified Negation, Datalog is recursive conjunctive queries with no or only stratified negation. Datalog also has nothing to do with triples, they are two completely orthogonal concepts. I've been slaving away in this very space for years and this post heavily reminds my of my initial hubris. Building something truly foundational that can be universally implemented and perform in all potential target languages from Javascript, to WASM via e.g. Zig and Rust, while at the same time being both conceptually simple, and easy to implement, is really really really difficult with a lot of pain lurking in the details.

blain_the_train 4y ago
do you write about your findings and ideas? i would be interested to read more.
- j-pb 4y ago
  I regularly regret not writing a blog... But if you want to talk about some ideas feel free to drop by at discord.gg/tribles
deleted 4y ago
[deleted]

dustingetz 4y ago

One possible solution to the "sync" layer (how do you sync the local db shard with the global consensus) looks like this: https://www.researchgate.net/publication/359578461_Continuous_Integration_of_Data_Histories_into_Consistent_Namespaces https://www.researchgate.net/publication/359578461_Continuou... (2022) also need to fit whole view updates into 16ms frame budget (not just one query but every query on the page that is impacted by a change as well as downstream reactive views). At some tipping point it can be faster to move relational queries to the cloud (sacrificing local first), and treat local first as an edge case (not all page components need to be live in offline mode - depending on the app, you may just need document edits not relations)

stopachka 4y ago
The research paper looks intriguing. I'll look into it. I am not as convinced about moving queries to the cloud, when it comes to the north stars of Figma / Linear / Notion. It would be hard to get the same kind of UX.