4 ms·

Web apps are so over-complicated because of piecemeal client side data stores and the need to sync with the latest fashion in api protocol. Imagine how simple

Web apps are so over-complicated because of piecemeal client side data stores and the need to sync with the latest fashion in api protocol. Imagine how simple apps would be if your data model and query interface was exactly the same on the server and client. With automatic caching and fetching logic. There is such an incredibly complex chain of code to do the simplest of things. What we lack is a good client-side relational+graph database that has an identical server-side implementation.

HatchedLake721 5y ago
Orbit.js does some of these things and coordinates client with a variety of data sources through a standard set of interfaces and using normalized data structures. I believe Orbit.js was inspired by Ember Data, as I know Dan Gebhardt is involved with Ember.js and https://jsonapi.org https://jsonapi.org
okal 5y ago
PouchDB has offered something like that for close to a decade now, if I'm not mistaken. Might be worth looking into. It is document-based, though, rather than relational+graph, though I suppose you could build graph utilities on top of it.
- hunterb123 5y ago
  RxDB does this. Recently PouchDB integration is more abstracted to swap it out but it's still the best client side database. RxDB syncs with GraphQL and Pouch/CouchDBs There's WatermelonDB which uses IndexedDB on web and SQLite on native, it's nice for syncing to custom backends. There's GUN and Orbit for distributed graph databases. Ontopic: TinyBase looks really really nice, fills the gap in-between hefty client side databases and state system solutions like Redux + Persist. I'd like to see Redux middleware integration for time travel debugging, event lots, and snapshotting if possible. The analytics and rollback APIs are a nice touch. Size is enticing.
- samwillis 5y ago
  PouchDB is incredible, so much respect for the devs that built it. The trouble is though, it is now quite an ageing codebase with relatively little maintenance. The original developers have moved on and it’s a little neglected. A few community members have picked up the mantle in the last 6 months but I would be careful picking it for something new. Last time I looked the bug tracker auto closed tickets after a month if there was no activity. That makes it look like there are few bugs being tracked. Problem is there is loads, they are just all closed.
bcherny 5y ago
The requirements on clients are fundamentally different than on the server. On clients, you want at least: 1. Batching, to avoid network waterfalls (a consequence of network use being more expensive on clients than on the server). 3. Data dependencies, so the framework knows how to stitch queries together when batching them. 3. Consistency, so all UIs update when an update comes in (a consequence of SPAs being long-lived, as opposed to the typical stateless server request). These all take some code/framework overhead. It’s valuable to pay that cost on clients, but is unnecessarily verbose for the server. People have been trying to unify local and remote function calls for years, and it’s a similar problem: the two are fundamentally different.
- wrnr 5y ago
  I is just me or does anyone else have to keep endless models and endpoints and commands and clients in sync in different languages?
  - bcherny 5y ago
    Have you tried Swagger, or another solution to codegen client endpoints/schemas from server definitions?
- vaughan 5y ago
  "code/framework overhead" is fine as long as its hidden in a library. Too many people re-implement a data layer client-side that is always a poor mapping to the server data model. My frontend should act as if its running off a local database and sync with the server. It could batch together all requests made to this database that it cannot fulfil or that are stale.
zonotope 5y ago
There's Datascript[0] and Datomic[1]. While not "identical", they are definitely complimentary. There's a (now defunct) library[2] for keeping them in sync too. 1: https://github.com/tonsky/datascript https://github.com/tonsky/datascript 2: https://www.datomic.com/ https://www.datomic.com/ 3: https://github.com/metasoarous/datsync https://github.com/metasoarous/datsync
samwillis 5y ago
SQLite on WASM[0] is absolutely what you are looking for. There is also “Absurd SQL”[1] which extends it to use indexedDB as a VFS for storage allowing proper atomic transactions and not loading the whole thing into memory. Combine it with the various JavaScript ORMs and you have a nice developer UX. I’m waiting for someone with more time than myself to build a syncing feature on top of SQLites Sessions[2] so that changes locally are synced back to the server. (Feel like I’m a cheer leader posting this comment every week) 0: https://sql.js.org/ https://sql.js.org/ 1: https://github.com/jlongster/absurd-sql https://github.com/jlongster/absurd-sql 2: https://www.sqlite.org/sessionintro.html https://www.sqlite.org/sessionintro.html
- steve_adams_86 5y ago
  Would this suffer from the overhead of using WASM, both to make the calls and to transfer the data back to the UI thread? I'm not sure but I expect it would depend on your use cases. High frequency i/o of small data might be slower compared to low frequency i/o of very large data. Mind you I'm working off of a cursory understanding of WASM performance issues from 2 years ago. Maybe this has changed a lot, or my understanding was incorrect in the first place. Do you know much about this?
- vaughan 5y ago
  Absurd is really cool. I've always thought it would be great to have an SQL db implemented in native JS though. With the database running in the same process as your app, you could store native JS Objects of your table rows in the cache, which you could bind events to. So if you modify a row in a table in the db, events could be instantly propagated to the UI. Haven't thought it through fully yet. SQL is also a bit cumbersome for things like deeply nested data, hierarchies, trees, graphs which a lot of client-side application state ends up being. Sessions looks cool!
- dotancohen 5y ago
  > (Feel like I’m a cheer leader posting this comment every week) I haven't seen it yet, so thank you! And in the name of others like me who haven't seen it yet, keep cheering please.
DrFell 5y ago
The entire point of an API is decoupling. A decoupled API makes double sense for a SPA. It's the client-server architecture on a silver platter. This idea got muddled when people started making websites as SPAs, which necessitated SSR for SEO and first-page load. Now you blurred the lines between client and server, otherwise known as isomorphism, which is a good sounding word for a bad idea.
PaulBGD_ 5y ago
Funny enough, a lot of this is describing blitz.js.

HatchedLake721 5y ago

Orbit.js does some of these things and coordinates client with a variety of data sources through a standard set of interfaces and using normalized data structures. I believe Orbit.js was inspired by Ember Data, as I know Dan Gebhardt is involved with Ember.js and https://jsonapi.org https://jsonapi.org

okal 5y ago

PouchDB has offered something like that for close to a decade now, if I'm not mistaken. Might be worth looking into. It is document-based, though, rather than relational+graph, though I suppose you could build graph utilities on top of it.

hunterb123 5y ago
RxDB does this. Recently PouchDB integration is more abstracted to swap it out but it's still the best client side database. RxDB syncs with GraphQL and Pouch/CouchDBs There's WatermelonDB which uses IndexedDB on web and SQLite on native, it's nice for syncing to custom backends. There's GUN and Orbit for distributed graph databases. Ontopic: TinyBase looks really really nice, fills the gap in-between hefty client side databases and state system solutions like Redux + Persist. I'd like to see Redux middleware integration for time travel debugging, event lots, and snapshotting if possible. The analytics and rollback APIs are a nice touch. Size is enticing.
samwillis 5y ago
PouchDB is incredible, so much respect for the devs that built it. The trouble is though, it is now quite an ageing codebase with relatively little maintenance. The original developers have moved on and it’s a little neglected. A few community members have picked up the mantle in the last 6 months but I would be careful picking it for something new. Last time I looked the bug tracker auto closed tickets after a month if there was no activity. That makes it look like there are few bugs being tracked. Problem is there is loads, they are just all closed.

bcherny 5y ago

The requirements on clients are fundamentally different than on the server. On clients, you want at least: 1. Batching, to avoid network waterfalls (a consequence of network use being more expensive on clients than on the server). 3. Data dependencies, so the framework knows how to stitch queries together when batching them. 3. Consistency, so all UIs update when an update comes in (a consequence of SPAs being long-lived, as opposed to the typical stateless server request). These all take some code/framework overhead. It’s valuable to pay that cost on clients, but is unnecessarily verbose for the server. People have been trying to unify local and remote function calls for years, and it’s a similar problem: the two are fundamentally different.

wrnr 5y ago
I is just me or does anyone else have to keep endless models and endpoints and commands and clients in sync in different languages?
- bcherny 5y ago
  Have you tried Swagger, or another solution to codegen client endpoints/schemas from server definitions?
vaughan 5y ago
"code/framework overhead" is fine as long as its hidden in a library. Too many people re-implement a data layer client-side that is always a poor mapping to the server data model. My frontend should act as if its running off a local database and sync with the server. It could batch together all requests made to this database that it cannot fulfil or that are stale.

zonotope 5y ago

There's Datascript[0] and Datomic[1]. While not "identical", they are definitely complimentary. There's a (now defunct) library[2] for keeping them in sync too. 1: https://github.com/tonsky/datascript https://github.com/tonsky/datascript 2: https://www.datomic.com/ https://www.datomic.com/ 3: https://github.com/metasoarous/datsync https://github.com/metasoarous/datsync

samwillis 5y ago

SQLite on WASM[0] is absolutely what you are looking for. There is also “Absurd SQL”[1] which extends it to use indexedDB as a VFS for storage allowing proper atomic transactions and not loading the whole thing into memory. Combine it with the various JavaScript ORMs and you have a nice developer UX. I’m waiting for someone with more time than myself to build a syncing feature on top of SQLites Sessions[2] so that changes locally are synced back to the server. (Feel like I’m a cheer leader posting this comment every week) 0: https://sql.js.org/ https://sql.js.org/ 1: https://github.com/jlongster/absurd-sql https://github.com/jlongster/absurd-sql 2: https://www.sqlite.org/sessionintro.html https://www.sqlite.org/sessionintro.html

steve_adams_86 5y ago
Would this suffer from the overhead of using WASM, both to make the calls and to transfer the data back to the UI thread? I'm not sure but I expect it would depend on your use cases. High frequency i/o of small data might be slower compared to low frequency i/o of very large data. Mind you I'm working off of a cursory understanding of WASM performance issues from 2 years ago. Maybe this has changed a lot, or my understanding was incorrect in the first place. Do you know much about this?
vaughan 5y ago
Absurd is really cool. I've always thought it would be great to have an SQL db implemented in native JS though. With the database running in the same process as your app, you could store native JS Objects of your table rows in the cache, which you could bind events to. So if you modify a row in a table in the db, events could be instantly propagated to the UI. Haven't thought it through fully yet. SQL is also a bit cumbersome for things like deeply nested data, hierarchies, trees, graphs which a lot of client-side application state ends up being. Sessions looks cool!
dotancohen 5y ago
> (Feel like I’m a cheer leader posting this comment every week) I haven't seen it yet, so thank you! And in the name of others like me who haven't seen it yet, keep cheering please.

DrFell 5y ago

The entire point of an API is decoupling. A decoupled API makes double sense for a SPA. It's the client-server architecture on a silver platter. This idea got muddled when people started making websites as SPAs, which necessitated SSR for SEO and first-page load. Now you blurred the lines between client and server, otherwise known as isomorphism, which is a good sounding word for a bad idea.

PaulBGD_ 5y ago

Funny enough, a lot of this is describing blitz.js.