7 ms·
I find that moving the full query system into the front end is where most front end devs really want to be. They want a full power query system for the data in
by BackBlast 3y ago
I find that moving the full query system into the front end is where most front end devs really want to be. They want a full power query system for the data instead of continuous rounds of re-inventing the transport layer, REST, GraphQL, *RPC, etc.
It's hard to adopt such a system in most traditional web shops with their specialized backend and frontend teams. You're pulling out the database, backend, transport, and auth layers and replacing them with this single block system. Most system architects grew up in the backend so they are generally pretty ignorant of this issue. As it touches both sides extensively you're probably not fitting this into an existing system, which leaves only green field new development. Finally your backend is not an AWS or Asure service, neither is it lambda friendly. All of this means that most architect types I talk to will never touch it.
This style of system mostly already exists with old tech, CouchDB+PouchDB. Which works pretty well for some things. The downsides are that the query system isn't really ideal and the auth and data scoping system is pretty foreign to most people. The easiest model to work with is when the data is totally owned by a single user, and then you use the out-of-the-box database-per-user model. High data segmentation with CRDTs removes a lot of conflict issues.
It has scaling issues though, CouchDB has really high CPU requirements when you're connecting 10k to 100k users. The tech is long in the tooth though it is maintained. On the system design side it gets really complicated when you start sharing data between users, which makes it rather unsuitable as you're just moving the complexity rather than solving it.
This approach seems to hit the same target though will likely have similar scaling issues.
Look forward to see the evolution of the system. Looks like a first step into the world.
- cratermoon 3y agoYay, we're moving back to fat clients! What has been is what will be, and what was done is what will be done, there is nothing new under the sun.
- BackBlast 3y agoI'm on the fat client train with my company and I nudge my clients that way if they're open. It's just a great way to build a system.
- cratermoon 3y agoGreat until you have to support n versions on m platforms and half your customers are enterprisey and stay on a 6-year-old version from before your last LTS version on a now-unsupported platform because they built a core part of their business processes on a a misfeature.
- sp332 3y agoYes but targeting WASM and SQLite minimizes that pain quite a bit.
- cratermoon 3y agoRemember when targeting Macromedia Flash was going to solve the web compatibility and interactivity conundrum?
- sp332 3y agoYeah? It was targeted for destruction by Apple because it was buggy and insecure, not because it wasn't delivering.
- password4321 3y agoDon't forget horrendous mobile performance (battery drain)!
- MaxBarraclough 3y agoDon't forget proprietary.
- sp332 3y agoAnd the lack of accessibility. You generally couldn't even copy and paste text out of it.
- whoisthemachine 3y agoThose pesky backends are so annoying, so why don't we just put a backend on every client?
- throwaway290 3y agoSchema and data migrations are too tricky, so why not have every client do it.
- srcreigh 3y agoconnecting to remote DB and impersonating a user account for bug hunting is too easy right now, let’s create the need for a way to proxy to local client computers with greater access to their private information.
- tdeck 3y agoHow is this approach meant to handle data visibility and access control? Often a large part of a backend is materializing raw data into a form that the active user is allowed to view.
- BackBlast 3y agoSo if the user owns all their own data, their "data view" is their data set. A To-Do system, a personal finance app, any kind of note-taking or personal record keeping fits this model. You create a database per user and the auth and sync are all self contained within that database. This system is multi-master, which means that any change on a client or on the server will be replicated to every other. There is no "authority" which trumps the others. The server is simply a central hub that requires the right authentication to allow the sync process to happen. When you want to create a set of data that crosses user boundaries, it gets complicated. It's possible to do, but you're not on the easy train anymore. Creating a system that's both easy to use, and scopes the right data view out of the system wide tables and rows we usually think of databases, is not the CouchDB nor SQLSync model.
- presidentender 3y agoCorrect me if I'm wrong: we can avoid the idea of a master for this use case because we suppose that only a single client (also server, I guess) will write at a time?
- danielheath 3y agoYou’re wrong if clients can be used offline and sync when they come back online.
- BackBlast 3y agoOne user can have multiple clients. This is frequently the case, many to most users have both a PC and a phone. Also when one allows reasonable sharing of the account with family, 5+ connected clients is common.
- 3y ago
- asaddhamani 3y agoReminds me of Meteorjs. It would let you sync a subset of your data to the client and then the client could query it any which way it wanted. They called this “Minimongo”.
- BackBlast 3y agoI've used Meteor. I thought it was a good system. It didn't have offline capability, at least not back when I used it. It really needed to be connected to work. But conceptually, yes, it had a very similar system.
- egeozcan 3y agoIf Meteor could scale, we'd probably hear about it way more these days. I remember having problems with 200 users serving from my above average dev pc for testing internal tools. It's a DX dream though.
- asaddhamani 3y agoYeah, faced the scaling challenges. What was recommended was using the "methods" which was basically RPC to fetch the data, rather than using the PubSub data syncing. Meteor maintained a full copy of what data each client had in memory, so this made it use a lot of RAM, and it did some kind of diffing for these on each update to only send the required updates over the wire, so this made it have high CPU utilisation. I had to move the chat portion of my app over to SocketIO which scaled wonderfully, but the DX was not quite as nice. Rails and Phoenix are doing something similar (kind of) these days, and those seem to scale better, you just move all the data heavy lifting to the server and do almost nothing on the client.
- miunau 3y agoMeteor works fine. We've served tens of thousands of concurrent connections on a low grade EC2 instance with some apps.
- egeozcan 3y ago
- crooked-v 3y agoIt's not quite shipping the DB to the client, but I like the Supabase/PostgREST approach for replacing everything between the client and SQL server with a single very thin proxy that maps tables and functions to REST calls. Even the auth mechanism is fundamentally just Postgres RLS with signed tokens that you can query against in your RLS policies.