7 ms·
Ask HN: How Can I Make My Front End React to Database Changes in Real-Time?
I'm exploring ways to make my frontend applications reactive to database changes without overloading the backend.
- Traditional methods like polling are resource-intensive and rerun entire queries, which isn't efficient.
- I’m familiar with some libraries and companies are doing this by building materialized views close to the DB [1][2], but I want to update the frontend, not just the backend.
- Some platforms like Supabase Realtime [3] and Firebase offer subscription models to database changes, but these solutions fall short when dealing with complex queries involving joins or group-bys.
My vision is that the modern frontend to behave like a series of materialized views that dynamically update as the underlying data changes. Current state management libraries handle state trees well but don't seamlessly integrate with relational or graph-like database structures.
The only thing I can think of is to implement it by myself, which sounds like a big PITA.
Anything goes, Brainstorm with me. Is it causing you headaches as well? Are you familiar with an efficient solution? how are you all tackling it?
[1] https://readyset.io/
[2] https://materialize.com/
[3] https://supabase.com/docs/guides/realtime/postgres-changes
[4] https://github.com/hasura/graphql-engine/blob/master/architecture/live-queries.md
- helloworld42024 2y agoYou're overcomplicating everything. Websocket endpoints. Simple.
- codingdave 2y ago...and that oversimplifies everything. Yes, websockets to the browser are likely to be a piece of the answer. But a server needs to maintain the session for the websocket and send messages. And that server needs to subscribe to data updates in the database... while also respecting auth of the current user. It is certainly a problem with an achievable solution, but it is not simple.
- bruchim 2y agoAgree.
- helloworld42024 2y agoIn his initial message, the original poster provided a link to Supabase, which integrates PostgreSQL with PostgREST out of the box. This setup automatically offers websocket server functionality: 1. The websocket server, database data subscriptions, and all session management are consolidated in this layer. 2. Updates to data are efficiently managed using a built-in PostgreSQL function. 3. User authentication is managed via React, employing either JWT or OAuth. As I mentioned earlier, it's quite simple.
- DamonHD 2y agoJust very generally, event-driven and data-flow architectures have been with us forever, and I have implemented products on them. Should not be too hard to base what you want on that sort of thing.
- bruchim 2y agojust thinking about implementing incremental materialized views and maintaining web sockets with clients makes me nervous, but maybe I'm the only one. Have you implemented such a solution e2e? from db to frontend?
- DamonHD 2y agoThe systems I was designing/building were for trading, and years ago. Some involved push live updated recomputed instrument values into live spreadsheets. And yes it was ugly! There was not a relational 'DB' in the live path.
- illuminant 2y agoWhat your trying to do is very difficult (particularly since your data views are relying upon joins and group queries.) You may need layered solutions. I've been looking at https://deepstream.io https://deepstream.io for keeping clients in sync. You will still have to watch the db, though the client and back end sync is handled!
- bruchim 2y agoVery interesting, thank you. I looks like a layer on top of your database, not a standalone baas, which is great. Have you used it in production?
- illuminant 2y agoI am evaluating it for a project. The maintainer has another great project, Golden Layout https://golden-layout.com/ https://golden-layout.com/ Too bad it still depends upon jQuery.
- rewind94 2y agoI'm interested in this problem also! I think there is a large overlap with projects that market/focus on offline-first experiences. AFAIK this problem can be solved by: 1) Considering a client-side copy of the database that gets synced with the remote DB. This is an approach [PowerSync](https://www.powersync.com/ https://www.powersync.com/) and [ElectricSql](https://electric-sql.com/ https://electric-sql.com/) and [rxdb](https://rxdb.info/ https://rxdb.info/) take! For the first two, they effectively use postgres CDC and then have a sync process that is provided for you. For rxdb I think you have to roll your own syncing but there is an interface for you to implement. One difference between ElectricSQL and PowerSync (copied this from another [HN comment](https://news.ycombinator.com/item?id=37600917 https://news.ycombinator.com/item?id=37600917)) -> `ElectricSQL uses “shapes” (currently a WIP feature) defined on the client to partition data, while PowerSync uses server-side “sync rules”. The sync rules approach allows data to be transformed and filtered using basic SQL queries, which [PowerSync] expect to scale well (benchmarks pending). PowerSync guarantees consistency over all synced data, while it appears like ElectricSQL only guarantees consistency within each shape. PowerSync also automatically adds and removes sync buckets when changes are made to either the sync rules or data, while ElectricSQL likely requires you to subscribe and remove shapes yourself.` ^ Please note that comment faced rebuttal from ElectricSQL. On the same thread (6 months ago) , someone from ElectricSQL said this `Electric is designed to support partial sync, and so you don’t have to sync your whole dataset. (Note that this is feature is under development and not yet public)` 2) Another exciting entry is [Convex](https://www.convex.dev/ https://www.convex.dev/) which supports reactive queries but it is not based on a well known existing DB like postgres, it's fully custom and new - but looks easy to get started with! They recently open-sourced their backend so you can give that a try. 3) [SurrealDB](https://surrealdb.com/features#connectivity https://surrealdb.com/features#connectivity) also supports realtime queries? but I haven't looked into this yet 4) [Triplit](https://www.triplit.dev/ https://www.triplit.dev/) Another option! Triplit is an open-source database that syncs data between server and browser in real-time but again, this not based on a well-known db like postgres, instead it uses another new solution. It is open-source, but I haven't tried it yet. Looks quick to set uo. 4) Yes, you can use supabase realtime/ postgres events to listen to changes in your db and refetch all your data as you said - not ideal but yep. Other projects/technologies to look into include Debezium for CDC and also potentially using GraphQL subscriptions to keep your frontend up to date See also: [Prisma Pulse](https://www.prisma.io/data-platform/pulse https://www.prisma.io/data-platform/pulse) which I looks something similar to Supabase Realtime. Supabase are in the midst of their announcement week. I have quietly hoping that they will launch an offline-first / realtime query feature soon. [This github issue](https://github.com/orgs/supabase/discussions/357 https://github.com/orgs/supabase/discussions/357) seems to tease it. Best of luck with this! It's a super interesting problem and I'd like to know what you end up doing! :)
- stuiterbal 2y agoI've been using Hasura.io (through nhost.io) for quite some time now and I think it would suit your needs very well. It is very good in handling complex relationships with subscriptions.
- roshanj 2y agoYou already mention Materialize -- have you looked at the SUBSCRIBE feature? https://materialize.com/docs/sql/subscribe/ https://materialize.com/docs/sql/subscribe/ It's fairly straightforward to write a websocket endpoint in whatever backend you have that issues a SUBSCRIBE query on connect and then continuously forwards the updates to the client. The complicated parts you'd need to build are on the frontend -- handling updates and merging them into whatever state you render from. Disclaimer -- I work at Materialize -- but it does seem like the perfect fit for what you need (efficiently updated real-time materialized views).
- bruchim 2y agoThanks! > It's fairly straightforward to write a websocket endpoint in whatever backend you have It indeed solves half of the problem, but managing websockets for thousands of users is not that trivial IMO
- fernandotakai 2y agoyou should try phoenix channels https://hexdocs.pm/phoenix/channels.html https://hexdocs.pm/phoenix/channels.html they scale really really easily for millions of users.
- huevosabio 2y agoUse convex.dev It's really the best developer experience.
- bruchim 2y agoIt looks very relevant to my problem! Have you used it? is it production-ready?
- brudgers 2y agoThere's no generic solution to database problems. They all depend on the specifics of the data and the specifics of access and the specifics of the use. Then you make engineering tradeoffs per the CAP theorem. Is the application launch-the-missiles critical? Is it newest-lol-cat-pictures critical? What's the network latency? What is the CDN strategy? How frequently does the data change? How much data changes per second/minute/hour/day/week/month? How much cache can you use? Etc. Good luck.
- methinkscode 2y agoeasy, add a new version on a table every time you change anything, poll to check version, update FE if version changed.
- austin-cheney 2y agoDo it the way the database wants you to do it. Don’t use HTTP. Instead use TCP sockets (WebSockets when in the browser). Then put a tigger on the tables you care about. Every time the table changes the DB sends a message to the browser in real time. Super simple.
- bruchim 2y agoIt's super simple for a simple demo app, not easy for a large scale.
- austin-cheney 2y agoIt remains just as simple. The transmission model does not change as the size of application code changes. The only thing to worry about is load balancing socket restoration when the server recovers from an outage so it’s not overwhelmed reopening all sockets simultaneously. As for the database side you should only be concerned with changes to a select few tables and not the entirety of data. That could also be centrally managed with a script.
- bionhoward 2y agoNoria sounds like what you mention with materialized views, but it’s a server side rust thing, might be pretty heavy. https://docs.rs/noria/latest/noria/ https://docs.rs/noria/latest/noria/ Consider Server Sent Events as an alternative to sockets because this lets you keep Client events restful instead of doing everything on sockets. https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events https://developer.mozilla.org/en-US/docs/Web/API/Server-sent... Might be wise to make a “sync” service? Make sure your client has a local database in indexeddb and listen for server events with data updates, apply the event to the database, and use stale-while-revalidate to double check data (perhaps debounce revalidation substantially unless you have extremely dynamic data) One idea would be to use redis queues to hold server events, let clients subscribe to the redis, don’t let them touch the DB behind the server, then it’s less of a push and more of a pull when client is ready
- gnat 2y agoWhat’s the downside with sockets that makes you want to keep events RESTful?
- ENGNR 2y agoI reckon you want some kind of publish-subscribe websocket setup, with event streams. Something like Apache Pekko, or Elixir, or maybe Redis, so the web browser subscribes to a topic they're authorised to receive updates on. And then separately, subscribe listeners on the backend to the tables, and fan out updates to the various topics. They'd get moved out to the frontend. "reactive" would be the name of the game, so event streams, managing backpressure, etc, etc. Something like the "stale while revalidate" strategy (eg as used by react-query) is wayyyy simpler however, and gets you most of the way there. You can set a timeout for when to invalidate the frontend cache to something short if you wanted (eg 3 seconds), and if the content is still being viewed by a component after that time it just fetches itself again while keeping the stale data in place. And for the few pieces that truly are realtime, like a stock ticker or something, just use a direct websocket.
- bruchim 2y agoExpiration-based cache (e.g., 3 seconds) is not acceptable in this case, and more than that will reduce the load on the server + db dramatically. Imaging subscribing to SQL query not just for a table; isn't it very complicated?
- hnthrowaway0328 2y agoIn the last stage of your data pipelines, when you are sure the data is correct and only need to be pushed to a table, maybe branch out a script that send the "hot" data to the frontend.
- sdwr 2y ago> Firebase offer subscription models to database changes, but falls short when dealing with complex queries involving joins or group-bys. Firebase is the only tool here I'm familiar with, and this makes sense to me. It's real time data "for dummies", where you don't have to worry about the internals. They can guarantee that simple gets return events quickly, but if you're asking for complex queries, you actually have to engineer the tradeoffs. Like, what happens if you are frequently writing to 2 tables, and are subscribed to a view that joins both?
- SJC_Hacker 2y agowebsockets and LISTEN/NOTIFY (if using Postgres)
- littlecranky67 2y agoAlso, consider using server-sent events over HTTP/2 (instead of websockets) if the data stream is unidirectional (server -> frontend).
- sgt 2y agoI'd use PowerSync for this: https://powersync.com/ https://powersync.com/ People are often blown away by the real time reactivity.
- Leftium 2y agoMaybe SignalDB?: https://signaldb.js.org/core-concepts/#signals-and-reactivity https://signaldb.js.org/core-concepts/#signals-and-reactivit... I swear I read that MeteorJS and/or MongoDB implemented signals for the DB layer. While searching for the source, I found SignalDB. --- edit: Found the source: https://dev.to/this-is-learning/the-evolution-of-signals-in-javascript-8ob#comment-2561f https://dev.to/this-is-learning/the-evolution-of-signals-in-... > ...Meteor Tracker is a good prior art on signals and effects history in JS (although not using the same terminology). The really magical thing about Meteor is that their "signals" and "effects" are database connected: you use variables in your frontend template, and those values can come directly from a backend DB...
- laborcontract 2y agoSWR
- motleydev 2y agoYou listed Hasura as a footnote but not in your message. Did you try it out? I wrote these guides a while back: https://hasura.io/blog/getting-started-with-react-query-and-graphql https://hasura.io/blog/getting-started-with-react-query-and-... https://hasura.io/blog/a-simple-real-time-event-driven-architecture-with-qr-codes https://hasura.io/blog/a-simple-real-time-event-driven-archi... And did this YouTube series about building realtime apps with Hasura and React: https://www.youtube.com/playlist?list=PLTRTpHrUcSB8elpwJKDIQNfNlqpPTNHGG https://www.youtube.com/playlist?list=PLTRTpHrUcSB8elpwJKDIQ...
- Leftium 2y agoI edited my first comment, but perhaps this deserves its own comment: https://docs.meteor.com/api/tracker.html https://docs.meteor.com/api/tracker.html > Meteor has a simple dependency tracking system which allows it to automatically rerun templates and other computations whenever Session variables, database queries, and other data sources change. > Unlike most other systems, you don’t have to manually declare these dependencies — it “just works”. The mechanism is simple and efficient. When you call a function that supports reactive updates (such as a database query), it automatically saves the current Computation object, if any (representing, for example, the current template being rendered). Later, when the data changes, the function can “invalidate” the Computation, causing it to rerun (rerendering the template).
- lolc 2y agoI enjoyed using Meteor. It is magic at first, and still rather elegant when you get into advanced stuff. The main downside for me was that it uses Mongodb. We had to denormalize heavily to get it working acceptably fast. There was talk about Postgres support, so maybe that is solved.
- wtf_is_up 2y agoThrowing RethinkDB in the mix as well. https://rethinkdb.com/ https://rethinkdb.com/
- mhuffman 2y agoThis is the answer for what OP wants. It literally is built to do what he wants and shows a demo doing it on the front page of the site. Too bad they couldn't make it into a business, but the project seems to be alive and well.
- oliwary 2y agoPocketbase [0] is a possibility. It offers a way to subscribe to collections, meaning the client will be notified if any of the records in that collection change. [1] Should be quite efficient too, the FAQ claims that 10k realtime connections on a small hetzner VPS is no problem [2] [0] https://pocketbase.io/ https://pocketbase.io/ [1] https://pocketbase.io/docs/api-realtime/ https://pocketbase.io/docs/api-realtime/ [2] https://pocketbase.io/faq https://pocketbase.io/faq
- Gys 2y agoCurrently my preference as well. I like the auth part and admin UI. Combined with htmx it makes building reactive apps (without React) relatively easy.
- louwhopley 2y agoI would suggest checking out PowerSync (https://www.powersync.com/ https://www.powersync.com/) for this. I've built various React apps (and iOS apps before that) in the past on a bunch of stacks. My experience with the following tools were as follows: - Pusher: love the tool, but required lots of layers on top of it to be used for keeping DB's in sync. It does make sockets trivially easy to get going and use. - MeteorJS: lots of cool "realtime magic", but hard to break out (it's too opinionated and constrained). We ended up breaking out and running ExpressJS + React + Pusher and ditching MeteorJS. - Firebase: used it for two projects. Platform lock-in, like weird Firebase-specific nuances to workaround and solve for, felt like quite a lot. You basically have to park your experience with other stacks and databases and become an expert in Firebase. A big one here was around access policies for the data in the database → limiting what data a user can access. My current stack is now mostly Cloudflare and keeping real-time requirements in the product design to a minimum, to reduce the overhead.
- bruchim 2y agoThanks, super helpful. re Cloudflare, are you using them to reduce latency? what about a centralized database?
- BenoitP 2y agoI did something akin to that with the following monstrosity of DevOps: - set up Kafka - set up Flink - CDC on your DB, into Kafka - have your server maintain a websocket to clients, read Kafka from it, and dispatch messages according to a field from Kafka - put your reaction logic into Flink, insert into the Kafka topic read by the server I even went to opening JIRA tickets in Apache Flink, so that you may choose the Kafa topic to insert to dynamically. In case you have multiple servers to dispatch messages to. It was very clunky, and a pain to operate. However it was short-lived and is no longer a concern of mine. What I got away from that: adding a time to data serving is just multiplying the complexity of data bugs you can have.
- mysterydip 2y agoMaybe I'm misunderstanding, but you said anything goes, so here goes: If rerunning entire queries is the problem, can you implement a changes table (could potentially double as an audit table if desired) which on a basic level just has time and rowid. Then you poll that table for anything newer than your current timestamp. Say for a report with 1000 rows, only 2 have changed in the last second. Then you can just do a query of the big table on those exact rowids. The benefit over having timestamp in the big table is the changes table can be kept to a minimal size. You don't need to store any more than, say, a minute's worth of changes. Anyone visiting the page for the first time or that far out of sync will poll the whole report.
- IgorPartola 2y agoI don’t think I understand what you are after. Do you want a frontend framework that takes care of table and row based state management? A notification system that pushes changes made to the backend database to the browsers? A database that can push out results of complex queries to browsers using some sort of magical scalability? Your requirements aren’t really clear, but I suspect you are asking the wrong questions.
- yes_but_no 2y agoYou might be interested in checking out perspective[0] they server side `tables` which propagate changes to frontend charts by sending row diffs. [0] https://perspective.finos.org/ https://perspective.finos.org/
- matt_oriordan 2y agoAbly’s LiveSync (https://ably.com/livesync https://ably.com/livesync) is conceptually designed to solve exactly this problem by providing a database connector that subscribes to changes committed using the outbox pattern, and a models SDK to subscribe to those changes in your app. It’s early days as it’s just been released, but at the core the team’s goal in designing that was to solve this exact problem. Interested if this is meets your needs? Disclaimer. I am the co-founder of Ably.