4 ms·
I'm not sure I would call this relatively easy even without joins. Without joins Google Cloud Firestore does exactly what you're describing. The initial query r
by rockwotj 6y ago
I'm not sure I would call this relatively easy even without joins. Without joins Google Cloud Firestore does exactly what you're describing. The initial query runs against the DB then the client gets a filtered stream of updates to only that query. Its distributed and scales logarithmically with the number of queries, as it doesn't need to keep the query results in memory/materialized.
The fun part of this problem is that it's really inverted from most traditional database literature out there. Mostly the problem is you have a query and need to find the matching rows. To make updates efficient you need to do this the other way around - if there is a row (that has changed) find all the matching queries.
With joins (or any data dependant query where you can't tell a row is in the result set without looking at other data) you need to keep the query results materialized otherwise you can't have enough information without going back to disk or keeping everything in memory, which isn't really feasible in most cases.
Source: I worked on Google Cloud Firestore from it's launch until 2020 and was the on responsible for the current implementation and data structures of how changes get broadcasted to subscribed queries.
- davty 6y agoThis sounds fascinating. Do you have any recommended source to read more about this? Like a book or a paper?
- rockwotj 6y agoSadly I never thought to trying to publish a white paper with my co workers while I was working on this. I'd love the chance to do so. We did do many internal company presentations on the matter however. That being said as far as I know there isn't any published works on a "Reverse Query Engine" or RQM for short which is the name we settled on internally for this subsystem within Firestore.
- wikibob 6y agoI would love to read this paper when you write it.
- toolslive 6y agoMind: - the query language is quite limited (fe, searching on the absence of an attribute is not possible) - shame you can't do projections on the changes server side. you get the whole thing and not just the fields you're interested in
- rockwotj 6y agoThe query language is limited, but a lot of the limitations are intentional. Here is a post from a while back where I explained some of the reason for some of these limitations; https://groups.google.com/g/google-cloud-firestore-discuss/c/9DT5ZX9EmmM/m/wNyFP259DAAJ https://groups.google.com/g/google-cloud-firestore-discuss/c... Granted not all of them are due to this philosophy and some are just hard problems due to other constraints like the security rules verification system. RE projections: this is actually due to some limitations in processing security rules and not a limitation in the query matching. Edit: RE matching on missing attributes. This is due to Firestore having sparse indexes and being schemaless, if you write explicit null values you can query for those fields being null.
- toolslive 6y agoIf your building a reactive website then firestore is actually great. Edit: but yes IMNSHO, there are also some design mistakes (schemaless, silly limitations on keys, ...) both in the service and the client libs. I only used the python and js clients though. I might be biased
- rockwotj 6y agoYupp, certainly don't disagree. It's not a perfect system - to comment on those two issues: there were some pipe dreams when I was there to support schemas, including partial schemas, but we needed a better migration story (and someone to convince the powers that be it was the right thing to work on). The limitations on keys are mostly due to the system being compatible with Cloud Datastore.