3 ms·
When an update happens the trick is going to be figuring out an update mechanism for the derived data without needing to run more queries. You're probably right
by josephg 10y ago
When an update happens the trick is going to be figuring out an update mechanism for the derived data without needing to run more queries. You're probably right - in the general case thats really hard to do automatically. But lots of aggregate values can be calculated based on an update without reference to the original set. `SUM()` and `MEAN()` are easy. `MAX()` and `MEDIAN()` are hard. In the case of the notification count query from blog post, you can do that efficiently with 3 map reduces:
1. Given a list of `(message, room, timestamp)` triples, generate `(room, max(timestamp))` tuples. In general max is expensive to calculate, but in this case the message collection is immutable, and the rule to update max(timestamp) when an edit happens doesn't need any extra processing based on an insert.
2. Given `(room, message_timestamp, read_timestamp)` generate `(room, message_timestamp > read_timestamp)`. Easy.
3. Then a reduce to calculate the sum.
One of the things I really like about the multi-process architecture is the diversity of solutions we should have access to here. Right now your data lives in one vendor's system. Its really hard to use a map-reduce job if your primary data store doesn't support it. And map reduces might be too complicated for simple workloads - so with small data sets you might want something simpler. With this architecture different vendors could provide different indexing systems and you could swap them out as your requirements / data size changes.
Finally as for consistency, I've got a bunch of ideas of different ways to make it work. Its not so bad because its not a generic graph or consensus problem. Its just a DAG. You can give each primary store a unique name and version using vector clocks. (If each primary store is itself distributed thats fine - just use whatever versioning system the primary store uses as the value for that database in the vector clock.) Then when a query comes in that needs to hit both an index and a primary store, pick the most recent database version and query asking for the result set at that version. You'll need some MVCC to make it work - but thats a solvable problem. If the named version is too new for the index, the query stalls until the index sees the corresponding update from the primary store. If the version is too old (and its expired out of the MVCC working set) then we punt back to whatever is running the query and it retries. The whole system is much simpler than a distributed consensus problem because each node in the DAG can be thought of as authoritative, even if they're individually implemented as a small redundant set.