2 ms·
I've been thinking about this a lot recently, and I'm convinced we need a Storm[1]-like architecture for databases. Something which would allow views / denormal
by josephg 10y ago
I've been thinking about this a lot recently, and I'm convinced we need a Storm[1]-like architecture for databases. Something which would allow views / denormalization / indexing to run in separate processes. Preferably in the same language the app developers speak. And at the same time be part of the database transaction logic - triggered idempotently by the database so you always see a consistent view of the data no matter how you update & query it. (You can do this efficiently with MVCC if you proxy queries through the indexer / denormalizer / view logic / whatever you want to call it).
I've scattered some more thoughts about this here, but its a big topic: https://josephg.com/blog/composing-databases/ https://josephg.com/blog/composing-databases/
[1] http://storm.apache.org/ http://storm.apache.org/
- jerf 10y ago"triggered idempotently by the database so you always see a consistent view of the data no matter how you update & query it" As I sit here trying to build the thing in my head that would do that, I see some major performance issues arising. Specifying how you want the data denormalized isn't that big a deal. Taking a frozen normalized database and denormalizing it according to that spec is a somewhat bigger deal; the performance implications are non-obvious because the denormalization can invoke arbitrary amounts of data in arbitrary combinations in arbitrary orders, even before we let the denormalization specification be Turing complete. Mathematically it all works but if in practice my denormalization pass is randomly accessing huge swathes of the database every time I insert a row that's not going to work very well. If we're trying to avoid slow queries when the user asks for data, and we're trying to denormalize in advance, we're basically trying to have our already-presumably-pretty-busy database pre-emptively guess all possible slow queries it might run in the future, and run them now. And then run them again when a small change comes in, having possibly never used the first set of denormalized, expensive queries I just ran. That seems... likely to generate performance problems, to understate my case by a lot. Even accounting for possible efficiencies the denormalizer might be able to take advantage of; there's still the bound of what has to be written out. OK, so that doesn't really work. But it doesn't need to; the database could do this more lazily. But, doing it fully lazily is just the same as the case where you use the normalized data, so we can't do it fully lazily. Somehow the database needs to figure out what it should and should not precompute. Well, that doesn't sound like that big a deal, right? No, it's a nightmare to even begin to specify what that might be, in the face of full Turing chaos. I can almost see how one might try to apply some basic "learning" algorithms to the task of pre-computing things partially, but it's getting pretty complicated in the backend there. There's an awful lot of symbolic computation there where I really want things to just be computing for performance. Furthermore, if one insists on consistency in the denormalized fields... yeow, that's complicated. Again, it's not hard to specify the math but making it happen would be crazy insane. Oh, and I don't mean distributed consistency... I mean even consistency on a single busy host is going to be a challenge. A feasible challenge, certainly, but a challenge that shouldn't be ignored. It seems to me there might be a use case for this to be a bullet-point feature for a database, for developers to very, very carefully use on very high-use fields. But it wouldn't be very hard for a real-world denormalization engine to become a performance nightmare. Or to end up taking more "index tuning" to be usable engine than the code we already have.
- josephg 10y agoWhen an update happens the trick is going to be figuring out an update mechanism for the derived data without needing to run more queries. You're probably right - in the general case thats really hard to do automatically. But lots of aggregate values can be calculated based on an update without reference to the original set. `SUM()` and `MEAN()` are easy. `MAX()` and `MEDIAN()` are hard. In the case of the notification count query from blog post, you can do that efficiently with 3 map reduces: 1. Given a list of `(message, room, timestamp)` triples, generate `(room, max(timestamp))` tuples. In general max is expensive to calculate, but in this case the message collection is immutable, and the rule to update max(timestamp) when an edit happens doesn't need any extra processing based on an insert. 2. Given `(room, message_timestamp, read_timestamp)` generate `(room, message_timestamp > read_timestamp)`. Easy. 3. Then a reduce to calculate the sum. One of the things I really like about the multi-process architecture is the diversity of solutions we should have access to here. Right now your data lives in one vendor's system. Its really hard to use a map-reduce job if your primary data store doesn't support it. And map reduces might be too complicated for simple workloads - so with small data sets you might want something simpler. With this architecture different vendors could provide different indexing systems and you could swap them out as your requirements / data size changes. Finally as for consistency, I've got a bunch of ideas of different ways to make it work. Its not so bad because its not a generic graph or consensus problem. Its just a DAG. You can give each primary store a unique name and version using vector clocks. (If each primary store is itself distributed thats fine - just use whatever versioning system the primary store uses as the value for that database in the vector clock.) Then when a query comes in that needs to hit both an index and a primary store, pick the most recent database version and query asking for the result set at that version. You'll need some MVCC to make it work - but thats a solvable problem. If the named version is too new for the index, the query stalls until the index sees the corresponding update from the primary store. If the version is too old (and its expired out of the MVCC working set) then we punt back to whatever is running the query and it retries. The whole system is much simpler than a distributed consensus problem because each node in the DAG can be thought of as authoritative, even if they're individually implemented as a small redundant set.
- sixdimensional 10y agoYou may find this article about "turning the database inside-out" using Apache Samza to be quite interesting: http://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ http://www.confluent.io/blog/turning-the-database-inside-out... From my experience, a decoupling of the functions that a database management system provides (storage, optimization, handling of streams, etc.) and the ability to think about those capabilities as modules within a framework is a powerful concept.