3 ms·
It all comes down to Query = Function(All data). The architecture I describe is a general approach for computing arbitrary functions on an arbitrary dataset (bo
by nathanmarz 14y ago
It all comes down to Query = Function(All data). The architecture I describe is a general approach for computing arbitrary functions on an arbitrary dataset (both in scale and content) in realtime. Any alternative to this approach must also be able to compute arbitrary functions on arbitrary data. Storing data in Cassandra while also indexing it in elastic search does not satisfy this, and it also fails to address the very serious complexity problems discussed in the first half of the presentation of mutability and conflating data with queries.
The Lambda Architecture is the only design I know of that:
1) Is based upon immutability at the core.
2) Cleanly separates how data is stored and modeled from how queries are satisfied.
3) Is general enough for computing arbitrary functions on arbitrary data (which encapsulates every possible data system)
- wellpast 14y agoOkay, sure -- if your need is realtime Query = Function(All data) -- then your approach may be necessary. But this requirement (realtime Function(All data)) is a rarity, I would think. And in this rare case, perhaps, the complexity you're introducing is warranted.
- DanWaterworth 14y agoI'm writing a distributed DBMS based on immutable data. I have a different way of providing fast realtime arbitrary querying of all data and it's much simpler. You just cache intermediate results. For example, if you want to know the sum of the elements in a tree, but you've already calculated the sum of the elements of another tree that differs on only one path, then you won't need to do much work, you can use the precomputed work for most of the tree and just perform calculations for the parts that are different.
- DanWaterworth 14y agoIn fact, my approach is more general than yours because it isn't restricted to catamorphisms.
- nathanmarz 14y agoAfraid not. Nothing is more general than function(all data).
- DanWaterworth 14y agoYou rely on mapreduce, so you don't have the ability to calculate any function. You can only calculate catamorphisms of your data, ( http://en.wikipedia.org/wiki/Catamorphism http://en.wikipedia.org/wiki/Catamorphism ). You can't solve a SAT problem effectively using mapreduce over the tree of the expression for example. I can't think of a case where you'd want to calculate a function that isn't a catamorphism, but saying that you aren't restricted to them is wrong.
- Evbn 14y agoIs "median" expressible as a catamorphism? What about a neural network propagation?
- DanWaterworth 14y agoFirst, let me explain what I mean by catamorphism. I mean a function that operates on the data within a node of a tree and the values returned by performing the catamorphism on its child nodes. To find the median, you need the data sorted. You can express something like mergesort as a catamorphism. You take the lists of the child nodes and the singleton list of the value at the node and merge them together. However you have to do a linear amount of work at each node and store a linear amount of data, so it's not really within the spirit of mapreduce because it doesn't scale well, although it is possible. Catamorphisms are operations on trees, so the first problem with neural networks is that you'd have to embed your neural network in a tree. In the worst possible case, a fully connected network, to find the next value of each neuron, you'll have to find the current value of every other neuron. The problem is that you don't have access to "cousin nodes'" data, so it isn't possible.
- richhickey 14y agoDatomic satisfies these three points.