3 ms·
My 2 cents. Compute and storage capabilities keep growing rapidly, if one structures their data well, uses a reasonable query processor, and some form of under
by lumost 6y ago
My 2 cents.
Compute and storage capabilities keep growing rapidly, if one structures their data well, uses a reasonable query processor, and some form of underlying columnar storage then computing calculations on TBs of data can be accomplished in seconds for low costs.
Being able to recompute the world from scratch is a p0 requirement for most analytic workloads, as otherwise migrations and un-forseen computation changes due to new product requirements and other activities become painful.
This leaves differential techniques in an awkward spot where to be effective they need to
1) Operate on vast quantities of data or sufficiently complex calculations such that optimization of compute is a concern to the end-user.
2) Operate in a computational environment that is sufficiently constrained such that all present and future changes to the computation can be reasonably accounted for.
3) Be transparent enough that engineers don't feel that they are duplicating logic.
It's hard to think of applications that would meet this criteria outside of intelligent caching of computation within DB engines.
- polskibus 6y agoDepends how you look at it. Single thread perf is pretty much stagnant. Only recently ryzen has budged multi core perf, we're still to see how long this growth path can be. I agree that cloud has made computing power more accessible, but there are known limits regarding scalability. Moreover, using distributed computing (spark, etc) kills efficiency (see all those posts about a laptop beating a medium sized big data cluster).
- igorkraw 6y agoIf I am not misremembering, "all those blog posts" are all written by Frank McSherry, one of the cofounders of materialize materialize and developer of differential dataflow :-) http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf
- lumost 6y agoVery few data processing activities are bottlenecked by single thread performance. Similarly as memory, thread count, and storage capacities of modern servers have continued Moore's law the number of applications that can fit within the same finite cost has also improved. In terms of efficiency, the big question is "does an incremental gain in efficiency matter for this application?" if we're talking about < 10x performance/cost change the answer will be no for most teams. Consider how many of these big data applications are implemented in python or other interpreted languages.