2 ms·
Our software needs to scale and to have fallbacks when "infinite compute and memory" is not available. I used to detest when I was forced to make my code run on
by sitkack 12y ago
Our software needs to scale and to have fallbacks when "infinite compute and memory" is not available. I used to detest when I was forced to make my code run on machine with 512MB, I'd argue that we are already charging them tens of thousands of dollars, why not just ship them a couple sticks. 2G would make the code 10x as clean. But I was wrong.
Our code needs to scale down as well as up, but it should do so in the libraries and the abstractions. That is why "big data" was such an issue, we weren't working on abstractions that blurred the lines between in and out of core and now we are.
If we turn from imperative to declarative and any-typed to algebraic the semantics remain consistent and it is up to the runtime to fit the code to the available compute and memory. Cache Oblivious was a good start, memory and compute oblivious is the next level. A Lisp with an RDD [0] feels about right.
[0] http://www.cs.berkeley.edu/~pwendell/strataconf/api/core/spark/RDD.html http://www.cs.berkeley.edu/~pwendell/strataconf/api/core/spa...