3 ms·
you have a good point, but for what it's worth here's the practical example that is the basis of my opinion on this. I had a very large corpus of text and I wa
by kurtosis 18y ago
you have a good point, but for what it's worth here's the practical example that is the basis of my opinion on this. I had a very large corpus of text and I wanted to construct a word association graph. This involves multiplying together lots of sparse 10^5 x 10^9 matrices that are too large to fit in single memory. This is out of core.
Getting together 6 computers and setting up hadoop and HDFS took me about 2 days starting from nothing.
Figuring out how to do all of these out of core sparse matrix manipulations honestly took me about a week of tinkering before I had anything worth even trying. I'm certainly not an expert at this stuff so maybe a better coder could have done it a lot faster but that's my problem.
What would be a real advance in bringing massive scale data analysis to "the people" would be something like hadoop R or MATLAB that makes doing things like this completely automagic.
- rjurney 18y agoRegarding the real advance: That stuff is the very 'revolution' thats coming that I talk about :) Visicalc kindly sucked, but Excel is pretty good. Its still very early, but the way forward has become clear.