3 ms·
This is a fairly uninteresting insight--if your data fits on one machine, use one machine to process it! But in many cases, your data does not fit on one machi
by necubi 14y ago
This is a fairly uninteresting insight--if your data fits on one machine, use one machine to process it!
But in many cases, your data does not fit on one machine, even one with gobs of ram and SSDs. In my work, we receive hundreds megs of data per second and our jobs run from hundreds of terabytes to petabytes of input data. That is the scale where "big data" ceases to be a meaningless buzz word and becomes a serious technical challenge, requiring distributed storage and processing over hundreds of machines.
Really, this is about the devaluation of "big data" to mean anything over a few gigabytes, which is a drastically simpler problem than peta-scale computing.