7 ms·

MapReduce is handy at terabyte-data volumes, but how often do we run jobs with input size > 1 TB? According to the paper: at least two analytics production clu

MapReduce is handy at terabyte-data volumes, but how often do we run jobs with input size > 1 TB? According to the paper: at least two analytics production clusters (at Microsoft and Yahoo) have median job input sizes under 14 GB, and 90% of jobs on a Facebook cluster have input sizes under 100 GB The other point the authors make is that DRAM costs are following Moore's law and terabyte-data workloads should "soon" be cost-feasible on single servers with DRAM.

chubot 13y ago
The thing is that MapReduce is still a very good programming paradigm for a single box. Machines have 8-64 cores these days -- you don't want to write multi-threaded code every time you want to do an analysis. So you can write MapReduce, but use a multicore framework instead of a cluster framework (hadoop). The unfortunate thing is that there is no popular open source implementation of a multicore mapreduce, so people use Hadoop, which is wasteful on small data sets, as mentioned. But the great part about it is that you will use the same application code for both. I fully expect in 5 years or so that people will be running multicore mapreduce jobs on 100 or 1000 core boxes.
- scott_s 13y ago
  Not all algorithms map well to MapReduce, which is the authors' main point. They explain one such example in the paper (section 3).
- eropple 13y ago
  > The unfortunate thing is that there is no popular open source implementation of a multicore mapreduce Not popular, I'd agree, but I have had a lot of success with one-off Akka projects. My mappers and reducers are usually under 10 lines of Scala (more if I'm stuck writing Java, obviously).
- kyzyl 13y ago
  There is Disco[1], which is a MR framework written in Python and Erlang. It's open source and pretty awesome, and if I'm not mistaken it will leverage multi-core processors, no need for a cluster. [1] http://discoproject.org/about http://discoproject.org/about
  - marshray 13y ago
    Speaking of Erlang, I've got my eye on Riak pipes. Now that the Basho folks have a large object store, I wonder if more in that department is a natural path.

chubot 13y ago

The thing is that MapReduce is still a very good programming paradigm for a single box. Machines have 8-64 cores these days -- you don't want to write multi-threaded code every time you want to do an analysis. So you can write MapReduce, but use a multicore framework instead of a cluster framework (hadoop). The unfortunate thing is that there is no popular open source implementation of a multicore mapreduce, so people use Hadoop, which is wasteful on small data sets, as mentioned. But the great part about it is that you will use the same application code for both. I fully expect in 5 years or so that people will be running multicore mapreduce jobs on 100 or 1000 core boxes.

scott_s 13y ago
Not all algorithms map well to MapReduce, which is the authors' main point. They explain one such example in the paper (section 3).
eropple 13y ago
> The unfortunate thing is that there is no popular open source implementation of a multicore mapreduce Not popular, I'd agree, but I have had a lot of success with one-off Akka projects. My mappers and reducers are usually under 10 lines of Scala (more if I'm stuck writing Java, obviously).
kyzyl 13y ago
There is Disco[1], which is a MR framework written in Python and Erlang. It's open source and pretty awesome, and if I'm not mistaken it will leverage multi-core processors, no need for a cluster. [1] http://discoproject.org/about http://discoproject.org/about
- marshray 13y ago
  Speaking of Erlang, I've got my eye on Riak pipes. Now that the Basho folks have a large object store, I wonder if more in that department is a natural path.