3 ms·
The thing is that MapReduce is still a very good programming paradigm for a single box. Machines have 8-64 cores these days -- you don't want to write multi-th
by chubot 14y ago
The thing is that MapReduce is still a very good programming paradigm for a single box.
Machines have 8-64 cores these days -- you don't want to write multi-threaded code every time you want to do an analysis. So you can write MapReduce, but use a multicore framework instead of a cluster framework (hadoop).
The unfortunate thing is that there is no popular open source implementation of a multicore mapreduce, so people use Hadoop, which is wasteful on small data sets, as mentioned.
But the great part about it is that you will use the same application code for both. I fully expect in 5 years or so that people will be running multicore mapreduce jobs on 100 or 1000 core boxes.
- scott_s 14y agoNot all algorithms map well to MapReduce, which is the authors' main point. They explain one such example in the paper (section 3).
- eropple 14y ago> The unfortunate thing is that there is no popular open source implementation of a multicore mapreduce Not popular, I'd agree, but I have had a lot of success with one-off Akka projects. My mappers and reducers are usually under 10 lines of Scala (more if I'm stuck writing Java, obviously).
- kyzyl 14y agoThere is Disco[1], which is a MR framework written in Python and Erlang. It's open source and pretty awesome, and if I'm not mistaken it will leverage multi-core processors, no need for a cluster. [1] http://discoproject.org/about http://discoproject.org/about
- marshray 14y agoSpeaking of Erlang, I've got my eye on Riak pipes. Now that the Basho folks have a large object store, I wonder if more in that department is a natural path.