3 ms·

Three big myths about MapReduce

20 points by timf 17y ago

mahmud 17y ago
Good read: Right now, this is a distinction pretty much without a difference. If you choose an implementation of MapReduce — like pure Hadoop (say in the Cloudera distribution) or Hadoop-Vertica or Aster Data’s SQL/MapReduce – you’re basically picking an entire technology stack. But those stacks are going to do a whole lot of changing and maturing in the near future – and as they do, it’s likely that projects will interact or even combine in all sorts of interesting ways.
jfager 17y ago
Strawman. Of course mapping followed by reducing isn't new, nobody serious ever claimed that it was (the Dean & Ghemawat paper certainly doesn't). Systems dedicated to performing map-reduce operations using giant clusters of commodity hardware over tera/petabytes of data are, though, if for no other reason than that the presence of and resources to economically store that much data have only recently become widely available. MapReduce, as a term, refers to those systems, not just the act of mapping and reducing.
mumrah 17y ago
For the longest time, after I read about MapReduce, I wondered what the big fuss was about. MPI has equivalent functionality (Broadcast/Scatter and Reduce) along with many other useful high-level communication functions. It does restrict you to a limited (and unpopular) set of languages for web programming though.
- vicaya 17y ago
  MapReduce is about hiding explicit parallel primitives like threads and MPI from users (programmers): You only need to implement two functions map and reduce in a typical framework. Yes, you can certainly implement MapReduce with MPI partially, but typical MapReduce framework like Hadoop also handles fault tolerance transparently to users.
jmount 17y ago
A good point, but while Map Reduce is not new I feel it emphasized clarity and simplicity (at least for the problem of sorting), so that is probably why it markets easier than MPI or a database. I wrote a bit on this point some time ago: http://www.win-vector.com/blog/2009/01/map-reduce-a-good-idea/ http://www.win-vector.com/blog/2009/01/map-reduce-a-good-ide...

20 points by timf 17y ago

Good read: Right now, this is a distinction pretty much without a difference. If you choose an implementation of MapReduce — like pure Hadoop (say in the Cloudera distribution) or Hadoop-Vertica or Aster Data’s SQL/MapReduce – you’re basically picking an entire technology stack. But those stacks are going to do a whole lot of changing and maturing in the near future – and as they do, it’s likely that projects will interact or even combine in all sorts of interesting ways.

jfager 17y ago

Strawman. Of course mapping followed by reducing isn't new, nobody serious ever claimed that it was (the Dean & Ghemawat paper certainly doesn't). Systems dedicated to performing map-reduce operations using giant clusters of commodity hardware over tera/petabytes of data are, though, if for no other reason than that the presence of and resources to economically store that much data have only recently become widely available. MapReduce, as a term, refers to those systems, not just the act of mapping and reducing.

mumrah 17y ago

For the longest time, after I read about MapReduce, I wondered what the big fuss was about. MPI has equivalent functionality (Broadcast/Scatter and Reduce) along with many other useful high-level communication functions. It does restrict you to a limited (and unpopular) set of languages for web programming though.

vicaya 17y ago
MapReduce is about hiding explicit parallel primitives like threads and MPI from users (programmers): You only need to implement two functions map and reduce in a typical framework. Yes, you can certainly implement MapReduce with MPI partially, but typical MapReduce framework like Hadoop also handles fault tolerance transparently to users.

jmount 17y ago

A good point, but while Map Reduce is not new I feel it emphasized clarity and simplicity (at least for the problem of sorting), so that is probably why it markets easier than MPI or a database. I wrote a bit on this point some time ago: http://www.win-vector.com/blog/2009/01/map-reduce-a-good-idea/ http://www.win-vector.com/blog/2009/01/map-reduce-a-good-ide...