3 ms·

MapReduce is difficult to implement well. It's just ironic that the hard part is the part that's not in the name: the shuffle phase.

srean 12y ago
Heartily agreed ! Or at least Hadoop does it poorly. It is really that bad. I say this with hesitation and a lot of reluctance, its an open source project, I have not contributed anything towards it, so it is really unfair of me to complain. Here is my personal experience with these tools at Google and at Yahoo (lightly edited from an old comment) == I have had the opportunity to try out Google's implementation of mapreduce implemented in C++ way back in time (6 years ago). These would run on fairly impoverished processors, essentially laptop grade of that time. Have done stuff on Yahoo's Hadoop setup as well, these used high end multicore machines provisioned with oodles of RAM. If I were to be generous, Hadoop ran 4 times slower as measured by wall clock times. Not only that, Hadoop required about 4 times more memory for similar sized jobs. So you ended up requiring more RAM, running for longer and potentially burning more electricity. This is by no means a benchmark or anything like that, just an anecdote. That Hadoop would require much more memory did not surprise me, that was expected. What was really surprising was that it was so much slower. Four times might not seem like much, but I was being generous to Hadoop. It makes a big difference when you can make multiple run through the data in a single day and make changes to the code/model. Debugging and ironing out issues is a lot more efficient when your iteration loop is shorter. I think Hadoop (by virtue of its comparative crappiness) gave Google a significant competitive edge over the rest, probably still does.
timr 12y ago
A big part of implementing the shuffle correctly at a large scale is having a good distributed filesystem upon which to build.
- dsymonds 12y ago
  Sure. I didn't mean to imply a good distributed FS was unnecessary or easy. Just saying that MR isn't easy.

srean 12y ago

Heartily agreed ! Or at least Hadoop does it poorly. It is really that bad. I say this with hesitation and a lot of reluctance, its an open source project, I have not contributed anything towards it, so it is really unfair of me to complain. Here is my personal experience with these tools at Google and at Yahoo (lightly edited from an old comment) == I have had the opportunity to try out Google's implementation of mapreduce implemented in C++ way back in time (6 years ago). These would run on fairly impoverished processors, essentially laptop grade of that time. Have done stuff on Yahoo's Hadoop setup as well, these used high end multicore machines provisioned with oodles of RAM. If I were to be generous, Hadoop ran 4 times slower as measured by wall clock times. Not only that, Hadoop required about 4 times more memory for similar sized jobs. So you ended up requiring more RAM, running for longer and potentially burning more electricity. This is by no means a benchmark or anything like that, just an anecdote. That Hadoop would require much more memory did not surprise me, that was expected. What was really surprising was that it was so much slower. Four times might not seem like much, but I was being generous to Hadoop. It makes a big difference when you can make multiple run through the data in a single day and make changes to the code/model. Debugging and ironing out issues is a lot more efficient when your iteration loop is shorter. I think Hadoop (by virtue of its comparative crappiness) gave Google a significant competitive edge over the rest, probably still does.

timr 12y ago

A big part of implementing the shuffle correctly at a large scale is having a good distributed filesystem upon which to build.

dsymonds 12y ago
Sure. I didn't mean to imply a good distributed FS was unnecessary or easy. Just saying that MR isn't easy.