3 ms·

Personally, I don't think Hadoop's model is all that complex (obviously, if you can solve the same problem with a quick script, you have to be insane not to) bu

Personally, I don't think Hadoop's model is all that complex (obviously, if you can solve the same problem with a quick script, you have to be insane not to) but it's nowhere near in terms of polish to the tools Enterprise Java devs are used to. Creating a new cluster for example is a pain compared to the usual "unzip, copy datasource XMLs and jars, run start" for application servers.

earl 16y ago
Hadoop is extremely complex to do anything requiring communication. My favorite example is ranking: say you have a set of <set (string), id (string), score (float in [0,1])> tuples and for each unique class, you want to rank ids by their scores. This is trivial on a single core. Now try doing it on hadoop: first thing you'll do is try to do it all on the head node, but that won't work if you're doing this on more than a trivial amount of data. Next you may try partitioning by score, but scores aren't distributed uniformly. Partitioning by set won't work since you have more data than a single node can possibly handle. It took me a week's worth of work to write a distributed ranker that will work on billions of ids in hundreds of sets; the algorithm is not obvious. On a single node you'd write a for loop that did a sort. Those are the sort of obstacles that the hadoop programming model introduces. Again, if you haven't used it for real work, you'll be surprised at how few tasks are truly parallelizable without communication. The model is very simple, but accomplishing tasks that don't quite fit in the model is difficult. Another example is a matrix mult. There is no way to segment the data to eliminate internode communication. It can be accomplished in an iterative fashion, but it's decidedly nontrivial. Or most optimization problems -- I don't think there even are known algorithms for no communication parallelization.

earl 16y ago

Hadoop is extremely complex to do anything requiring communication. My favorite example is ranking: say you have a set of <set (string), id (string), score (float in [0,1])> tuples and for each unique class, you want to rank ids by their scores. This is trivial on a single core. Now try doing it on hadoop: first thing you'll do is try to do it all on the head node, but that won't work if you're doing this on more than a trivial amount of data. Next you may try partitioning by score, but scores aren't distributed uniformly. Partitioning by set won't work since you have more data than a single node can possibly handle. It took me a week's worth of work to write a distributed ranker that will work on billions of ids in hundreds of sets; the algorithm is not obvious. On a single node you'd write a for loop that did a sort. Those are the sort of obstacles that the hadoop programming model introduces. Again, if you haven't used it for real work, you'll be surprised at how few tasks are truly parallelizable without communication. The model is very simple, but accomplishing tasks that don't quite fit in the model is difficult. Another example is a matrix mult. There is no way to segment the data to eliminate internode communication. It can be accomplished in an iterative fashion, but it's decidedly nontrivial. Or most optimization problems -- I don't think there even are known algorithms for no communication parallelization.