3 ms·

Hadoop Every idiot out there wants to spray some "big data" retardation all over whatever they're doing, so they throw together a cluster -- if one were really

by earl 16y ago

Hadoop Every idiot out there wants to spray some "big data" retardation all over whatever they're doing, so they throw together a cluster -- if one were really generous -- of a handful of machines so they can use hadoop too! Meh. It's not big data until you need a couple hundred cores. However -- it's an opportunity to cash in for devs. Of course hadoop is a needlessly difficult programming model for most people, but never mind that. My .02 is if you can possibly solve your problem by scaling up instead of out, then scale up. Hadoop works well precisely for perfectly parallelizable no communication required problems, which are few and far between. For everything else, it's a giant pita to shoehorn the solution into hadoop. Of course, most managers are retards, so spending maybe $25K on a 256GB ram, 16 xeon machine stuffed full of 15k rpm sas or ssd drives is out of the question, but wasting months of dev time to sprinkle Real Big Data (TM) bullshit on stuff is worthwhile. Sigh. 15 years into this industry and still, everybody views salary expenses / opportunity costs of dev time and equipment as completely separate things. I'm grumpy today. However, hadoop has a steep learning curve for anything but toy problems, and I get 2-3 requests a week on linkedin just because of my employer and having hadoop on my resume. So it's great for devs. I know people charging hundreds of dollars as their consulting rates. Edit: just checked silicon mechanics to make sure my hardware cost wasn't too far off. If you're doing most numerical stuff, parallelization is hard, so: 256GB RAM, 2 of the fastest QPI highest cache procs, and 6x intel 160GB SSDs in a 2U: less than $30K delivered.

sitmack 16y ago
As a highering manager (who uses Hadoop) it easy for me to request more head count at a 200k cost to the company. It is not even the same order of magnitude amount of effort to request hardware at 1/4 the price. What I could get done with more hardware...
rubinelli 16y ago
Personally, I don't think Hadoop's model is all that complex (obviously, if you can solve the same problem with a quick script, you have to be insane not to) but it's nowhere near in terms of polish to the tools Enterprise Java devs are used to. Creating a new cluster for example is a pain compared to the usual "unzip, copy datasource XMLs and jars, run start" for application servers.
- earl 16y ago
  Hadoop is extremely complex to do anything requiring communication. My favorite example is ranking: say you have a set of <set (string), id (string), score (float in [0,1])> tuples and for each unique class, you want to rank ids by their scores. This is trivial on a single core. Now try doing it on hadoop: first thing you'll do is try to do it all on the head node, but that won't work if you're doing this on more than a trivial amount of data. Next you may try partitioning by score, but scores aren't distributed uniformly. Partitioning by set won't work since you have more data than a single node can possibly handle. It took me a week's worth of work to write a distributed ranker that will work on billions of ids in hundreds of sets; the algorithm is not obvious. On a single node you'd write a for loop that did a sort. Those are the sort of obstacles that the hadoop programming model introduces. Again, if you haven't used it for real work, you'll be surprised at how few tasks are truly parallelizable without communication. The model is very simple, but accomplishing tasks that don't quite fit in the model is difficult. Another example is a matrix mult. There is no way to segment the data to eliminate internode communication. It can be accomplished in an iterative fashion, but it's decidedly nontrivial. Or most optimization problems -- I don't think there even are known algorithms for no communication parallelization.

by earl 16y ago

sitmack 16y ago

As a highering manager (who uses Hadoop) it easy for me to request more head count at a 200k cost to the company. It is not even the same order of magnitude amount of effort to request hardware at 1/4 the price. What I could get done with more hardware...

rubinelli 16y ago

Personally, I don't think Hadoop's model is all that complex (obviously, if you can solve the same problem with a quick script, you have to be insane not to) but it's nowhere near in terms of polish to the tools Enterprise Java devs are used to. Creating a new cluster for example is a pain compared to the usual "unzip, copy datasource XMLs and jars, run start" for application servers.

earl 16y ago
Hadoop is extremely complex to do anything requiring communication. My favorite example is ranking: say you have a set of <set (string), id (string), score (float in [0,1])> tuples and for each unique class, you want to rank ids by their scores. This is trivial on a single core. Now try doing it on hadoop: first thing you'll do is try to do it all on the head node, but that won't work if you're doing this on more than a trivial amount of data. Next you may try partitioning by score, but scores aren't distributed uniformly. Partitioning by set won't work since you have more data than a single node can possibly handle. It took me a week's worth of work to write a distributed ranker that will work on billions of ids in hundreds of sets; the algorithm is not obvious. On a single node you'd write a for loop that did a sort. Those are the sort of obstacles that the hadoop programming model introduces. Again, if you haven't used it for real work, you'll be surprised at how few tasks are truly parallelizable without communication. The model is very simple, but accomplishing tasks that don't quite fit in the model is difficult. Another example is a matrix mult. There is no way to segment the data to eliminate internode communication. It can be accomplished in an iterative fashion, but it's decidedly nontrivial. Or most optimization problems -- I don't think there even are known algorithms for no communication parallelization.