3 ms·

This is quite an interesting document. I'd love to have pov from Hadoop "power-user" on this.

blibble 14y ago
I'd agree with most of his points, but he has conflated mapreduce, yarn and hdfs into "hadoop". the "assembly language" stuff is only accurate if you're writing raw mapreduce jobs, but he's ignored projects like hbase, hive, pig, impala and cascading. the HA issues with the namenode have mostly gone away with CDH4.1. the one thing that's a complete pain is the lack of snapshotting functionality in hdfs, making consistent backups nearly impossible. the APIs for hdfs and yarn are poorly designed and pretty buggy, they're very obtuse and hard to debug. if you manage to find any documentation it's almost certainly wrong for the version you're using. nine times out of ten it's quicker to step into the source code rather than digging into the documentation. however once you've abstracted away the poor APIs it does work very well, and the pay is good...
shuzchen 14y ago
I'm not a "power-user", but I'm in the process of building a pretty big product powered by hadoop (more specifically, mrjob and amazon's EMR). I have a hard time reading this as anything but satire. The author definitely knows a lot about hadoop, yet displays a misunderstanding of such a basic piece of it (conflating hadoop with map-reduce). Just a few key things I can refute: Hadoop as wasteful: production systems are built to withstand the expected peaks of traffic (i.e., some website that sees a huge influx when the east coast wakes up on monday morning will design with that load+buffer in mind), which means that ~90% (probably even higher!) of the time resources are underutilized. even personally my own laptop isn't chugging at 100% on all cores all day every day, I don't see how anybody expects any collection of computers to do so. On the flip side, this is being offset by platforms like AWS. When I run map-reduce jobs on hadoop, I spin up an EMR jobflow, run a few tasks and then shut it down. Presumably, somebody else gets to use those machines after I'm done. Lack of third party support: Honestly, with mrjob and EMR, I've never needed to deal with any hadoop management, setup, maintenance or scaling. Also, all my code is python, I've not had to write a line of Java. And these aren't the only tools built on top of hadoop. Hadoop as deprecated: there are certainly more big-data crunching platforms coming out nowadays, but hadoop is mature, stable, and still going strong. Also, the basic map-reduce framework it's based on is not going anywhere soon. I mean, I cut my teeth on map-reduce with mongodb, and it seems most other nosql solutions there are going that route for complicated things their standard query engine can't support.

I'd agree with most of his points, but he has conflated mapreduce, yarn and hdfs into "hadoop". the "assembly language" stuff is only accurate if you're writing raw mapreduce jobs, but he's ignored projects like hbase, hive, pig, impala and cascading. the HA issues with the namenode have mostly gone away with CDH4.1. the one thing that's a complete pain is the lack of snapshotting functionality in hdfs, making consistent backups nearly impossible. the APIs for hdfs and yarn are poorly designed and pretty buggy, they're very obtuse and hard to debug. if you manage to find any documentation it's almost certainly wrong for the version you're using. nine times out of ten it's quicker to step into the source code rather than digging into the documentation. however once you've abstracted away the poor APIs it does work very well, and the pay is good...

shuzchen 14y ago

I'm not a "power-user", but I'm in the process of building a pretty big product powered by hadoop (more specifically, mrjob and amazon's EMR). I have a hard time reading this as anything but satire. The author definitely knows a lot about hadoop, yet displays a misunderstanding of such a basic piece of it (conflating hadoop with map-reduce). Just a few key things I can refute: Hadoop as wasteful: production systems are built to withstand the expected peaks of traffic (i.e., some website that sees a huge influx when the east coast wakes up on monday morning will design with that load+buffer in mind), which means that ~90% (probably even higher!) of the time resources are underutilized. even personally my own laptop isn't chugging at 100% on all cores all day every day, I don't see how anybody expects any collection of computers to do so. On the flip side, this is being offset by platforms like AWS. When I run map-reduce jobs on hadoop, I spin up an EMR jobflow, run a few tasks and then shut it down. Presumably, somebody else gets to use those machines after I'm done. Lack of third party support: Honestly, with mrjob and EMR, I've never needed to deal with any hadoop management, setup, maintenance or scaling. Also, all my code is python, I've not had to write a line of Java. And these aren't the only tools built on top of hadoop. Hadoop as deprecated: there are certainly more big-data crunching platforms coming out nowadays, but hadoop is mature, stable, and still going strong. Also, the basic map-reduce framework it's based on is not going anywhere soon. I mean, I cut my teeth on map-reduce with mongodb, and it seems most other nosql solutions there are going that route for complicated things their standard query engine can't support.