4 ms·
A few months ago I got my hadoop cluster stabilized (64 cores with 64 TB hdfs space). It wasn't a pleasant experience. The most memorable frustrating parts of
by seiji 16y ago
A few months ago I got my hadoop cluster stabilized (64 cores with 64 TB hdfs space). It wasn't a pleasant experience. The most memorable frustrating parts of the setup:
Searching for help online results in solutions for ancient versions or incomplete wiki pages ("we'll finish this soon!" from three years ago).
If Apple is one extreme end of the user friendly spectrum, hadoop is at the polar opposite end -- the error conditions and error messages can be downright hostile. The naming conventions are wonky too: namenode and secondary namenode, but the secondary isn't a backup, it's a copy. And don't get me started on tasktracker versus jobtracker (primarily because I can never remember the difference).
Restarting a tracker doesn't make it vanish in the namenode, so you have to restart the namenode too (at least in my CDH3 setup).
Everything is held together with duct tape shell scripts.
On the good side, I got everything hadoop related managed in puppet. All I need to do for a cluster upgrade is load a new CDH repo, reboot the cluster, then make sure nothing is borked.
If I didn't have to deal with isomorphic SQL<->hadoop queries, I'd start over using http://discoproject.org/ http://discoproject.org/
- xal 16y agoDisco looks incredible. Are you aware of any big production uses of it outside of nokia?
- DavidMcLaughlin 16y agoI work at Nokia and I'm not aware of any big production uses of it inside Nokia. Most of the teams in my part of the company use Hadoop. Oh well.
- vtuulos 16y agoActually the largest MapReduce cluster at Nokia runs Disco. Feel free to contact me at ville.h.tuulos at nokia dot com if you want to give it a try.
- phren0logy 16y agoIt seems to be a very active project: https://github.com/tuulos/disco/commits/master https://github.com/tuulos/disco/commits/master
- konradjg 16y agoI have to disagree completely with your example of horrible naming conventions regarding the jobtracker and tasktracker. The names are very descriptive and convey enough details to know what their purpose is. http://hadoop.apache.org/mapreduce/docs/current/mapred_tutorial.html#Job+Submission+and+Monitoring http://hadoop.apache.org/mapreduce/docs/current/mapred_tutor... "The MapReduce framework consists of a single master JobTracker and one slave TaskTracker per cluster-node. The master is responsible for scheduling the jobs' component tasks on the slaves, monitoring them and re-executing the failed tasks. The slaves execute the tasks as directed by the master." If your problems is in remembering the difference perhaps you have not spent enough time understanding the tools you are using.
- programnature 16y agoI think OP is saying he understands the difference in what they do, but can't keep the names straight. I felt the same way for the first few months of using hadoop. I eventually got that in real-life, a task is just a subpiece of a job. But the naming is not immediately evocative of the relationship, particularly since in computing, "job" is pretty overloaded. Far better would be "job" and "subjob".
- ajays 16y agoI think the word "job" comes from the old mainframe days. Each program was encapsulated in a "job". There was even a language on some systems (aptly named "Job Control Language", or JCL) to specify the different aspects of the job.
- hvs 16y agoI don't think the OP was saying he doesn't know what "job" means, he was saying that they use two terms "task" and "job" that mean the same thing, so he doesn't know what the difference between the trackers are.
- yid 16y agoTake a look at Hive if you need SQL semantics on Hadoop.
- nickik 16y agoBacktype have there own language for quering. https://github.com/nathanmarz/cascalog https://github.com/nathanmarz/cascalog