6 ms·

The Dark Side of Hadoop

132 points by nadahalli 15y ago

rjurney 15y ago
I'd like to know what distribution they are running, and if they've tried others.
- nathanmarz 15y ago
  We used to use CDH3b2, but in the past month we switched to EMR. These and other issues existed in both distributions.
kordless 15y ago
If you want general distributed process management, check out Mesos: http://www.mesosproject.org/ http://www.mesosproject.org/ The guys working on it are over at Twitter nowadays.
seiji 15y ago
A few months ago I got my hadoop cluster stabilized (64 cores with 64 TB hdfs space). It wasn't a pleasant experience. The most memorable frustrating parts of the setup: Searching for help online results in solutions for ancient versions or incomplete wiki pages ("we'll finish this soon!" from three years ago). If Apple is one extreme end of the user friendly spectrum, hadoop is at the polar opposite end -- the error conditions and error messages can be downright hostile. The naming conventions are wonky too: namenode and secondary namenode, but the secondary isn't a backup, it's a copy. And don't get me started on tasktracker versus jobtracker (primarily because I can never remember the difference). Restarting a tracker doesn't make it vanish in the namenode, so you have to restart the namenode too (at least in my CDH3 setup). Everything is held together with duct tape shell scripts. On the good side, I got everything hadoop related managed in puppet. All I need to do for a cluster upgrade is load a new CDH repo, reboot the cluster, then make sure nothing is borked. If I didn't have to deal with isomorphic SQL<->hadoop queries, I'd start over using http://discoproject.org/ http://discoproject.org/
- xal 15y ago
  Disco looks incredible. Are you aware of any big production uses of it outside of nokia?
  - DavidMcLaughlin 15y ago
    I work at Nokia and I'm not aware of any big production uses of it inside Nokia. Most of the teams in my part of the company use Hadoop. Oh well.
    - vtuulos 15y ago
      Actually the largest MapReduce cluster at Nokia runs Disco. Feel free to contact me at ville.h.tuulos at nokia dot com if you want to give it a try.
  - phren0logy 15y ago
    It seems to be a very active project: https://github.com/tuulos/disco/commits/master https://github.com/tuulos/disco/commits/master
- konradjg 15y ago
  I have to disagree completely with your example of horrible naming conventions regarding the jobtracker and tasktracker. The names are very descriptive and convey enough details to know what their purpose is. http://hadoop.apache.org/mapreduce/docs/current/mapred_tutorial.html#Job+Submission+and+Monitoring http://hadoop.apache.org/mapreduce/docs/current/mapred_tutor... "The MapReduce framework consists of a single master JobTracker and one slave TaskTracker per cluster-node. The master is responsible for scheduling the jobs' component tasks on the slaves, monitoring them and re-executing the failed tasks. The slaves execute the tasks as directed by the master." If your problems is in remembering the difference perhaps you have not spent enough time understanding the tools you are using.
  - programnature 15y ago
    I think OP is saying he understands the difference in what they do, but can't keep the names straight. I felt the same way for the first few months of using hadoop. I eventually got that in real-life, a task is just a subpiece of a job. But the naming is not immediately evocative of the relationship, particularly since in computing, "job" is pretty overloaded. Far better would be "job" and "subjob".
earl 15y ago
Yeah, hadoop is a bear to get working. One of the benefits of working at a more established company like quantcast, google, or presumably facebook is there are internal infrastructure teams to smooth all that over. I pretty much get to type submitJob and it more or less just works...
- technomancy 15y ago
  It would probably work better in a large company. We spent a couple weeks evaluating Hadoop in our small startup and came to the conclusion that we couldn't afford to have such an infrastructure team, and I imagine most startups considering Hadoop are in a similar boat.
  - harryh 15y ago
    We use hadoop at foursquare and it was set up and maintained by 1 person. It was a real bear of a job but he got it done.
    - tluyben2 15y ago
      It's definitely doable. It's just a lot of work.
- jshen 15y ago
  I'm at a big company that uses hadoop, and someone else maintains the infrastructure. But, the problem we have is that hadoop is a very leaky abstraction and it's very easy for someone that doesn't know implementation details to break the grid. It's relatively easy to exhaust the memory of the jobtracker, we constantly have problems with mapper starvation because someones seemingly innocent job is causing issues, etc, etc. My experience is that it doesn't just work, even when someone else maintains the infrastructure.
alecco 15y ago
It's good to have critical reviews. But perhaps it could have been better if the tone were more respectful towards a free and open source project. The approach of Hadoop is not my cup of tea, but I praise them for giving for free a working product solving such a hard problem. Playing Hadoop's side, where are the test cases, the patches or bug reports? Or even some missing documentation blurbs, like you mention.
- seiji 15y ago
  We've become spoiled. open source has different levels ranging from "made in basement and released to the world" (one maintainer, patches welcome) to "throw over the fence" (you can see the code, but nobody cares about patches) to "we're an open source corporation" (apache, patches welcome if you follow steps 1 to 22). Hadoop falls under apache governance which means jira, contributor license agreements, and spending a week of your life trying to get your point across. It's much easier to complain like the spoiled developers we've become.
  - timr 15y ago
    It's much easier to complain like the spoiled developers we've become." By your own comments, that's not really fair. The Apache project has become a disaster area of bureaucracy, crappy code and crappier documentation. You have to spend weeks in the muck to get to the point where you can even properly understand the deficiencies of any given Apache subproject, and by that point, you're so bound up in the project that it's hard to move away. From the build system of the code to the structure of the community, not a single thing about Apache is what I would characterize as nimble or lightweight. Can you blame people for feeling helpless? It's gotten to the point where I'll look anywhere for a competing implementation of an idea before using Apache code to do something. It's just not worth the pain.
    - blasdel 15y ago
      I feel exactly the same way about Apache projects, and even now it's very rare to meet a fellow traveler. Years ago I had the same emperor has no clothes realization that the ASF was an enormous BureaucraticBullshitFactoryFactory — that it was seemingly founded just to host enterprise middleware crap while trading on the name of httpd (without really developing it of course). It took quite a while before I realized that the Apache Software Foundation was just the new IBM — and not just figuratively, if you actually looked at the active membership of the ASF, it was chock full of IBM employees contributing on company time.
forgotusername 15y ago
> This means that the Hadoop process which was using 1GB will temporarily "use" 2GB when it shells out. This is exactly not what happens, the new process copies only the parents address mappings (now marked read-only), which represents vastly less than 1gb physical memory. I think only a single 4kb page or two will ultimately be copied, representing a chunk of the calling thread's stack used to prepare the new child before finally calling execve() or similar.
- nathanmarz 15y ago
  As far as I understand it (I could be wrong as I've pieced this together from multiple sources), forking still reserves the same amount of memory that the parent process is using without actually copying the data (since it could very well use all that memory eventually). The wrench is that Linux has a feature called memory overcommit (which I haven't been able to decipher completely). Supposedly it causes forking to not actually reserve that much space, but by default it's in a "heuristic" mode so it may or may not take effect. These are the best resources I could find on what happens when you fork: http://developers.sun.com/solaris/articles/subprocess/subprocess.html http://developers.sun.com/solaris/articles/subprocess/subpro... http://lxr.linux.no/linux/Documentation/vm/overcommit-accounting http://lxr.linux.no/linux/Documentation/vm/overcommit-accoun...
  - blocke 15y ago
    I'm not anywhere near being a Java expert but I've seen similar behavior with a non-Hadoop workload. In my case I had several virtual machines hosting Java VMs with 2+GB heaps running an app that liked to fork and run external programs for short periods of time. If the entire heapsize was say 3GB and it forked twice Linux would act like it needed 9GB. The Linux overcommit heuristic regularly got things wrong and wanted RAM that it would never actually use. This usually resulted in the JVM failing in new and interesting ways. The workaround is to allocate a crapload of swap (mainly more than heap size times a guestimate of number of concurrent forks). It will never actually USE the swap but having it there seems keep the overcommit heuristic happy. Yay for cargo cult server tuning. I've never figured out how to get the kernel to not be so pessimistic and I can't modify the Java code in question so... eh.
jamii 15y ago
I would be interested to hear why riak, disco etc aren't viable alternatives. I've seen very few good comparisons of the various options (the Mozilla data blog being the only one that comes to mind).
- reinhardt 15y ago
  "Risking" a downvote for a basically "me-too" reply, I would also be very interested in such a comparison or even a simple blog post of the experience (good or bad) with alternatives, especially disco.
dougb 15y ago
I've been using Hadoop in production for about 3 years on a small 48 node cluster. I've seen many issues, but none of the ones mentioned in the blog post. My general theory is that if its an important tool for your business, you need at least 1 person to be an expert on it. The alternative is to pay Cloudera a significant amount per node for support. Another possible alternative is to use http://www.MapR.com/ http://www.MapR.com/, they are in beta and claim to be api compatible with Hadoop, but they are not free.
ssn 15y ago
Shouldn't Hadoop users help to document it and contribute to it? Why isn't this happening?
- code_duck 15y ago
  A lot of users of Hadoop are companies with serious competition. If you took the time to figure all of this out, would you go post a howto for your competitors? No, might as well let them expend their own time and money to do what you did. While that's not exactly the open source spirit, it is how plenty of people think and also happens to be the default, easiest thing to do. It would be nice if people could share to benefit themselves and each other.
api 15y ago
I never got the Hadoop craze. It performs terribly on memory and CPU, which is odd for something that prides itself on being HPC-oriented.
- tluyben2 15y ago
  Alternatives? Edit: Real alternatives I mean; something OSS (free is not so important, but no lock-in), able to manage huge amounts of data, actively developed, active projects built on top.
firebones 15y ago
We've seen all of these issues, but the primary causes appear to be either problems in configuration of the cluster (which you typically don't see until you're doing serious work, or working with an overstressed cluster at scale) and problems with underlying code quality of the jobs (e.g., processes hanging due to tasks which don't terminate due to inability to handle unexpected input gracefully--infinite loops or non-terminal operations). If you're working with big data, particularly data that isn't the cleanest, you'll start to see some of these issues and others arise. Despite those issues, the most remarkable thing about Hadoop is the out-of-the-box resilience to get the work done. The strategy of no side-effects and a write-only approach (with failed tasks discarding work in progress) ensures predictable results--even if the time it takes to get those results can't be guaranteed. The documentation isn't the greatest, and it's very confusing sorting out the sedimentary nature of the APIs and configuration (knowing which APIs and config match up across various versions such as 0.15, 0.17, 0.20.2, 0.21, etc., not to mention various distributions from Cloudera, Apache and Yahoo branches), but things are starting to finally converge. You're probably better off starting with one of the later, curated releases (such as the recent Cloudera distribution) where some work has been done to cherry pick features and patches from the main branches.
nivertech 15y ago
Hadoop is so old school... Even Google moving away from Map/Reduce. Prepare for new shiny things, which will be better, faster and cheaper to operate!
tluyben2 15y ago
Memory; it's Java. Java is great, but is a memory hog. Doesn't matter so much these days (well it does, but what are you going to do... There isn't much OSS competition for Hadoop). The documentation is horrible though. I'm not sure if this is a 'new' OSS trend, but now that i'm working a lot with Rails and gems I notice that on that side of OSS it's pretty normal to produce no or completely horrible documentation; use the force read the source and that kind of hilarious tripe. Apache projects which are Hadoop related (Hadoop, Pig, Hbase) all suffer from this; you are hard pressed to find anything remotely helpful and not incredibly outdated. At least for rails you can find tons of examples (no docs though) of how to achieve things; for Hadoop/Hbase everything is outdated, not functional and requiring you to jump into tons of code to get stuff done. Again; there is not so much competition for tasks you would accomplish with Hadoop (and Hbase) on the scale it has been tested (by Yahoo/Stumbleupon and many others).

132 points by nadahalli 15y ago

rjurney 15y ago

I'd like to know what distribution they are running, and if they've tried others.

nathanmarz 15y ago
We used to use CDH3b2, but in the past month we switched to EMR. These and other issues existed in both distributions.

kordless 15y ago

If you want general distributed process management, check out Mesos: http://www.mesosproject.org/ http://www.mesosproject.org/ The guys working on it are over at Twitter nowadays.

seiji 15y ago

A few months ago I got my hadoop cluster stabilized (64 cores with 64 TB hdfs space). It wasn't a pleasant experience. The most memorable frustrating parts of the setup: Searching for help online results in solutions for ancient versions or incomplete wiki pages ("we'll finish this soon!" from three years ago). If Apple is one extreme end of the user friendly spectrum, hadoop is at the polar opposite end -- the error conditions and error messages can be downright hostile. The naming conventions are wonky too: namenode and secondary namenode, but the secondary isn't a backup, it's a copy. And don't get me started on tasktracker versus jobtracker (primarily because I can never remember the difference). Restarting a tracker doesn't make it vanish in the namenode, so you have to restart the namenode too (at least in my CDH3 setup). Everything is held together with duct tape shell scripts. On the good side, I got everything hadoop related managed in puppet. All I need to do for a cluster upgrade is load a new CDH repo, reboot the cluster, then make sure nothing is borked. If I didn't have to deal with isomorphic SQL<->hadoop queries, I'd start over using http://discoproject.org/ http://discoproject.org/

xal 15y ago
Disco looks incredible. Are you aware of any big production uses of it outside of nokia?
- DavidMcLaughlin 15y ago
  I work at Nokia and I'm not aware of any big production uses of it inside Nokia. Most of the teams in my part of the company use Hadoop. Oh well.
  - vtuulos 15y ago
    Actually the largest MapReduce cluster at Nokia runs Disco. Feel free to contact me at ville.h.tuulos at nokia dot com if you want to give it a try.
- phren0logy 15y ago
  It seems to be a very active project: https://github.com/tuulos/disco/commits/master https://github.com/tuulos/disco/commits/master
konradjg 15y ago
I have to disagree completely with your example of horrible naming conventions regarding the jobtracker and tasktracker. The names are very descriptive and convey enough details to know what their purpose is. http://hadoop.apache.org/mapreduce/docs/current/mapred_tutorial.html#Job+Submission+and+Monitoring http://hadoop.apache.org/mapreduce/docs/current/mapred_tutor... "The MapReduce framework consists of a single master JobTracker and one slave TaskTracker per cluster-node. The master is responsible for scheduling the jobs' component tasks on the slaves, monitoring them and re-executing the failed tasks. The slaves execute the tasks as directed by the master." If your problems is in remembering the difference perhaps you have not spent enough time understanding the tools you are using.
- programnature 15y ago
  I think OP is saying he understands the difference in what they do, but can't keep the names straight. I felt the same way for the first few months of using hadoop. I eventually got that in real-life, a task is just a subpiece of a job. But the naming is not immediately evocative of the relationship, particularly since in computing, "job" is pretty overloaded. Far better would be "job" and "subjob".

earl 15y ago

Yeah, hadoop is a bear to get working. One of the benefits of working at a more established company like quantcast, google, or presumably facebook is there are internal infrastructure teams to smooth all that over. I pretty much get to type submitJob and it more or less just works...

technomancy 15y ago
It would probably work better in a large company. We spent a couple weeks evaluating Hadoop in our small startup and came to the conclusion that we couldn't afford to have such an infrastructure team, and I imagine most startups considering Hadoop are in a similar boat.
- harryh 15y ago
  We use hadoop at foursquare and it was set up and maintained by 1 person. It was a real bear of a job but he got it done.
  - tluyben2 15y ago
    It's definitely doable. It's just a lot of work.
jshen 15y ago
I'm at a big company that uses hadoop, and someone else maintains the infrastructure. But, the problem we have is that hadoop is a very leaky abstraction and it's very easy for someone that doesn't know implementation details to break the grid. It's relatively easy to exhaust the memory of the jobtracker, we constantly have problems with mapper starvation because someones seemingly innocent job is causing issues, etc, etc. My experience is that it doesn't just work, even when someone else maintains the infrastructure.

alecco 15y ago

It's good to have critical reviews. But perhaps it could have been better if the tone were more respectful towards a free and open source project. The approach of Hadoop is not my cup of tea, but I praise them for giving for free a working product solving such a hard problem. Playing Hadoop's side, where are the test cases, the patches or bug reports? Or even some missing documentation blurbs, like you mention.

seiji 15y ago
We've become spoiled. open source has different levels ranging from "made in basement and released to the world" (one maintainer, patches welcome) to "throw over the fence" (you can see the code, but nobody cares about patches) to "we're an open source corporation" (apache, patches welcome if you follow steps 1 to 22). Hadoop falls under apache governance which means jira, contributor license agreements, and spending a week of your life trying to get your point across. It's much easier to complain like the spoiled developers we've become.
- timr 15y ago
  It's much easier to complain like the spoiled developers we've become." By your own comments, that's not really fair. The Apache project has become a disaster area of bureaucracy, crappy code and crappier documentation. You have to spend weeks in the muck to get to the point where you can even properly understand the deficiencies of any given Apache subproject, and by that point, you're so bound up in the project that it's hard to move away. From the build system of the code to the structure of the community, not a single thing about Apache is what I would characterize as nimble or lightweight. Can you blame people for feeling helpless? It's gotten to the point where I'll look anywhere for a competing implementation of an idea before using Apache code to do something. It's just not worth the pain.
  - blasdel 15y ago
    I feel exactly the same way about Apache projects, and even now it's very rare to meet a fellow traveler. Years ago I had the same emperor has no clothes realization that the ASF was an enormous BureaucraticBullshitFactoryFactory — that it was seemingly founded just to host enterprise middleware crap while trading on the name of httpd (without really developing it of course). It took quite a while before I realized that the Apache Software Foundation was just the new IBM — and not just figuratively, if you actually looked at the active membership of the ASF, it was chock full of IBM employees contributing on company time.

forgotusername 15y ago

> This means that the Hadoop process which was using 1GB will temporarily "use" 2GB when it shells out. This is exactly not what happens, the new process copies only the parents address mappings (now marked read-only), which represents vastly less than 1gb physical memory. I think only a single 4kb page or two will ultimately be copied, representing a chunk of the calling thread's stack used to prepare the new child before finally calling execve() or similar.

nathanmarz 15y ago
As far as I understand it (I could be wrong as I've pieced this together from multiple sources), forking still reserves the same amount of memory that the parent process is using without actually copying the data (since it could very well use all that memory eventually). The wrench is that Linux has a feature called memory overcommit (which I haven't been able to decipher completely). Supposedly it causes forking to not actually reserve that much space, but by default it's in a "heuristic" mode so it may or may not take effect. These are the best resources I could find on what happens when you fork: http://developers.sun.com/solaris/articles/subprocess/subprocess.html http://developers.sun.com/solaris/articles/subprocess/subpro... http://lxr.linux.no/linux/Documentation/vm/overcommit-accounting http://lxr.linux.no/linux/Documentation/vm/overcommit-accoun...
- blocke 15y ago
  I'm not anywhere near being a Java expert but I've seen similar behavior with a non-Hadoop workload. In my case I had several virtual machines hosting Java VMs with 2+GB heaps running an app that liked to fork and run external programs for short periods of time. If the entire heapsize was say 3GB and it forked twice Linux would act like it needed 9GB. The Linux overcommit heuristic regularly got things wrong and wanted RAM that it would never actually use. This usually resulted in the JVM failing in new and interesting ways. The workaround is to allocate a crapload of swap (mainly more than heap size times a guestimate of number of concurrent forks). It will never actually USE the swap but having it there seems keep the overcommit heuristic happy. Yay for cargo cult server tuning. I've never figured out how to get the kernel to not be so pessimistic and I can't modify the Java code in question so... eh.

jamii 15y ago

I would be interested to hear why riak, disco etc aren't viable alternatives. I've seen very few good comparisons of the various options (the Mozilla data blog being the only one that comes to mind).

reinhardt 15y ago
"Risking" a downvote for a basically "me-too" reply, I would also be very interested in such a comparison or even a simple blog post of the experience (good or bad) with alternatives, especially disco.

dougb 15y ago

I've been using Hadoop in production for about 3 years on a small 48 node cluster. I've seen many issues, but none of the ones mentioned in the blog post. My general theory is that if its an important tool for your business, you need at least 1 person to be an expert on it. The alternative is to pay Cloudera a significant amount per node for support. Another possible alternative is to use http://www.MapR.com/ http://www.MapR.com/, they are in beta and claim to be api compatible with Hadoop, but they are not free.

ssn 15y ago

Shouldn't Hadoop users help to document it and contribute to it? Why isn't this happening?

code_duck 15y ago
A lot of users of Hadoop are companies with serious competition. If you took the time to figure all of this out, would you go post a howto for your competitors? No, might as well let them expend their own time and money to do what you did. While that's not exactly the open source spirit, it is how plenty of people think and also happens to be the default, easiest thing to do. It would be nice if people could share to benefit themselves and each other.

api 15y ago

I never got the Hadoop craze. It performs terribly on memory and CPU, which is odd for something that prides itself on being HPC-oriented.

tluyben2 15y ago
Alternatives? Edit: Real alternatives I mean; something OSS (free is not so important, but no lock-in), able to manage huge amounts of data, actively developed, active projects built on top.

firebones 15y ago

We've seen all of these issues, but the primary causes appear to be either problems in configuration of the cluster (which you typically don't see until you're doing serious work, or working with an overstressed cluster at scale) and problems with underlying code quality of the jobs (e.g., processes hanging due to tasks which don't terminate due to inability to handle unexpected input gracefully--infinite loops or non-terminal operations). If you're working with big data, particularly data that isn't the cleanest, you'll start to see some of these issues and others arise. Despite those issues, the most remarkable thing about Hadoop is the out-of-the-box resilience to get the work done. The strategy of no side-effects and a write-only approach (with failed tasks discarding work in progress) ensures predictable results--even if the time it takes to get those results can't be guaranteed. The documentation isn't the greatest, and it's very confusing sorting out the sedimentary nature of the APIs and configuration (knowing which APIs and config match up across various versions such as 0.15, 0.17, 0.20.2, 0.21, etc., not to mention various distributions from Cloudera, Apache and Yahoo branches), but things are starting to finally converge. You're probably better off starting with one of the later, curated releases (such as the recent Cloudera distribution) where some work has been done to cherry pick features and patches from the main branches.

nivertech 15y ago

Hadoop is so old school... Even Google moving away from Map/Reduce. Prepare for new shiny things, which will be better, faster and cheaper to operate!

tluyben2 15y ago

Memory; it's Java. Java is great, but is a memory hog. Doesn't matter so much these days (well it does, but what are you going to do... There isn't much OSS competition for Hadoop). The documentation is horrible though. I'm not sure if this is a 'new' OSS trend, but now that i'm working a lot with Rails and gems I notice that on that side of OSS it's pretty normal to produce no or completely horrible documentation; use the force read the source and that kind of hilarious tripe. Apache projects which are Hadoop related (Hadoop, Pig, Hbase) all suffer from this; you are hard pressed to find anything remotely helpful and not incredibly outdated. At least for rails you can find tons of examples (no docs though) of how to achieve things; for Hadoop/Hbase everything is outdated, not functional and requiring you to jump into tons of code to get stuff done. Again; there is not so much competition for tasks you would accomplish with Hadoop (and Hbase) on the scale it has been tested (by Yahoo/Stumbleupon and many others).