4 ms·

Yeah, hadoop is a bear to get working. One of the benefits of working at a more established company like quantcast, google, or presumably facebook is there are

• technomancy 16y ago
  It would probably work better in a large company. We spent a couple weeks evaluating Hadoop in our small startup and came to the conclusion that we couldn't afford to have such an infrastructure team, and I imagine most startups considering Hadoop are in a similar boat.
  • harryh 16y ago
    We use hadoop at foursquare and it was set up and maintained by 1 person. It was a real bear of a job but he got it done.
    • tluyben2 16y ago
      It's definitely doable. It's just a lot of work.
• jshen 16y ago
  I'm at a big company that uses hadoop, and someone else maintains the infrastructure. But, the problem we have is that hadoop is a very leaky abstraction and it's very easy for someone that doesn't know implementation details to break the grid. It's relatively easy to exhaust the memory of the jobtracker, we constantly have problems with mapper starvation because someones seemingly innocent job is causing issues, etc, etc. My experience is that it doesn't just work, even when someone else maintains the infrastructure.