4 ms·

I agree with the general thrust of this article. But hadoop isn't just for scaling up the absolute size of the data set. It is also useful for scaling up the

I agree with the general thrust of this article. But hadoop isn't just for scaling up the absolute size of the data set. It is also useful for scaling up the absolute amount of CPU power you can throw at a problem. If I have 1 GB data set, but the computations that I need to do on that data set are complex enough that it would take a single machine a long time to do them, then hadoop is still useful. I gain tremendously by being able to fire up 100 extra large EC2 servers and run my computation much more quickly than I could with SQL or Python on a single machine. Now some might counter this point with the observation others have made here that using hadoop imposes a ~10x slowdown. But even then, my 100 EC2 servers will get the job done 10x faster. Running a job in 1 hour with hadoop is MUCH better than running the same job in 10 hours without it, especially when you're doing data analysis and you need to iterate rapidly. So there is a point where using hadoop is not productive. But that limit is not 5 TB and depends on a lot more variables. Over simplification makes for catchy blog posts, but is rarely the way to make good engineering decisions.

pjbringer 13y ago
In that case, your IO problem is easy, because it's small with regard to CPU time. You can get away with putting the whole data in sql databases, and/or making multiple copies of your data. Then you can use as many workers as you want, with usually simple partitioning logic.
- lgieron 13y ago
  Originally I did just that, but ultimately decided to move to Hadoop. When combined with Amazon EMR, launching arbitrarily large cluster is just a few clicks. You can then monitor progress, have robust cluser-wide error handling, and your data gets nicely merged into output files in S3 (not so easy with the home-baked solution).
  - vosper 13y ago
    We've had a lot of success with EMR as well - we have an hourly Pig job that produces data for our analytics database. It's not a particularly complex script, but our traffic volume is unpredictable so it's reassuring to know that we can add resources to a slow job and have it finish faster. The downside of EMR is that it can be fairly expensive once you start needing the beefy machines. We're lucky that we can afford to have our analytics delayed an hour or two and can thus run on Spot instances (except for the Master node). When we move to a streaming architecture I'm not sure EMR will still be competitive, since we won't be able to have those machines go away on us. Edit: clarity.
- mightybyte 13y ago
  If you only look at CPU time, then yes, maybe you could do that. But there are many more factors at play.
jcampbell1 13y ago
I can understand this use case. e.g. you have some 100 parameter financial instrument that can only be valued with monte-carlo methods. You could have 1 GB of data, but many core years of computation. In this case, is Hadoop actually a good tool? When you have such a small amount of data, it is unlikely you will ever be faced with a "reduce" challenge, and thus the problem is better solved with a simple queue rather than a hadoop cluster.
acdha 13y ago
This assumes that it's easier to manage 100 servers using Hadoop than, say, using the same 100 EC2 servers with a normal grid scheduler, task queue, or even something like GNU parallel. It's orders of magnitude easier to start 100 servers and reuse basic Unix skills than it is to setup and manage Hadoop on the same infrastructure.
- dekhn 13y ago
  does your grid scheduler checkpoint intermediate work automatically, so that node failures have small impact?
  - acdha 13y ago
    Yes, although you have control over that kind of thing: this is a problem HPC groups have been tackling for decades so there's a ton of prior art as well as options for whatever makes sense for your application (e.g. if you're doing some sort of shared memory simulation you have to restart every instance, not just the one which failed).
lars512 13y ago
We use Hadoop/Hive/EMR for a sub-TB dataset for exactly this reason. The flip side is that I've found the ecosystem to be pretty flakey (specifically Hive). Debugging code you wrote yourself is pretty different to working your way through failures on a distributed system.

pjbringer 13y ago

In that case, your IO problem is easy, because it's small with regard to CPU time. You can get away with putting the whole data in sql databases, and/or making multiple copies of your data. Then you can use as many workers as you want, with usually simple partitioning logic.

lgieron 13y ago
Originally I did just that, but ultimately decided to move to Hadoop. When combined with Amazon EMR, launching arbitrarily large cluster is just a few clicks. You can then monitor progress, have robust cluser-wide error handling, and your data gets nicely merged into output files in S3 (not so easy with the home-baked solution).
- vosper 13y ago
  We've had a lot of success with EMR as well - we have an hourly Pig job that produces data for our analytics database. It's not a particularly complex script, but our traffic volume is unpredictable so it's reassuring to know that we can add resources to a slow job and have it finish faster. The downside of EMR is that it can be fairly expensive once you start needing the beefy machines. We're lucky that we can afford to have our analytics delayed an hour or two and can thus run on Spot instances (except for the Master node). When we move to a streaming architecture I'm not sure EMR will still be competitive, since we won't be able to have those machines go away on us. Edit: clarity.
mightybyte 13y ago
If you only look at CPU time, then yes, maybe you could do that. But there are many more factors at play.

jcampbell1 13y ago

I can understand this use case. e.g. you have some 100 parameter financial instrument that can only be valued with monte-carlo methods. You could have 1 GB of data, but many core years of computation. In this case, is Hadoop actually a good tool? When you have such a small amount of data, it is unlikely you will ever be faced with a "reduce" challenge, and thus the problem is better solved with a simple queue rather than a hadoop cluster.

acdha 13y ago

This assumes that it's easier to manage 100 servers using Hadoop than, say, using the same 100 EC2 servers with a normal grid scheduler, task queue, or even something like GNU parallel. It's orders of magnitude easier to start 100 servers and reuse basic Unix skills than it is to setup and manage Hadoop on the same infrastructure.

dekhn 13y ago
does your grid scheduler checkpoint intermediate work automatically, so that node failures have small impact?
- acdha 13y ago
  Yes, although you have control over that kind of thing: this is a problem HPC groups have been tackling for decades so there's a ton of prior art as well as options for whatever makes sense for your application (e.g. if you're doing some sort of shared memory simulation you have to restart every instance, not just the one which failed).

lars512 13y ago

We use Hadoop/Hive/EMR for a sub-TB dataset for exactly this reason. The flip side is that I've found the ecosystem to be pretty flakey (specifically Hive). Debugging code you wrote yourself is pretty different to working your way through failures on a distributed system.