3 ms·

If your working set fits entirely in ram on a single machine, hadoop is the wrong tool for the job. Hadoop is more about getting obscene IO bandwidth from hund

If your working set fits entirely in ram on a single machine, hadoop is the wrong tool for the job. Hadoop is more about getting obscene IO bandwidth from hundreds of hard disk spindles going in parallel, not raw number of cores/parallel computation.

beagle3 15y ago
While that is what Hadoop is about, and it is successful at it, it is very far from being efficient; In my experience, you pay 10x compared to good use of the same hardware. "obscene IO bandwidth" is right, but at 10% of the practical maximum. This is a tradeoff that people in computing have been happily taking for years -- e.g. using a higher level language is a similar kind of tradeoff (use Python instead of C - pay x10 in performance, get x10 shorter development time). But this tradeoff only makes sense if the x10 thing you get is cheaper than the x10 thing you pay for. With hadoop, you pay x10 for hardware/computing time and more for administration (unless you have some kind of EC2 elastic mapreduce, at which point you pay x20 for hardware/computing but no administration cost), and you save some development time. Two hadoop projects I was consulting on (or rather, consulting _off_ hadoop) were paying ~$10K/month each to Amazon for a while, when <$20K in programmer time for each reduced it to working on one beefy machine that was already located in the office, with ~$100/month cost (power, cooling).
- jeffffff 15y ago
  I've found the slowdown from hadoop to be very problem dependent. I've seen anywhere from no slowdown to 1000x slower. It sounds to me like those projects should never have been using hadoop in the first place. If your problem is not IO constrained naturally, moving it to hadoop will make it IO constrained. In my experience doing that will get you a 10x-100x slowdown versus a more appropriate solution on the same hardware. If your problem is IO constrained, for example processing 10 TB of log data, hadoop will get all your disks to 100% utilization just as well as anything else will. It will leave your processors horribly underutilized, but that's just the nature of IO constrained problems. If you could get 10 TB of ram dedicated to your dataset then you could easily get a massive speedup over the hadoop solution but that's just not realistic.

beagle3 15y ago

While that is what Hadoop is about, and it is successful at it, it is very far from being efficient; In my experience, you pay 10x compared to good use of the same hardware. "obscene IO bandwidth" is right, but at 10% of the practical maximum. This is a tradeoff that people in computing have been happily taking for years -- e.g. using a higher level language is a similar kind of tradeoff (use Python instead of C - pay x10 in performance, get x10 shorter development time). But this tradeoff only makes sense if the x10 thing you get is cheaper than the x10 thing you pay for. With hadoop, you pay x10 for hardware/computing time and more for administration (unless you have some kind of EC2 elastic mapreduce, at which point you pay x20 for hardware/computing but no administration cost), and you save some development time. Two hadoop projects I was consulting on (or rather, consulting _off_ hadoop) were paying ~$10K/month each to Amazon for a while, when <$20K in programmer time for each reduced it to working on one beefy machine that was already located in the office, with ~$100/month cost (power, cooling).

jeffffff 15y ago
I've found the slowdown from hadoop to be very problem dependent. I've seen anywhere from no slowdown to 1000x slower. It sounds to me like those projects should never have been using hadoop in the first place. If your problem is not IO constrained naturally, moving it to hadoop will make it IO constrained. In my experience doing that will get you a 10x-100x slowdown versus a more appropriate solution on the same hardware. If your problem is IO constrained, for example processing 10 TB of log data, hadoop will get all your disks to 100% utilization just as well as anything else will. It will leave your processors horribly underutilized, but that's just the nature of IO constrained problems. If you could get 10 TB of ram dedicated to your dataset then you could easily get a massive speedup over the hadoop solution but that's just not realistic.