I'm biased in that I've been doing nothing but Hadoop consultancy for almost 10 years now. I have done maybe a hundred to two consulting hundred projects based

I'm biased in that I've been doing nothing but Hadoop consultancy for almost 10 years now. I have done maybe a hundred to two consulting hundred projects based on Hadoop (as you can imagine it's mostly short-term and troubleshooting stuff) and am a ASF committer myself. So take this with a grain of salt. I've seen the kinds of projects the article and lots of people in here refer to and I agree with lots of what's been said. I also agree that Hadoop (the whole ecosystem) probably not has a single "best of category" tool that ticks all the checkboxes but IMO that's totally missing the point. What it gives you is: - (Mostly) Open source stack (compare that to your Informatica/Talend/IBM stack that big companies have and not to your Python jobs they don't), lots of the code is crap (see my Apache JIRA history) but it's there and you can attach a debugger to your running cluster, that's fantastic and very hard to do with e.g. Python (at least I'm not able to find good ways of doing it) - Proven technology, it works most of the times and if not there's tons of SO answers, mailing lists, books. - All the boring enterprise stuff: Encryption at rest and on disk, strong authentication, strong authorization, auditing, high availability, failover etc. - Integration into Monitoring & alerting tools, distributions figure out the important stats and thresholds for you - Mostly easy to operate these days even at large scale thanks to Cloudera Manager et al. - no need to manually run/install stuff - "Coolness factor" - Costs way less than established tools with e.g. external storage filers or similar specialized hardware - It's not trivial but also not super hard to find people with Hadoop/Spark experience - Existing purchasing agreements with Hardware vendors or Cloud providers - Business/C-level gets and supports it Edit: Three more I thought of: - You can buy licenses and support for the tools. Both of which have been discussed multiple times here on HN, a donation doesn't work that well for companies - You can buy indemnification from license problems from at least Cloudera but I think Hortonworks as well - If the same stack has been used elsewhere in the company already chances are there are processes in place, someone has already vetted it, done the open source checks etc. See what I don't mention are things like performance or absolute money amounts or even the amount of data (as the whole ecosystem is much more than "big data" now) etc. as we see surprisingly little projects that care about specific SLAs or features. It's all relative. That said: At least 50% of the projects would still be better served by a simpler solution but that still leaves a whole big chunk of projects where Hadoop makes sense for other than technical or monetary reasons. But I very much disagree with this part of the post: > The only benefit to using Hadoop is scaling.