> Hadoop is great if you’re a data scientist who knows how to code in MapReduce or Pig I guess that kind of makes sense. What programmers like might not necess

> Hadoop is great if you’re a data scientist who knows how to code in MapReduce or Pig I guess that kind of makes sense. What programmers like might not necessarily be a good basis for the long-term. I do have to say, though, that part of the failure of Hadoop is it generated a lot of hype and so better tools and alternatives developed to meet that need. So when you're saying a better alternative is Spark or Kafka, I feel like it's almost as if you're saying "Oracle is a failure that never materialized the promised benefits. Instead we should be using PostGres and MySQL." But I also know that with a lot of big data hype, businesses were wanting to do Hadoop and NoSQL and all this stuff because it was the cool new thing, not that they actually needed it. I've heard data scientists make the joke that every business thinks they need these tools because they're having difficulty running their business out of a spreadsheet. I think it's important to remember that for most businesses, Spark, Hive, whatever; those aren't the right tools, either. SQL is still what most companies need. Businesses want machine learning, but usually what they need is boring old statistics. In an industry that always wants to be ahead of the curve, we tend to forget that it's not always the right thing to have the newest toys. Sometime companies do well utilizing the latest and greatest, but sometimes they just use what already exists wisely. I suspect that for many of those companies that used Hadoop and felt it didn't work them, the problem wasn't that they needed Spark instead, it was they were trying to solve problems that didn't exist. Man, I'm too young to sound this old. But, eh, yeah, we need to respect our elders' technology first, and consider the newest stuff only when we have a definable need for it.