34 ms·

Curious where you get "Hadoop is gone" by two companies (who employ hundreds of open source contributors and committers) merging. I think it's in fact the compl

Curious where you get "Hadoop is gone" by two companies (who employ hundreds of open source contributors and committers) merging. I think it's in fact the complete opposite. I get the whole buy-in into the FUD that Cloudera ships proprietary software so they're evil, but the fact of the matter is their core system (CDH) is 100% open source. Impala, open source. Kudu, open source. Anything that is in CDH that isn't in Apache Hadoop is in a pull request. You can literally download every last source file that you trust your data with.

jamesblonde 8y ago
Hadoop is not gone yet. But the intention is there. AWS EMR is killing them. Users/Companies just pick EMR by default, leaving few crumbs left for the now one company that develops the platform. The EMR team is what, a two pizza box team? Like you said, Cloudera pay for hundreds of top developers to develop a product that is repackaged by a two-pizza team whose company earns more than Cloudera on Hadoop. That situation will not continue, that is my prediction. And Hadoop will be the crossfire. My prediction is that Cloudera will release a new data platform in time that is not "hadoop", but they will leave Apache Hadoop in maintenance mode. It's the only logical way to address the AWS "problem".
- monstrado 8y ago
  That's where we disagree. You're taking some pretty radical jumps to get to the conclusion you want. Let's assume for a second that you're right, and EMR somehow kills an entire company (who sells much more than just a Cloud offering). This has absolutely no bearing on the survival of an open source project. Yes, Cloudera employs developers who's sole job is to work on Hadoop, but there are a ton of people who work on and contribute to Hadoop outside of both Cloudera & Hortonworks (e.g. Yahoo, Facebook, Amazon, Alibaba, ..., ..., ...) It's equally outrageous to assume that the developers at these companies would somehow just completely lose all passion and willingness to carry the project forward if they lost their job. The reason a lot of people at Cloudera & Hortonworks are there is because they were contributing to the project long before they were employees. Hadoop isn't going anywhere. It may become less relevant or more relevant based on how things play out and/or new projects coming to fruition, but the project will be around for a long time. Lastly, the project "Hadoop" is nothing like it was back 8 years ago. The project continues to evolve and projects born out of it are essentially tacked onto it (e.g. Flink, Spark, Kudu, HBase, Presto/Drill/Hive/Impala).

jamesblonde 8y ago

Hadoop is not gone yet. But the intention is there. AWS EMR is killing them. Users/Companies just pick EMR by default, leaving few crumbs left for the now one company that develops the platform. The EMR team is what, a two pizza box team? Like you said, Cloudera pay for hundreds of top developers to develop a product that is repackaged by a two-pizza team whose company earns more than Cloudera on Hadoop. That situation will not continue, that is my prediction. And Hadoop will be the crossfire. My prediction is that Cloudera will release a new data platform in time that is not "hadoop", but they will leave Apache Hadoop in maintenance mode. It's the only logical way to address the AWS "problem".

monstrado 8y ago
That's where we disagree. You're taking some pretty radical jumps to get to the conclusion you want. Let's assume for a second that you're right, and EMR somehow kills an entire company (who sells much more than just a Cloud offering). This has absolutely no bearing on the survival of an open source project. Yes, Cloudera employs developers who's sole job is to work on Hadoop, but there are a ton of people who work on and contribute to Hadoop outside of both Cloudera & Hortonworks (e.g. Yahoo, Facebook, Amazon, Alibaba, ..., ..., ...) It's equally outrageous to assume that the developers at these companies would somehow just completely lose all passion and willingness to carry the project forward if they lost their job. The reason a lot of people at Cloudera & Hortonworks are there is because they were contributing to the project long before they were employees. Hadoop isn't going anywhere. It may become less relevant or more relevant based on how things play out and/or new projects coming to fruition, but the project will be around for a long time. Lastly, the project "Hadoop" is nothing like it was back 8 years ago. The project continues to evolve and projects born out of it are essentially tacked onto it (e.g. Flink, Spark, Kudu, HBase, Presto/Drill/Hive/Impala).