4 ms·

So, that would mean the hadoop distro market would be down to just cloudera? A single vendor market is not a market, it's a legacy product.

by Joeri 7y ago

russtrotter 7y ago
AFAIK, AWS's EMR and Glue leverage Hadoop under the covers. Does that count towards that market? and/or at least keeping Hadoop development active?
- derefr 7y ago
  Hadoop ≠ Hadoop distro. Hadoop is fine and healthy; what is in question is whether larger repackagings of it with other vendor-specific components is an interesting product any more.
- filesystem 7y ago
  Google's dataproc and Azure's HDInsight as well. The cloud players are swallowing whatever is left of the hadoop market.
- threeseed 7y ago
  EMR is their managed distro of Hadoop. DataPipeline is their managed Hadoop for ETL. Glue ETL is their managed Spark. And it all absolutely counts towards the market.
buryat 7y ago
Everybody is moving to the cloud so EMR, Dataproc, Dataflow, etc. are still going to be around Plus everyone is moving away from Hadoop and HDFS, so it's Spark, S3, GCS, Databricks, etc
badpun 7y ago
Is Microsoft Office or Microsoft Windows a legacy product?
nwsm 7y ago
Azure's HDInsight all-in-one suite uses it.
- zjaffee 7y ago
  HDInsite is built by hortonworks which was acquired by cloudera.
- opportune 7y ago
  In addition to HDInsight being cloudera, it's also basically a legacy product. Azure technical solutions people are now all pushing databricks. And personally, when designing a new project, I don't see why you would use HDInsight instead of a bespoke distributed vm architecture with one (azure storage) or two-tier storage (one local to the vm, one in Azure storage) unless you really need to use some hadoop jar you found lying around.
jamesblonde 7y ago
Actually, there is another vendor. A new one - www.logicalclocks.com - we are selling a next-generation version of Hadoop, with consistent distributed metadata for HDFS, it's built on TLS/SSL (not kerberos), and it's laser focussed on Data Science. We have customers with racks with 10s of GPUs training models with 10s of TBs of feature data. And we also built the industry's first enterprise feature store for ML on this platform. References: https://www.logicalclocks.com/feature-store/ https://www.logicalclocks.com/feature-store/ https://www.logicalclocks.com/introducing-hops-hadoop/ https://www.logicalclocks.com/introducing-hops-hadoop/

So, that would mean the hadoop distro market would be down to just cloudera? A single vendor market is not a market, it's a legacy product.

by Joeri 7y ago

So, that would mean the hadoop distro market would be down to just cloudera? A single vendor market is not a market, it's a legacy product.

russtrotter 7y ago

AFAIK, AWS's EMR and Glue leverage Hadoop under the covers. Does that count towards that market? and/or at least keeping Hadoop development active?

derefr 7y ago
Hadoop ≠ Hadoop distro. Hadoop is fine and healthy; what is in question is whether larger repackagings of it with other vendor-specific components is an interesting product any more.
filesystem 7y ago
Google's dataproc and Azure's HDInsight as well. The cloud players are swallowing whatever is left of the hadoop market.
threeseed 7y ago
EMR is their managed distro of Hadoop. DataPipeline is their managed Hadoop for ETL. Glue ETL is their managed Spark. And it all absolutely counts towards the market.

buryat 7y ago

Everybody is moving to the cloud so EMR, Dataproc, Dataflow, etc. are still going to be around Plus everyone is moving away from Hadoop and HDFS, so it's Spark, S3, GCS, Databricks, etc

badpun 7y ago

Is Microsoft Office or Microsoft Windows a legacy product?

nwsm 7y ago

Azure's HDInsight all-in-one suite uses it.

zjaffee 7y ago
HDInsite is built by hortonworks which was acquired by cloudera.
opportune 7y ago
In addition to HDInsight being cloudera, it's also basically a legacy product. Azure technical solutions people are now all pushing databricks. And personally, when designing a new project, I don't see why you would use HDInsight instead of a bespoke distributed vm architecture with one (azure storage) or two-tier storage (one local to the vm, one in Azure storage) unless you really need to use some hadoop jar you found lying around.

jamesblonde 7y ago

Actually, there is another vendor. A new one - www.logicalclocks.com - we are selling a next-generation version of Hadoop, with consistent distributed metadata for HDFS, it's built on TLS/SSL (not kerberos), and it's laser focussed on Data Science. We have customers with racks with 10s of GPUs training models with 10s of TBs of feature data. And we also built the industry's first enterprise feature store for ML on this platform. References: https://www.logicalclocks.com/feature-store/ https://www.logicalclocks.com/feature-store/ https://www.logicalclocks.com/introducing-hops-hadoop/ https://www.logicalclocks.com/introducing-hops-hadoop/