4 ms·
Do you actually work in this industry ? Because Hadoop has not been killed off. In fact it is far bigger than it has ever been and growing each year. It's just
by threeseed 7y ago
Do you actually work in this industry ?
Because Hadoop has not been killed off. In fact it is far bigger than it has ever been and growing each year. It's just that it is all transitioning to the cloud with parts like HDFS being replaced with more scalable solutions like S3/EMRFS.
But you can't say it's been killed off when on AWS you have managed Hadoop (EMR), managed Hadoop pipelines (DataPipeline) managed Spark (Glue ETL), managed Hive Metastore (Glue Catalog) etc. And similar on Azure or GCP.
- capkutay 7y ago> Because Hadoop has not been killed off. > HDFS being replaced with more scalable solutions like S3/EMRFS. Thank you for re-affirming my point. I'm not going to go into the typical HN "I'm going to nitpick your point down to the bone and argue you over the pointless details", but 2012 Hadoop is not the same thing as the tools you're describing.
- parasubvert 7y agoExcept you’re not being just a bit inaccurate, you’re flat wrong. Simply put, Hadoop is not HDFS, it is a much larger Apache ecosystem that includes Spark, Hive, MR, NiFi, etc. All of which are doing fine.
- kristjansson 7y agoI’m not sure how common that usage is . ‘Hadoop Ecosystem’ surely encompasses Spark and the rest, but I’d argue ‘Hadoop’ most commonly refers to MR, HDFS, and maybe YARN. Which is only to say I’d be surprised to hear an application described as ‘using Hadoop’ and find it using Spark on data in S3
- capkutay 7y agoSpark is NOT hadoop. You don't even need hadoop for spark anymore. Spark is more or less completely independent from the hadoop ecosystem. Just because they're compatible doesn't mean they're synonymous. You can use spark with mesos or kubernetes instead of YARN.