7 ms·

Hadoop “fails us”

62 points by cdvonstinkpot 10y ago

ghc 10y ago
The article is on point about how we need better data infrastructure to support data scientists and analysts. In the past I've worked to develop very scalable data infrastructure to support data science workloads on high variety sensor dat,a but it always felt like the only reason we were doing it was that nobody developed tools made for companies like ours. We can build better data infrastructure for data scientists, but in practice it's hard to sell "10x easier to use" into organizations with hadoop or half-broken bespoke infrastructure because the IT groups running the show don't really care that they're making their data scientists miserable. If change is to come, it's going to have to be from data scientists embedded within business units demanding better tools, because hadoop works just fine if you don't care how hard it is for your users to access their data.
- sixdimensional 10y ago
  To be clear, many of the comments are from Bob Muglia, CEO of Snowflake Computing which is a cloud based columnar data warehouse solution. Presumably, much easier to implement than Hadoop, so you have to take those comments with a grain of salt. Also, those entities who the article claims have a grasp on Hadoop are some of the main people who helped develop it - lest we forget that the early work on Hadoop came out of Yahoo. "Web-scale" companies by their very nature have to own this technology, because almost nobody else on earth did what they have done on that scale (except maybe governments and scientists). However, I agree the article is right on. As someone who has done a lot of work with different Hadoop distributions (Cloudera, Hortonworks and MapR to name a few), I can say that compared to the performance and ease of use of more traditional data platforms, Hadoop is a different and more complex creature. It has evolved quickly in some regards and slowly in others. It's way easier to get a Cloudera system up and running today (for example) than it was in the past. "Hadoop" by itself is a misnomer, it's not like it's one technology. HDFS, MapReduce (and MR2), Hive, HBase, Spark, Pig, Flume, etc. etc. - it's an ecosystem. It's also a mashup of many things depending on what distro you're talking about - too many things - a "data hub", an "operational data store", a "pre-data warehouse staging", an ETL platform, a "search" platform, a data governance platform, a data archiving solution, a real-time data query / streaming data processing etc. etc. etc. I think Hadoop did one important thing to our overall awareness, and that was to think of a "data platform", "data processing", "data architecture" and "data infrastructure" as a major, and "big" strategic part of an organization. I think the rise of "big data" thinking along with the rise of the "chief data officer", "data scientists" and "data engineers" and the like all goes together - a shift in thinking we made where we realized we could do more with more data (for better or worse). There are many new and old technologies that help people implement the data infrastructure in their organizations - not just Hadoop. Hadoop just happened to be a major player at the epicenter at this (and open source as well), and if anything, has spurred technological advancement in databases, NoSQL and NewSQL and OldSQL alike, ETL, data mining, AI/machine learning, etc. So it has played its role, whether it specifically stays relevant or not as we perceive it.
samspenc 10y ago
I worked on Hadoop and HBase extensively from 2011 - 2013, working on engines processing 30 billion raw data points a month and storing a subset of those, and then we migrated to other Big Data technologies. Just wanted to add my thoughts here. Hadoop (and its general ecosystem, which includes HBase), is a fairly good idea. Its core ideas - map/reduce on Hadoop, and a large distributed key/value store for HBase - are actually pretty solid. And for many years, there were simply no alternatives to Hadoop. Think of the years 2008 to 2012/13. If you had to process terabytes or petabytes of data, what were your solutions? No wonder Yahoo and Facebook (and others) put in so much effort into their Hadoop solutions. But, IMHO, there were several issues with Hadoop and their ilk. 1. The core infrastructure wasn't stable enough. Hadoop / HBase were supposed to be distributed systems, and they worked well, but small failures could bring down your entire cluster. Given that Hadoop and HBase were being used in mission-critical systems in the cloud, and given the amount of DevOps or sys-admin work that went into maintaining these, I'm not surprised people eventually migrated to distributed systems that were easier to maintain and run. 2. There are now plenty of "hosted on the cloud" solutions such as Amazon DynamoDB or similar cloud solutions. When your company depends on 99.99% or similar SLAs, you don't want to have downtime on your database systems and spend time debugging complicated core dumps on your Hadoop or HBase clusters when you can just store it "on the cloud" and be done with it. Sure, there's a higher price point, but those are the trade-offs you live with. 3. If you want to be in-house, there are plenty of alternatives out there as well today. Apache Spark for processing, Kafka for a messaging bus / streaming data, ElasticSearch for large scale storage, with multiple indices. Many of them are much more robust than Apache Hadoop / HBase, and I'm not surprised they've gotten more traction recently. Ultimately, I think Hadoop / HBase are just showing their age. They were fantastic for the first wave of Big Data technologies, and you had little alternative if you were building large-scale systems circa 2008 to 2013, but now, you just have a plethora of choices from various vendors.
- toast0 10y ago
  Another factor is that memory density has increased a lot since 2008; you can easily get 1TB of ram in a dual socket system, so for memory space limited jobs, you can run on fewer machines, and maybe just one, if your needs haven't grown as fast as technology has.
darose 10y ago
This really strikes me as more of a marketing piece by Snowflake than a well-researched piece of reporting. The article mostly just quotes one person - Bob Muglia - who is, as they say on Wall Street, "talking his book" - i.e. giving an opinion that is not coincidentally in line with his own financial interests. Sure, Hadoop is getting old, and is quickly becoming replaced by spark. But loads of organizations have used, and continue to use hadoop /spark successfully. And the part about Kafka replacing Hadoop /Spark is just silly. They're completely different technologies, used for very different purposes, and many organizations use both side by side.
- agibsonccc 10y ago
  Disclaimer: I run a technology vendor partially invested in the success of the hadoop ecosystem. You are making a very common mistake of coupling the file system and the map reduce implementation and the scheduler . You are right about this post and kafka though. Let me expand on this point a bit more. Hadoop isn't what it was in 2004. It's now a complex beast with several decoupled components which now make it very hard to identify what "hadoop" is for people outside the space. The hadoop ecosystem is actually very healthy if you actually look at all the streaming platforms built on top of it (kafka, apex, spark, flink,tez,..) There are also databases such as hbase,cassandra and more recently kudu, specialized for different workloads. Don't even get me started on all the sql implementations (again with their own trade offs) such as impala and hive. If we step back for a second here and focus on just the compute part: Yes map reduce is for the most part dead. This is supplanted by the streaming and batch platforms such as flink and spark. The scheduler part (YARN) is competing with mesos largely in part thanks to spark and flink being able to leverage both with mesos being way more flexible. (Most hadoop distros only use YARN though). Then we also have the distributed consensus part in zookeeper. Etcd is up and coming in this piece but your hadoop cluster uses zookeeper (both mesos and kafka rely on it for example)
- MilnerRoute 10y ago
  The article also quotes Bobby Johnson, who helped run Facebook's Hadoop cluster, as well as the creator of Kafka (who ran Hadoop clusters at LinkedIn). For what it's worth, all three of them seemed pretty down on Hadoop.
c3534l 10y ago
> Hadoop is great if you’re a data scientist who knows how to code in MapReduce or Pig I guess that kind of makes sense. What programmers like might not necessarily be a good basis for the long-term. I do have to say, though, that part of the failure of Hadoop is it generated a lot of hype and so better tools and alternatives developed to meet that need. So when you're saying a better alternative is Spark or Kafka, I feel like it's almost as if you're saying "Oracle is a failure that never materialized the promised benefits. Instead we should be using PostGres and MySQL." But I also know that with a lot of big data hype, businesses were wanting to do Hadoop and NoSQL and all this stuff because it was the cool new thing, not that they actually needed it. I've heard data scientists make the joke that every business thinks they need these tools because they're having difficulty running their business out of a spreadsheet. I think it's important to remember that for most businesses, Spark, Hive, whatever; those aren't the right tools, either. SQL is still what most companies need. Businesses want machine learning, but usually what they need is boring old statistics. In an industry that always wants to be ahead of the curve, we tend to forget that it's not always the right thing to have the newest toys. Sometime companies do well utilizing the latest and greatest, but sometimes they just use what already exists wisely. I suspect that for many of those companies that used Hadoop and felt it didn't work them, the problem wasn't that they needed Spark instead, it was they were trying to solve problems that didn't exist. Man, I'm too young to sound this old. But, eh, yeah, we need to respect our elders' technology first, and consider the newest stuff only when we have a definable need for it.
- xapata 10y ago
  Nit: I believe if you wanted to abbreviate PostgreSQL, "Postgres" would be more appropriate than "PostGres".
- xapata 10y ago
  You were on target with the comment that businesses switched to Hadoop because they had trouble with spreadsheets. I've been a culprit in helping folks make that transition. I warned them, but they insisted. Who am I to refuse? They had heard Hadoop didn't require conforming to a schema and would allow data-driven insights. What they failed to verify in advance was whether those insights would increase profit. I don't see how any other database-ish technology would change the situation. It ain't the tech, or the analysis, it's the business processes.
ktamura 10y ago
Hadoop didn't fail us. We failed Hadoop. The decline of Hadoop as a software category is Software Product Marketing 101: it did not identify pervasive killer use cases critical to running . Yes, it's true that Hadoop was a revolutionary way to store and process massive datasets on commodity hardware, but what's the use case for that? If you are Visa/AMEX (fraud detection), Facebook/Google (various ML-based data products) and a few other types of companies with obvious applications of massive data processing, yes, Hadoop has been great. But here's the thing: beyond a few such corner cases, it never found a use case that enterprise data warehouse couldn't handle. Then came Redshift, then BigQuery, and now Snowflake (as a BigQuery on AWS, really). While there are some key technical differences between Redshift and BigQuery/Snowflake, they are all _much_ cheaper than the previous generation of data warehouses (Vertica, Netezza, Greenplum, etc.) The lower price meant greater access, and developers who previously couldn't imagine using data warehouses could finally spin one up with a credit card swipe. Hadoop, too, took a lot of collateral damage because many developers realized that they didn't need much of Hadoop beyond SQL-on-Hadoop. Redshift was a beautiful feat of product strategy and marketing: They just took what used to cost a lot and offered it for much less in an environment where developers already had a lot of data (AWS). This was much simpler to execute than what Hadoop had to do: introduce new technology, identify use cases, and finally compete with incumbent solutions. We failed Hadoop (as you can see from www.cloudera.com, even Cloudera, the Hadoop company, hardly mentions Hadoop on its top page). Not the other way around.
- bsg75 10y ago
  > because many developers realized that they didn't need much of Hadoop beyond SQL-on-Hadoop I wonder how many of those are just SQL-on-HDFS (Drill, Spark, Presto) ?
- vgt 10y ago
  Minor correction - unless you consider Paraccel a part of Redshift (you probably should), BigQuery GA precedes Redshift release by around a year, and Dremel at least 6.
shmerl 10y ago
What about competition like HPCC[1]? [1]: https://en.wikipedia.org/wiki/HPCC https://en.wikipedia.org/wiki/HPCC
75dvtwin 10y ago
My observation that traditional database technologies are transforming themselves into 'hybrids' (as far document-oriented data types go). Example is Postgres. Now has with JSONB as document-oriented field type. Now has Postgres XL, as horizontally scalable ACID database. Will have, by approximately august', abiltity to maintain views in memory (aka lambda.architecture speed layer) via PipelineDB extension, for fast streaming analytics. It seems that a combination of Postgres (with extensions) + Kafka +Redis -- is a strong stack for lambda architecture and, as initial data hub component of the overall puzzle While Spark (or even Python+Dask) can be viewed as distributed data analytics platforms that replaces 'non-ui centric' BI. I think ui-centric BI (eg adhoc reports/visualizatiosn) are going to be still dominated within enterprises by Tableu/QlickView type of solutions. For traditional BI-oriented data marts (that organized downstream from the datahub) -- probably traditional column oriented databases, and the new open source ones make sense. To me ,the promise of hadoop being a silver bullet for 'all the big data needs' -- was always nothing more than unsubstantiated hype. So it definitely failed the ones who believed, the hype -- but did not fail others who did not buy into it.
frik 10y ago
Kafka relies on ZooKeeper of the Hadoop eco-system. ZooKeeper is not so great.
nazilla12 10y ago
Disclaimer : I work for a corporate IT consulting giant. The trend I'm seeing the big Data sphere in my company is a by-and-large move away from technologies that implement MapReduce and and complicated batch processing with HDFS as a data store. More and more customers want insight as soon as they get/produce their data so we've seen a particularly large increase in interest in technologies such as Kafka/ Samza and spark/ pySpark. I see a trend in Kafka and but I think the community needs to jump behind it too, keep it as a pipeline tool and not a querying engine. I dont see Hadoop-based solutions going away any time soon though.
- slv77 10y ago
  How do these customers plan on doing historical analysis of raw data without Hadoop?
  - agibsonccc 10y ago
    Spark is still "hdfs". These guys just use spark sql instead of map reduce jobs. SPark is heavily used for batch workloads.

62 points by cdvonstinkpot 10y ago

ghc 10y ago

The article is on point about how we need better data infrastructure to support data scientists and analysts. In the past I've worked to develop very scalable data infrastructure to support data science workloads on high variety sensor dat,a but it always felt like the only reason we were doing it was that nobody developed tools made for companies like ours. We can build better data infrastructure for data scientists, but in practice it's hard to sell "10x easier to use" into organizations with hadoop or half-broken bespoke infrastructure because the IT groups running the show don't really care that they're making their data scientists miserable. If change is to come, it's going to have to be from data scientists embedded within business units demanding better tools, because hadoop works just fine if you don't care how hard it is for your users to access their data.

sixdimensional 10y ago
To be clear, many of the comments are from Bob Muglia, CEO of Snowflake Computing which is a cloud based columnar data warehouse solution. Presumably, much easier to implement than Hadoop, so you have to take those comments with a grain of salt. Also, those entities who the article claims have a grasp on Hadoop are some of the main people who helped develop it - lest we forget that the early work on Hadoop came out of Yahoo. "Web-scale" companies by their very nature have to own this technology, because almost nobody else on earth did what they have done on that scale (except maybe governments and scientists). However, I agree the article is right on. As someone who has done a lot of work with different Hadoop distributions (Cloudera, Hortonworks and MapR to name a few), I can say that compared to the performance and ease of use of more traditional data platforms, Hadoop is a different and more complex creature. It has evolved quickly in some regards and slowly in others. It's way easier to get a Cloudera system up and running today (for example) than it was in the past. "Hadoop" by itself is a misnomer, it's not like it's one technology. HDFS, MapReduce (and MR2), Hive, HBase, Spark, Pig, Flume, etc. etc. - it's an ecosystem. It's also a mashup of many things depending on what distro you're talking about - too many things - a "data hub", an "operational data store", a "pre-data warehouse staging", an ETL platform, a "search" platform, a data governance platform, a data archiving solution, a real-time data query / streaming data processing etc. etc. etc. I think Hadoop did one important thing to our overall awareness, and that was to think of a "data platform", "data processing", "data architecture" and "data infrastructure" as a major, and "big" strategic part of an organization. I think the rise of "big data" thinking along with the rise of the "chief data officer", "data scientists" and "data engineers" and the like all goes together - a shift in thinking we made where we realized we could do more with more data (for better or worse). There are many new and old technologies that help people implement the data infrastructure in their organizations - not just Hadoop. Hadoop just happened to be a major player at the epicenter at this (and open source as well), and if anything, has spurred technological advancement in databases, NoSQL and NewSQL and OldSQL alike, ETL, data mining, AI/machine learning, etc. So it has played its role, whether it specifically stays relevant or not as we perceive it.

samspenc 10y ago

I worked on Hadoop and HBase extensively from 2011 - 2013, working on engines processing 30 billion raw data points a month and storing a subset of those, and then we migrated to other Big Data technologies. Just wanted to add my thoughts here. Hadoop (and its general ecosystem, which includes HBase), is a fairly good idea. Its core ideas - map/reduce on Hadoop, and a large distributed key/value store for HBase - are actually pretty solid. And for many years, there were simply no alternatives to Hadoop. Think of the years 2008 to 2012/13. If you had to process terabytes or petabytes of data, what were your solutions? No wonder Yahoo and Facebook (and others) put in so much effort into their Hadoop solutions. But, IMHO, there were several issues with Hadoop and their ilk. 1. The core infrastructure wasn't stable enough. Hadoop / HBase were supposed to be distributed systems, and they worked well, but small failures could bring down your entire cluster. Given that Hadoop and HBase were being used in mission-critical systems in the cloud, and given the amount of DevOps or sys-admin work that went into maintaining these, I'm not surprised people eventually migrated to distributed systems that were easier to maintain and run. 2. There are now plenty of "hosted on the cloud" solutions such as Amazon DynamoDB or similar cloud solutions. When your company depends on 99.99% or similar SLAs, you don't want to have downtime on your database systems and spend time debugging complicated core dumps on your Hadoop or HBase clusters when you can just store it "on the cloud" and be done with it. Sure, there's a higher price point, but those are the trade-offs you live with. 3. If you want to be in-house, there are plenty of alternatives out there as well today. Apache Spark for processing, Kafka for a messaging bus / streaming data, ElasticSearch for large scale storage, with multiple indices. Many of them are much more robust than Apache Hadoop / HBase, and I'm not surprised they've gotten more traction recently. Ultimately, I think Hadoop / HBase are just showing their age. They were fantastic for the first wave of Big Data technologies, and you had little alternative if you were building large-scale systems circa 2008 to 2013, but now, you just have a plethora of choices from various vendors.

toast0 10y ago
Another factor is that memory density has increased a lot since 2008; you can easily get 1TB of ram in a dual socket system, so for memory space limited jobs, you can run on fewer machines, and maybe just one, if your needs haven't grown as fast as technology has.

darose 10y ago

This really strikes me as more of a marketing piece by Snowflake than a well-researched piece of reporting. The article mostly just quotes one person - Bob Muglia - who is, as they say on Wall Street, "talking his book" - i.e. giving an opinion that is not coincidentally in line with his own financial interests. Sure, Hadoop is getting old, and is quickly becoming replaced by spark. But loads of organizations have used, and continue to use hadoop /spark successfully. And the part about Kafka replacing Hadoop /Spark is just silly. They're completely different technologies, used for very different purposes, and many organizations use both side by side.

agibsonccc 10y ago
Disclaimer: I run a technology vendor partially invested in the success of the hadoop ecosystem. You are making a very common mistake of coupling the file system and the map reduce implementation and the scheduler . You are right about this post and kafka though. Let me expand on this point a bit more. Hadoop isn't what it was in 2004. It's now a complex beast with several decoupled components which now make it very hard to identify what "hadoop" is for people outside the space. The hadoop ecosystem is actually very healthy if you actually look at all the streaming platforms built on top of it (kafka, apex, spark, flink,tez,..) There are also databases such as hbase,cassandra and more recently kudu, specialized for different workloads. Don't even get me started on all the sql implementations (again with their own trade offs) such as impala and hive. If we step back for a second here and focus on just the compute part: Yes map reduce is for the most part dead. This is supplanted by the streaming and batch platforms such as flink and spark. The scheduler part (YARN) is competing with mesos largely in part thanks to spark and flink being able to leverage both with mesos being way more flexible. (Most hadoop distros only use YARN though). Then we also have the distributed consensus part in zookeeper. Etcd is up and coming in this piece but your hadoop cluster uses zookeeper (both mesos and kafka rely on it for example)
MilnerRoute 10y ago
The article also quotes Bobby Johnson, who helped run Facebook's Hadoop cluster, as well as the creator of Kafka (who ran Hadoop clusters at LinkedIn). For what it's worth, all three of them seemed pretty down on Hadoop.

c3534l 10y ago

> Hadoop is great if you’re a data scientist who knows how to code in MapReduce or Pig I guess that kind of makes sense. What programmers like might not necessarily be a good basis for the long-term. I do have to say, though, that part of the failure of Hadoop is it generated a lot of hype and so better tools and alternatives developed to meet that need. So when you're saying a better alternative is Spark or Kafka, I feel like it's almost as if you're saying "Oracle is a failure that never materialized the promised benefits. Instead we should be using PostGres and MySQL." But I also know that with a lot of big data hype, businesses were wanting to do Hadoop and NoSQL and all this stuff because it was the cool new thing, not that they actually needed it. I've heard data scientists make the joke that every business thinks they need these tools because they're having difficulty running their business out of a spreadsheet. I think it's important to remember that for most businesses, Spark, Hive, whatever; those aren't the right tools, either. SQL is still what most companies need. Businesses want machine learning, but usually what they need is boring old statistics. In an industry that always wants to be ahead of the curve, we tend to forget that it's not always the right thing to have the newest toys. Sometime companies do well utilizing the latest and greatest, but sometimes they just use what already exists wisely. I suspect that for many of those companies that used Hadoop and felt it didn't work them, the problem wasn't that they needed Spark instead, it was they were trying to solve problems that didn't exist. Man, I'm too young to sound this old. But, eh, yeah, we need to respect our elders' technology first, and consider the newest stuff only when we have a definable need for it.

xapata 10y ago
Nit: I believe if you wanted to abbreviate PostgreSQL, "Postgres" would be more appropriate than "PostGres".
xapata 10y ago
You were on target with the comment that businesses switched to Hadoop because they had trouble with spreadsheets. I've been a culprit in helping folks make that transition. I warned them, but they insisted. Who am I to refuse? They had heard Hadoop didn't require conforming to a schema and would allow data-driven insights. What they failed to verify in advance was whether those insights would increase profit. I don't see how any other database-ish technology would change the situation. It ain't the tech, or the analysis, it's the business processes.

ktamura 10y ago

Hadoop didn't fail us. We failed Hadoop. The decline of Hadoop as a software category is Software Product Marketing 101: it did not identify pervasive killer use cases critical to running . Yes, it's true that Hadoop was a revolutionary way to store and process massive datasets on commodity hardware, but what's the use case for that? If you are Visa/AMEX (fraud detection), Facebook/Google (various ML-based data products) and a few other types of companies with obvious applications of massive data processing, yes, Hadoop has been great. But here's the thing: beyond a few such corner cases, it never found a use case that enterprise data warehouse couldn't handle. Then came Redshift, then BigQuery, and now Snowflake (as a BigQuery on AWS, really). While there are some key technical differences between Redshift and BigQuery/Snowflake, they are all _much_ cheaper than the previous generation of data warehouses (Vertica, Netezza, Greenplum, etc.) The lower price meant greater access, and developers who previously couldn't imagine using data warehouses could finally spin one up with a credit card swipe. Hadoop, too, took a lot of collateral damage because many developers realized that they didn't need much of Hadoop beyond SQL-on-Hadoop. Redshift was a beautiful feat of product strategy and marketing: They just took what used to cost a lot and offered it for much less in an environment where developers already had a lot of data (AWS). This was much simpler to execute than what Hadoop had to do: introduce new technology, identify use cases, and finally compete with incumbent solutions. We failed Hadoop (as you can see from www.cloudera.com, even Cloudera, the Hadoop company, hardly mentions Hadoop on its top page). Not the other way around.

bsg75 10y ago
> because many developers realized that they didn't need much of Hadoop beyond SQL-on-Hadoop I wonder how many of those are just SQL-on-HDFS (Drill, Spark, Presto) ?
vgt 10y ago
Minor correction - unless you consider Paraccel a part of Redshift (you probably should), BigQuery GA precedes Redshift release by around a year, and Dremel at least 6.

shmerl 10y ago

What about competition like HPCC[1]? [1]: https://en.wikipedia.org/wiki/HPCC https://en.wikipedia.org/wiki/HPCC

75dvtwin 10y ago

My observation that traditional database technologies are transforming themselves into 'hybrids' (as far document-oriented data types go). Example is Postgres. Now has with JSONB as document-oriented field type. Now has Postgres XL, as horizontally scalable ACID database. Will have, by approximately august', abiltity to maintain views in memory (aka lambda.architecture speed layer) via PipelineDB extension, for fast streaming analytics. It seems that a combination of Postgres (with extensions) + Kafka +Redis -- is a strong stack for lambda architecture and, as initial data hub component of the overall puzzle While Spark (or even Python+Dask) can be viewed as distributed data analytics platforms that replaces 'non-ui centric' BI. I think ui-centric BI (eg adhoc reports/visualizatiosn) are going to be still dominated within enterprises by Tableu/QlickView type of solutions. For traditional BI-oriented data marts (that organized downstream from the datahub) -- probably traditional column oriented databases, and the new open source ones make sense. To me ,the promise of hadoop being a silver bullet for 'all the big data needs' -- was always nothing more than unsubstantiated hype. So it definitely failed the ones who believed, the hype -- but did not fail others who did not buy into it.

frik 10y ago

Kafka relies on ZooKeeper of the Hadoop eco-system. ZooKeeper is not so great.

nazilla12 10y ago

Disclaimer : I work for a corporate IT consulting giant. The trend I'm seeing the big Data sphere in my company is a by-and-large move away from technologies that implement MapReduce and and complicated batch processing with HDFS as a data store. More and more customers want insight as soon as they get/produce their data so we've seen a particularly large increase in interest in technologies such as Kafka/ Samza and spark/ pySpark. I see a trend in Kafka and but I think the community needs to jump behind it too, keep it as a pipeline tool and not a querying engine. I dont see Hadoop-based solutions going away any time soon though.

slv77 10y ago
How do these customers plan on doing historical analysis of raw data without Hadoop?
- agibsonccc 10y ago
  Spark is still "hdfs". These guys just use spark sql instead of map reduce jobs. SPark is heavily used for batch workloads.