4 ms·

The article is on point about how we need better data infrastructure to support data scientists and analysts. In the past I've worked to develop very scalable d

by ghc 10y ago

The article is on point about how we need better data infrastructure to support data scientists and analysts. In the past I've worked to develop very scalable data infrastructure to support data science workloads on high variety sensor dat,a but it always felt like the only reason we were doing it was that nobody developed tools made for companies like ours. We can build better data infrastructure for data scientists, but in practice it's hard to sell "10x easier to use" into organizations with hadoop or half-broken bespoke infrastructure because the IT groups running the show don't really care that they're making their data scientists miserable. If change is to come, it's going to have to be from data scientists embedded within business units demanding better tools, because hadoop works just fine if you don't care how hard it is for your users to access their data.

sixdimensional 10y ago
To be clear, many of the comments are from Bob Muglia, CEO of Snowflake Computing which is a cloud based columnar data warehouse solution. Presumably, much easier to implement than Hadoop, so you have to take those comments with a grain of salt. Also, those entities who the article claims have a grasp on Hadoop are some of the main people who helped develop it - lest we forget that the early work on Hadoop came out of Yahoo. "Web-scale" companies by their very nature have to own this technology, because almost nobody else on earth did what they have done on that scale (except maybe governments and scientists). However, I agree the article is right on. As someone who has done a lot of work with different Hadoop distributions (Cloudera, Hortonworks and MapR to name a few), I can say that compared to the performance and ease of use of more traditional data platforms, Hadoop is a different and more complex creature. It has evolved quickly in some regards and slowly in others. It's way easier to get a Cloudera system up and running today (for example) than it was in the past. "Hadoop" by itself is a misnomer, it's not like it's one technology. HDFS, MapReduce (and MR2), Hive, HBase, Spark, Pig, Flume, etc. etc. - it's an ecosystem. It's also a mashup of many things depending on what distro you're talking about - too many things - a "data hub", an "operational data store", a "pre-data warehouse staging", an ETL platform, a "search" platform, a data governance platform, a data archiving solution, a real-time data query / streaming data processing etc. etc. etc. I think Hadoop did one important thing to our overall awareness, and that was to think of a "data platform", "data processing", "data architecture" and "data infrastructure" as a major, and "big" strategic part of an organization. I think the rise of "big data" thinking along with the rise of the "chief data officer", "data scientists" and "data engineers" and the like all goes together - a shift in thinking we made where we realized we could do more with more data (for better or worse). There are many new and old technologies that help people implement the data infrastructure in their organizations - not just Hadoop. Hadoop just happened to be a major player at the epicenter at this (and open source as well), and if anything, has spurred technological advancement in databases, NoSQL and NewSQL and OldSQL alike, ETL, data mining, AI/machine learning, etc. So it has played its role, whether it specifically stays relevant or not as we perceive it.

by ghc 10y ago

sixdimensional 10y ago

To be clear, many of the comments are from Bob Muglia, CEO of Snowflake Computing which is a cloud based columnar data warehouse solution. Presumably, much easier to implement than Hadoop, so you have to take those comments with a grain of salt. Also, those entities who the article claims have a grasp on Hadoop are some of the main people who helped develop it - lest we forget that the early work on Hadoop came out of Yahoo. "Web-scale" companies by their very nature have to own this technology, because almost nobody else on earth did what they have done on that scale (except maybe governments and scientists). However, I agree the article is right on. As someone who has done a lot of work with different Hadoop distributions (Cloudera, Hortonworks and MapR to name a few), I can say that compared to the performance and ease of use of more traditional data platforms, Hadoop is a different and more complex creature. It has evolved quickly in some regards and slowly in others. It's way easier to get a Cloudera system up and running today (for example) than it was in the past. "Hadoop" by itself is a misnomer, it's not like it's one technology. HDFS, MapReduce (and MR2), Hive, HBase, Spark, Pig, Flume, etc. etc. - it's an ecosystem. It's also a mashup of many things depending on what distro you're talking about - too many things - a "data hub", an "operational data store", a "pre-data warehouse staging", an ETL platform, a "search" platform, a data governance platform, a data archiving solution, a real-time data query / streaming data processing etc. etc. etc. I think Hadoop did one important thing to our overall awareness, and that was to think of a "data platform", "data processing", "data architecture" and "data infrastructure" as a major, and "big" strategic part of an organization. I think the rise of "big data" thinking along with the rise of the "chief data officer", "data scientists" and "data engineers" and the like all goes together - a shift in thinking we made where we realized we could do more with more data (for better or worse). There are many new and old technologies that help people implement the data infrastructure in their organizations - not just Hadoop. Hadoop just happened to be a major player at the epicenter at this (and open source as well), and if anything, has spurred technological advancement in databases, NoSQL and NewSQL and OldSQL alike, ETL, data mining, AI/machine learning, etc. So it has played its role, whether it specifically stays relevant or not as we perceive it.