3 ms·

I'd love to understood how the Hadoop hype and marketing team generated so much unwarranted interest in Hadoop. I'm witnessing a feeding frenzy for Hadoop tale

by dbecker 13y ago

I'd love to understood how the Hadoop hype and marketing team generated so much unwarranted interest in Hadoop. I'm witnessing a feeding frenzy for Hadoop talent in situations where there's absolutely no need for Hadoop, and I can't recall anything like this for any other software.

meshko 13y ago
I was with you until "and I can't recall anything like this for any other software."
- dbecker 13y ago
  Just for my curiosity, what other software had the same level of unwarranted demand and hype?
  - beagle3 13y ago
    Java around 1998. All bugs a thing of the past! Write once run anywhere! Applets taking over the world!
  - stefanve 13y ago
    I don't think it's about software per se, but more about the idea of Big Data. We have seen many ideas that have come and go most of the times a good idea but a good idea is not something you should use for everything. From the top of my head Ajax (very handy but lot of times misused and misunderstood or just DHTML) No SQL (can be handy, but for some people it is like a religion) XML (why use CSV when you are able to use XML) Cloud (Oh by cloud you mean the internet?) and many more :)
bunderbunder 13y ago
I think it goes like this: Certain very popular companies that everyone wants to emulate, who have truly enormous needs in the initial data crunching (e.g., ETL) department, were running into bottlenecks related to raw I/O bandwidth. They hit on a "let the mountain come to Mohammed" insight that helped them get past that problem, so that they could do their ETL jobs in less time and ultimately keep their workhorse databases (e.g., web search indexes) better-fed. Simultaneously, a whole lot of people who were not having the same problems, but who want to believe that they are like companies who have those problems (because who doesn't want to be Google?) started also running into problems with handling large amounts of data. Unfortunately, the most popular mistake in applying Feynman's Algorithm[1] is to skip the first step. Rather than investigating their problems and recognizing that the issue was poor tooling or inefficient implementations and they weren't actually coming anywhere close to any true limits of the kind that the companies that came up with Big Data were trying to get around, they instead just went, "Hey, X company that we look up to is also having problems that look cosmetically similar to ours, and they use Y technology - let's give that a try!" and proceeded to dive straight into constructing bamboo control towers and coconut radios without ever looking back. After that, well, I think it's a tragedy of mostly-rational behavior. Managers don't understand these technologies well enough to take programmers' advice skeptically, so they have to listen to their engineers. Engineers want to make their CV's look nice and impress their managers, so they've got every reason to come up with excuses to use $HOT_NEW_TOY. As usual, everyone individually acting in accordance with their rational self-interest is not the same thing as everyone collectively acting in a way that produces ideal results for the parties involved. [1] http://c2.com/cgi/wiki?FeynmanAlgorithm http://c2.com/cgi/wiki?FeynmanAlgorithm
alan_cx 13y ago
Er, Cloud?

by dbecker 13y ago

meshko 13y ago

I was with you until "and I can't recall anything like this for any other software."

dbecker 13y ago
Just for my curiosity, what other software had the same level of unwarranted demand and hype?
- beagle3 13y ago
  Java around 1998. All bugs a thing of the past! Write once run anywhere! Applets taking over the world!
- stefanve 13y ago
  I don't think it's about software per se, but more about the idea of Big Data. We have seen many ideas that have come and go most of the times a good idea but a good idea is not something you should use for everything. From the top of my head Ajax (very handy but lot of times misused and misunderstood or just DHTML) No SQL (can be handy, but for some people it is like a religion) XML (why use CSV when you are able to use XML) Cloud (Oh by cloud you mean the internet?) and many more :)

bunderbunder 13y ago

I think it goes like this: Certain very popular companies that everyone wants to emulate, who have truly enormous needs in the initial data crunching (e.g., ETL) department, were running into bottlenecks related to raw I/O bandwidth. They hit on a "let the mountain come to Mohammed" insight that helped them get past that problem, so that they could do their ETL jobs in less time and ultimately keep their workhorse databases (e.g., web search indexes) better-fed. Simultaneously, a whole lot of people who were not having the same problems, but who want to believe that they are like companies who have those problems (because who doesn't want to be Google?) started also running into problems with handling large amounts of data. Unfortunately, the most popular mistake in applying Feynman's Algorithm[1] is to skip the first step. Rather than investigating their problems and recognizing that the issue was poor tooling or inefficient implementations and they weren't actually coming anywhere close to any true limits of the kind that the companies that came up with Big Data were trying to get around, they instead just went, "Hey, X company that we look up to is also having problems that look cosmetically similar to ours, and they use Y technology - let's give that a try!" and proceeded to dive straight into constructing bamboo control towers and coconut radios without ever looking back. After that, well, I think it's a tragedy of mostly-rational behavior. Managers don't understand these technologies well enough to take programmers' advice skeptically, so they have to listen to their engineers. Engineers want to make their CV's look nice and impress their managers, so they've got every reason to come up with excuses to use $HOT_NEW_TOY. As usual, everyone individually acting in accordance with their rational self-interest is not the same thing as everyone collectively acting in a way that produces ideal results for the parties involved. [1] http://c2.com/cgi/wiki?FeynmanAlgorithm http://c2.com/cgi/wiki?FeynmanAlgorithm

alan_cx 13y ago

Er, Cloud?