4 ms·

Linux lessons for Hadoop doubters

21 points by joxie 14y ago

mattmanser 14y ago
I'm not entirely sure the point the author is trying to make. As an honest question as someone who's not even played with it, why would hadoop ever get past 1% or probably even 0.01%? Doesn't it have a very specific use case? Is he only talking about penetration in the BigData market? If not, why would it ever supplant RDBMS or NoSQL datastores?
- sounds 14y ago
  I'm obviously the wrong person to talk about Hadoop - having so little experience with it - but this article strikes me as link bait. To answer your question, the author seems to think if you have Hadoop, you'll never need NoSQL. Here's his quote: "I think we're going to see Hadoop produce the next Oracle-sized database company. We're also likely to see such a company emerge from the NoSQL ranks, but Hadoop is a near certain bet right now." This quote also seems wrong to me: "In fact, if anything, we are still only on the first few steps of the Big Data marathon, a race that Hadoop seems set to win despite its many shortcomings." I believe the Big Data industry is well on its way. There are multiple big players offering an array of cloud services and software.
- lmm 14y ago
  Hadoop is the first NoSQL datastore to reach a level of maturity and big-name support that many enterprises insist on. In these companies the only thing for it to supplant is traditional relational databases, and the advantages there should be obvious.
  - mattmanser 14y ago
    I don't see any obvious advantages of NoSQL in most enterprise apps. Although it's such a fuzzy term these days.
- monstrado 14y ago
  I wouldn't think of Hadoop being a "specific use case", because it's really just a platform that you design around. Of course, everything needs a use case, but Hadoop's place in the industry is not nearly as linear as people think it is. The problem at the moment is people don't understand or know how to work with Hadoop, and that's what companies like Cloudera and Hortonworks are trying to solve, they are trying to show businesses, 1) How easy it is to deploy and 2) How to actually apply it to their current datasets. As someone who works at a Hadoop company, I can tell you that customers are still trying to wrap their heads around the technology, but the good news is, there is major progress. I notice that, rather than Hadoop not giving enough to the user, the user is usually not giving enough to Hadoop. Hadoop challenges its users to think differently, which for some businesses can be quite difficult. Regarding your question about NoSQL, since Hadoop is a platform/framework, there are already efforts to design databases that take advantage of the stack (namely, HDFS <-- Hadoop Distributed Filesystem). The most popular of these contributions is HBase which is a NoSQL (KV) Column Based DB, based on BigTable and much like Cassandra. It's horizontally scalable and is often referred to as "highly consistent". Just to make a point though, there is so much that runs on the Hadoop platform. Top level projects like * Pig (Scripting language developed by Yahoo, think SQL meets Python) * Hive (Map RDBMS style schemas to data after it's been stored, let's you query data with SQL) * HBase (NoSQL column based database store based on Google's BigTable, think Cassandra but very consistent) * HDFS (Highly scalable and consistent distributed FS that was designed with MapReduce in mind) ^^ Some people use this JUST for storage since it's so easily scalable. * There are others like Mahout which is machine learning, and in my opinion, black magic :P Hope this helps :D
deleted 14y ago
[deleted]
dkhenry 14y ago
I'll go on the record ansd Say Hadoop won't ever make it past the 1% , but rather we will take the lessons learned from Hadoop and create a better and more usefull system. You can see this already with Pig and Hive. People are learning from the shortcommings of hadoop and building on top of it to try and make it usefull. Once the right mix of frustration and technical skill gets together they will evolve the system to make it usefull. That is when you will see it get enterprise traction.
- monstrado 14y ago
  I work regularly with Hadoop users and I there is rarely talks of "shortcomings" of Hadoop. More often than not, rather than Hadoop not giving enough to the user, the user is not giving enough to Hadoop. Once users start understanding the power of the platform, they start to do some really incredible things. Hive and Pig are just applications that interface with Hadoop, although they could be classified as "SQL" or "PigLatin" to Java Map/Reduce, they are also more than that. For example, Hive has a shared metastore so you can treat Hadoop kind of like a shared RDBMS that allows you to map schemas after the data has already been transformed. Most the people I talk to, especially on here, are still pretty wet behind the ears when it comes to Hadoop. It's a very daunting technology, but I can assure you there is nothing but progress and innovation in the community.
  - dkhenry 14y ago
    I agree that this is a new and daunting technology, but its also technically grounded at this point. I think the problems people have with it ( extensive knowledge needed to use it ) will be fixed iteravly, but will need a major change to the platform.
jandrewrogers 14y ago
Linux and Hadoop are not analogous. I've used Linux since the pre-1.0 days and been building distributed computing and database platforms since before Hadoop existed; I've seen both of these ecosystems grow from the inside. Linux won the server wars because it rapidly iterated and evolved in a time when a free UNIX-like operating system was desperately needed. In the early days of the Internet, no operating system was ready to be a heavy duty Internet server OS. FreeBSD, started from a stronger position than Linux but development was carefully controlled by academic purists. Eric Raymond's "The Cathedral and The Bazaar" was about these two communities. Operating systems, like Linux, tend toward natural monopolies. It is extraordinarily expensive to build a competitor. Linux was the most agile and rapidly evolving competitor in an early market that was poorly served by incumbents. While it took the rest of the world years to notice, it was obvious by the mid 1990s that Linux was on a trajectory to take over that market, not because it was good (back then, it wasn't) but because it rapidly evolved in response to user needs. Hadoop is more like MySQL than Linux. It provided a very early option in a market devoid of options. Technically the implementation is poor and the details of these choices are exposed to users as features and interface. This makes it difficult to evolve the core because it pisses off the early users who have become used to the way it works. Linux had the advantage that it copied a good design (UNIX), which allowed it to iterate the core without breaking too much for users. Over the long term, these poor technical choices early on limit the ability to grow because fixing them alienates the users. MySQL had a strong run of dominance, particularly in the early days, but the inability to easily evolve into a robust and full-featured system allowed PostgreSQL to eventually relegate it to relatively niche use cases that can be easily replaced by other bits of technology (like NoSQL data stores). Hadoop has one other quality that makes it fundamentally different than Linux. It is a relatively simple system that can easily be functionally copied. Making a Hadoop-like clone using alternative tool sets is not a big task. While very popular, it would not require an extraordinary investment to produce an alternative that is obviously superior. Some form of Hadoop will be with us for a long time. It will always have a use case. But unlike Linux (and more like MySQL), it will ultimately be marginalized by its many manifest technical weaknesses. Not only is it easy to build a significantly better "Hadoop" than the free Hadoop -- many companies already have -- but there are some very important distributed computing technologies and features which Hadoop does not have and which would be difficult to implement on that platform even if customers want them. The larger NoSQL and distributed computing space is evolving very rapidly with many potential alternatives. The only lessons for Hadoop from the history of Linux and open source generally is that it is in a ripe position to be marginalized over the long term. I've seen this hype cycle many times for products in similar positions. I remember, for example, when Perl was the programming language of the Web. Perl quickly dominated a void in the market at the time but where is it today?

21 points by joxie 14y ago

I'm not entirely sure the point the author is trying to make. As an honest question as someone who's not even played with it, why would hadoop ever get past 1% or probably even 0.01%? Doesn't it have a very specific use case? Is he only talking about penetration in the BigData market? If not, why would it ever supplant RDBMS or NoSQL datastores?

sounds 14y ago
I'm obviously the wrong person to talk about Hadoop - having so little experience with it - but this article strikes me as link bait. To answer your question, the author seems to think if you have Hadoop, you'll never need NoSQL. Here's his quote: "I think we're going to see Hadoop produce the next Oracle-sized database company. We're also likely to see such a company emerge from the NoSQL ranks, but Hadoop is a near certain bet right now." This quote also seems wrong to me: "In fact, if anything, we are still only on the first few steps of the Big Data marathon, a race that Hadoop seems set to win despite its many shortcomings." I believe the Big Data industry is well on its way. There are multiple big players offering an array of cloud services and software.
lmm 14y ago
Hadoop is the first NoSQL datastore to reach a level of maturity and big-name support that many enterprises insist on. In these companies the only thing for it to supplant is traditional relational databases, and the advantages there should be obvious.
- mattmanser 14y ago
  I don't see any obvious advantages of NoSQL in most enterprise apps. Although it's such a fuzzy term these days.
monstrado 14y ago
I wouldn't think of Hadoop being a "specific use case", because it's really just a platform that you design around. Of course, everything needs a use case, but Hadoop's place in the industry is not nearly as linear as people think it is. The problem at the moment is people don't understand or know how to work with Hadoop, and that's what companies like Cloudera and Hortonworks are trying to solve, they are trying to show businesses, 1) How easy it is to deploy and 2) How to actually apply it to their current datasets. As someone who works at a Hadoop company, I can tell you that customers are still trying to wrap their heads around the technology, but the good news is, there is major progress. I notice that, rather than Hadoop not giving enough to the user, the user is usually not giving enough to Hadoop. Hadoop challenges its users to think differently, which for some businesses can be quite difficult. Regarding your question about NoSQL, since Hadoop is a platform/framework, there are already efforts to design databases that take advantage of the stack (namely, HDFS <-- Hadoop Distributed Filesystem). The most popular of these contributions is HBase which is a NoSQL (KV) Column Based DB, based on BigTable and much like Cassandra. It's horizontally scalable and is often referred to as "highly consistent". Just to make a point though, there is so much that runs on the Hadoop platform. Top level projects like * Pig (Scripting language developed by Yahoo, think SQL meets Python) * Hive (Map RDBMS style schemas to data after it's been stored, let's you query data with SQL) * HBase (NoSQL column based database store based on Google's BigTable, think Cassandra but very consistent) * HDFS (Highly scalable and consistent distributed FS that was designed with MapReduce in mind) ^^ Some people use this JUST for storage since it's so easily scalable. * There are others like Mahout which is machine learning, and in my opinion, black magic :P Hope this helps :D

deleted 14y ago

[deleted]

dkhenry 14y ago

I'll go on the record ansd Say Hadoop won't ever make it past the 1% , but rather we will take the lessons learned from Hadoop and create a better and more usefull system. You can see this already with Pig and Hive. People are learning from the shortcommings of hadoop and building on top of it to try and make it usefull. Once the right mix of frustration and technical skill gets together they will evolve the system to make it usefull. That is when you will see it get enterprise traction.

monstrado 14y ago
I work regularly with Hadoop users and I there is rarely talks of "shortcomings" of Hadoop. More often than not, rather than Hadoop not giving enough to the user, the user is not giving enough to Hadoop. Once users start understanding the power of the platform, they start to do some really incredible things. Hive and Pig are just applications that interface with Hadoop, although they could be classified as "SQL" or "PigLatin" to Java Map/Reduce, they are also more than that. For example, Hive has a shared metastore so you can treat Hadoop kind of like a shared RDBMS that allows you to map schemas after the data has already been transformed. Most the people I talk to, especially on here, are still pretty wet behind the ears when it comes to Hadoop. It's a very daunting technology, but I can assure you there is nothing but progress and innovation in the community.
- dkhenry 14y ago
  I agree that this is a new and daunting technology, but its also technically grounded at this point. I think the problems people have with it ( extensive knowledge needed to use it ) will be fixed iteravly, but will need a major change to the platform.

jandrewrogers 14y ago

Linux and Hadoop are not analogous. I've used Linux since the pre-1.0 days and been building distributed computing and database platforms since before Hadoop existed; I've seen both of these ecosystems grow from the inside. Linux won the server wars because it rapidly iterated and evolved in a time when a free UNIX-like operating system was desperately needed. In the early days of the Internet, no operating system was ready to be a heavy duty Internet server OS. FreeBSD, started from a stronger position than Linux but development was carefully controlled by academic purists. Eric Raymond's "The Cathedral and The Bazaar" was about these two communities. Operating systems, like Linux, tend toward natural monopolies. It is extraordinarily expensive to build a competitor. Linux was the most agile and rapidly evolving competitor in an early market that was poorly served by incumbents. While it took the rest of the world years to notice, it was obvious by the mid 1990s that Linux was on a trajectory to take over that market, not because it was good (back then, it wasn't) but because it rapidly evolved in response to user needs. Hadoop is more like MySQL than Linux. It provided a very early option in a market devoid of options. Technically the implementation is poor and the details of these choices are exposed to users as features and interface. This makes it difficult to evolve the core because it pisses off the early users who have become used to the way it works. Linux had the advantage that it copied a good design (UNIX), which allowed it to iterate the core without breaking too much for users. Over the long term, these poor technical choices early on limit the ability to grow because fixing them alienates the users. MySQL had a strong run of dominance, particularly in the early days, but the inability to easily evolve into a robust and full-featured system allowed PostgreSQL to eventually relegate it to relatively niche use cases that can be easily replaced by other bits of technology (like NoSQL data stores). Hadoop has one other quality that makes it fundamentally different than Linux. It is a relatively simple system that can easily be functionally copied. Making a Hadoop-like clone using alternative tool sets is not a big task. While very popular, it would not require an extraordinary investment to produce an alternative that is obviously superior. Some form of Hadoop will be with us for a long time. It will always have a use case. But unlike Linux (and more like MySQL), it will ultimately be marginalized by its many manifest technical weaknesses. Not only is it easy to build a significantly better "Hadoop" than the free Hadoop -- many companies already have -- but there are some very important distributed computing technologies and features which Hadoop does not have and which would be difficult to implement on that platform even if customers want them. The larger NoSQL and distributed computing space is evolving very rapidly with many potential alternatives. The only lessons for Hadoop from the history of Linux and open source generally is that it is in a ripe position to be marginalized over the long term. I've seen this hype cycle many times for products in similar positions. I remember, for example, when Perl was the programming language of the Web. Perl quickly dominated a void in the market at the time but where is it today?