4 ms·

Actually I know several projects bursted into flames because of Hadoop. There is a notion among some developers of Hadoop solving all of the problems and this i

Actually I know several projects bursted into flames because of Hadoop. There is a notion among some developers of Hadoop solving all of the problems and this is the only solution they consider (literary). Needless to say that real time systems with tight SLA cannot really use Hadoop, and if you are trying to sell it for this use case, you might get fired. :)

sixdimensional 11y ago
I concur and I have seen the same, especially for folks with complex projects, tight deadlines and budgets, and older distros of Hadoop that were not as stable and/or production ready. In a rush to get to the next big thing, many have taken a lot of risk with Hadoop. Some with the resources to really support/make Hadoop their own have had a lot of success. But I think there are fewer who have success stories.
- istvan__ 11y ago
  On the other hand, Hadoop is great for analytics. If it is down there is no customer facing outage. Sometimes the failover works and only few things lost, you are back in ~15 minutes if you want to check the fs image consistency. For internal teams this is acceptable SLA. If you don't care about consistency than you can probably do this faster. I just don't see a customer facing SLA sensitive workload fitting this well. At least I would not sleep well being oncall... :)
  - sixdimensional 11y ago
    I don't know that Hadoop is always great for analytics in every case. Sometimes the overhead of spinning up mapreduces (for example) outweighs the cost of running analytics using other methods. Especially when you have demanding requirements for subsecond or milli-second response time to provide such analytics - I don't think Hadoop is there yet, although it's getting there. Generally, I agree though, in an extremely demanding SLA environment... I probably wouldn't sleep well in that case either! I think that was kind of the point of the paper as presented - Hadoop is seen as a panacea, when in reality there might be other, simpler approaches that work just as well or better. It really does depend on the use case, the volume/types of data, cost/requirements, etc. For that matter, what the Hadoop ecosystem "is" (vs. just the Apache Hadoop project itself) means so many things now. HDFS (storage), YARN (distributed job/resource management), mapreduces, Hive, HBase, etc. vs. new engines, like Apache Spark, for example, which can run inside or outside of Hadoop. Adding to that the different distros and fragmenting Hadoop ecosystem, constantly changing versions, etc. - I don't know about you but it can be a nightmare even for analytics (in some cases). When properly supported by knowledgeable staff with a deep grasp for what it can do as a platform, Hadoop can certainly be and do a lot of things for a lot of use cases.
    - istvan__ 11y ago
      I agree, I would not recommend Hadoop for a startup to install and maintain it. There are several datawarehouse as a service offerings out there. More recently there was a rise of good and cheap cloud based systems and Redshift got a lot better as well. Much easier to integrate, probably lover TCO too.

sixdimensional 11y ago

I concur and I have seen the same, especially for folks with complex projects, tight deadlines and budgets, and older distros of Hadoop that were not as stable and/or production ready. In a rush to get to the next big thing, many have taken a lot of risk with Hadoop. Some with the resources to really support/make Hadoop their own have had a lot of success. But I think there are fewer who have success stories.

istvan__ 11y ago
On the other hand, Hadoop is great for analytics. If it is down there is no customer facing outage. Sometimes the failover works and only few things lost, you are back in ~15 minutes if you want to check the fs image consistency. For internal teams this is acceptable SLA. If you don't care about consistency than you can probably do this faster. I just don't see a customer facing SLA sensitive workload fitting this well. At least I would not sleep well being oncall... :)
- sixdimensional 11y ago
  I don't know that Hadoop is always great for analytics in every case. Sometimes the overhead of spinning up mapreduces (for example) outweighs the cost of running analytics using other methods. Especially when you have demanding requirements for subsecond or milli-second response time to provide such analytics - I don't think Hadoop is there yet, although it's getting there. Generally, I agree though, in an extremely demanding SLA environment... I probably wouldn't sleep well in that case either! I think that was kind of the point of the paper as presented - Hadoop is seen as a panacea, when in reality there might be other, simpler approaches that work just as well or better. It really does depend on the use case, the volume/types of data, cost/requirements, etc. For that matter, what the Hadoop ecosystem "is" (vs. just the Apache Hadoop project itself) means so many things now. HDFS (storage), YARN (distributed job/resource management), mapreduces, Hive, HBase, etc. vs. new engines, like Apache Spark, for example, which can run inside or outside of Hadoop. Adding to that the different distros and fragmenting Hadoop ecosystem, constantly changing versions, etc. - I don't know about you but it can be a nightmare even for analytics (in some cases). When properly supported by knowledgeable staff with a deep grasp for what it can do as a platform, Hadoop can certainly be and do a lot of things for a lot of use cases.
  - istvan__ 11y ago
    I agree, I would not recommend Hadoop for a startup to install and maintain it. There are several datawarehouse as a service offerings out there. More recently there was a rise of good and cheap cloud based systems and Redshift got a lot better as well. Much easier to integrate, probably lover TCO too.