3 ms·

Cloudera and Hortonwork promise an easy way to deploy on-premise Hadoop clusters. Yet, HDP & CDH are still too hard to maintain for the majority of companies:

by dikei 5y ago

Cloudera and Hortonwork promise an easy way to deploy on-premise Hadoop clusters. Yet, HDP & CDH are still too hard to maintain for the majority of companies: * Tuning for HDFS and HBase is hard. Too many files and GC pause will kill you. Hive Metastore is also a bottleneck. * It's a nightmare to figure out how to configure some third-party software that's outside the bundle. * It's even harder, if you want to use a more up-to-date version of a software included in the bundle. We still haven't figured out how to use Spark 3 with our old HDP clusters. * Sometime the included software is a buggy mess. For example, HDP 3 provides HBase 2.0, which is absolutely unusable. Luckily, we were able to migrate to HBase 2.2 before putting too much data into it. All in all, it still takes a lot of ops work to own a Cloudera's Hadoop clusters. I can't wait until the time when we can move to a better solution.

pm90 5y ago
This might be a silly question but: Why is it so hard to use? What made cloudera not see how difficult their product was to use and invest in fixing that? It appears to have costed them the market, yes?
- nl 5y ago
  > Why is it so hard to use? What made cloudera not see how difficult their product was to use and invest in fixing that? It is packaged open source software, from multiple projects (some of which are financed by Cloudera) all of which release at different paces. And distributed systems are really hard to get right. There are lots of knobs to tune, all of which are needed in some circumstances, but.. there are lots of things to go wrong. > It appears to have costed them the market, yes? Cloudera is probably the market leader for on-prem Hadoop clusters. This is a big market (even now - lots of defence clients who have issues with cloud). But generally this market is getting eaten by Cloud SAAS products.
- dikei 5y ago
  TL DR: Distributed system is hard, and Hadoop cluster are not really designed for a dynamic environment. There are as probably as many services running inside a Hadoop clusters as a Micro Service meshes, except: * The services can be huge: HDFS Namenode can takes hundreds GB of RAM, hours to start up in a multi PB cluster. Updating configuration requires restarting, so it is a huge pain if you care about zero downtime. * The services are often bound to host and can not be easily migrated to other hosts. * The communication interfaces between services are not well-defined: Hive Metastore, for example, didn't have a formal protocol documentation for a long time, yet everything depends on it to store metadata (I'm not even sure if they have the protocol now). As a result, services are often tightly coupled: you need everything to be compiled with the same libraries version, or stuffs might not work correctly. Furthermore, due to dynamic code loading, issues might not surface until days later - in the middle of the night, making life miserable for everyone.

by dikei 5y ago

pm90 5y ago

This might be a silly question but: Why is it so hard to use? What made cloudera not see how difficult their product was to use and invest in fixing that? It appears to have costed them the market, yes?

nl 5y ago
> Why is it so hard to use? What made cloudera not see how difficult their product was to use and invest in fixing that? It is packaged open source software, from multiple projects (some of which are financed by Cloudera) all of which release at different paces. And distributed systems are really hard to get right. There are lots of knobs to tune, all of which are needed in some circumstances, but.. there are lots of things to go wrong. > It appears to have costed them the market, yes? Cloudera is probably the market leader for on-prem Hadoop clusters. This is a big market (even now - lots of defence clients who have issues with cloud). But generally this market is getting eaten by Cloud SAAS products.
dikei 5y ago
TL DR: Distributed system is hard, and Hadoop cluster are not really designed for a dynamic environment. There are as probably as many services running inside a Hadoop clusters as a Micro Service meshes, except: * The services can be huge: HDFS Namenode can takes hundreds GB of RAM, hours to start up in a multi PB cluster. Updating configuration requires restarting, so it is a huge pain if you care about zero downtime. * The services are often bound to host and can not be easily migrated to other hosts. * The communication interfaces between services are not well-defined: Hive Metastore, for example, didn't have a formal protocol documentation for a long time, yet everything depends on it to store metadata (I'm not even sure if they have the protocol now). As a result, services are often tightly coupled: you need everything to be compiled with the same libraries version, or stuffs might not work correctly. Furthermore, due to dynamic code loading, issues might not surface until days later - in the middle of the night, making life miserable for everyone.