> Another problem with hdfs design is capacity adding; hdfs has shitty rebalancing tools. The typical problem is this: hot data is new data. So if you add cap

> Another problem with hdfs design is capacity adding; hdfs has shitty rebalancing tools. The typical problem is this: hot data is new data. So if you add capacity to hdfs, your old capacity is say 80% full and your new capacity is 0% full so where does all new data go? To the new capacity. Which is quickly 100% saturated and choking your jobs. Properly rebalancing a dfs is a hard problem on it's own. Further, hdfs' use of triple replication (well, you can actually set the replication level) instead of better algorithms like erasure encoding does no favors to performance as well. The tools for HDFS balancing are quite primitive, which is why a lot of people like using MapR or Cassandra, but the rebalancing tools built in to HDFS after a 0.18.x along with a few good practices would get you there... just not the most efficient way possible.