There's an obvious tradeoff between number of partitions and overhead though. Too many, and the increased network traffic will impact the usefulness of the clus

There's an obvious tradeoff between number of partitions and overhead though. Too many, and the increased network traffic will impact the usefulness of the cluster, especially if they need to be fetched from multiple racks for a single computation. You sound well informed, I just wanted to point out that HFDS replication is not a simple problem. Also, hopefully you have more than one disk per node (assuming physical machines) ;)