I believe the network had become a bottleneck. As per the article: > [O]ur Spark cluster was able to sustain ... 1.1 GB/s/node network activity during the redu

I believe the network had become a bottleneck. As per the article: > [O]ur Spark cluster was able to sustain ... 1.1 GB/s/node network activity during the reduce phase, saturating the 10Gbps link available on these machines. If the network is the bottleneck it makes sense to reduce the number of nodes to reduce the network communications.