>>Eventually, however, data communication costs get too high Looks like the program is spending too much time in 'MPI_RECV'? Clearly the same time would be spe

>>Eventually, however, data communication costs get too high Looks like the program is spending too much time in 'MPI_RECV'? Clearly the same time would be spent by any other hardware device, and raw compute power wont fix this. You need a faster interconnect. Indeed, among the few fixes for this problem are complicated changes to the algorithm, such a maintaining some kind of network routing scheme, which often involves complicated logic difficult to implement on anything but a CPU.