Nice read, but I'd like to point out that this is a very well studied problem, particularly in HPC. I'd point to a few decent resources, but the only one that m

Nice read, but I'd like to point out that this is a very well studied problem, particularly in HPC. I'd point to a few decent resources, but the only one that my google-fu found was https://www.cs.fsu.edu/~engelen/courses/HPC/Algorithms1.pdf https://www.cs.fsu.edu/~engelen/courses/HPC/Algorithms1.pdf The main advice is that you need to carefully balance communication overhead against work done and also analyse the optimum number of workers per node.