The optimization problem at the inner loop of these methods doesn't handle latency very well. This is covered a bit in the DOTA 2 rerun work [1] (e.g., Figure 5

The optimization problem at the inner loop of these methods doesn't handle latency very well. This is covered a bit in the DOTA 2 rerun work [1] (e.g., Figure 5b on staleness). So for both large models (which are split up amongst multiple accelerators but effectively one "computer"), as well as learning from "your friends" you want to be more tightly coupled than something like seti/folding@home. There are plenty of workloads that are great for embarrassingly parallel (and now "pleasingly parallel") workloads, but training a single model isn't one of them. [1] https://cdn.openai.com/dota-2.pdf https://cdn.openai.com/dota-2.pdf