There are different ways to exploit massive parallelism. One is to perform distributed training of an individual model, and the degree to which you can see goo

There are different ways to exploit massive parallelism. One is to perform distributed training of an individual model, and the degree to which you can see good scaling here indeed depends on the network itself (as well as the quality of the distributed training algorithm itself, with things like Horovod being best-in-breed). A second way to exploit parallelism is via hyperparameter search, and in particular when leveraging early-stopping based approaches like Hyperband/ASHA. Speedups in this setting tend be fairly robust to the the choice of network. As you mentioned, in such settings you can do meaningful amounts of work independently, and moreover can leverage asynchrony to further avoid bottlenecks. Disclaimer: I am one of the co-founders of Determined, and also one of the developers of Hyperband/Asha.