3 ms·

Interestingly enough I think the people who gain the most from this paper are cloud providers. Very few orgs will buy 256 GPUs, but with linear scaling, renting

Interestingly enough I think the people who gain the most from this paper are cloud providers. Very few orgs will buy 256 GPUs, but with linear scaling, renting them makes a lot of sense.

shezi 9y ago
There are "only" 32 GPUs in the cluster, with 8 workers per GPU.
- dgacmu 9y ago
  They had 32 servers in the cluster, each with 8 P100 GPUs. Each GPU was one "worker" in their parlance. ("How to train ResNet-50 in one hour on two million dollars of hardware." :-)