4 ms·

What is being shared between GPUs? Training data is easy to duplicate and share nothing. Large models with shared weights get tricky but less frequent asynchr

by emcq 10y ago

What is being shared between GPUs? Training data is easy to duplicate and share nothing. Large models with shared weights get tricky but less frequent asynchronous updates with schemes like hogwild seem to work with SGD. I believe TF has support for this too. It won't scale linearly but might be good enough. There's some excitement about synthetic gradients to allow less communication and further parallelism. The hpc community has certainly leveraged 400 GPU clusters. It seems like a fun problem, and if you want to focus the resources, there isn't anything insurmountable about utilizing 400 GPUs :)

dharma1 10y ago
The DeepMind paper on synthetic gradients is super interesting for training in clusters. I hope it gets built into TF soon. Also really hope there will be more options for GPU hardware on public clouds. Eventually the pieces will come together and it will be trivial to deploy cloud 400gpus for an hour to run the load your local 4gpu workstation would spend 100 hours on, but we are definitely not there yet. We are working on a packaging format called snappy (http://www.snapcraft.io http://www.snapcraft.io) and starting to talk to TF guys about packaging TF with it (kubernetes is already packaged as a snap) - hopefully this will take some pain away once it's working