3 ms·

Many of the models people are building here, such as generative image models, take a few days to train (say, 100 hours) on our 4 GPU boxes. Research would be fa

by tlb 10y ago

Many of the models people are building here, such as generative image models, take a few days to train (say, 100 hours) on our 4 GPU boxes. Research would be faster if we could train on 400 GPUS in one hour, but the communication bandwidth required makes it hard to scale.

deleted 10y ago
[deleted]
dragandj 10y ago
I am aware of that. The question was not whether more than 8 GPUs would be useful in ideal circumstances, it was about how many people actually use that functionality with other frameworks other than DL4J? BTW I do not have any connection with DL4J, nor do I use it.
emcq 10y ago
What is being shared between GPUs? Training data is easy to duplicate and share nothing. Large models with shared weights get tricky but less frequent asynchronous updates with schemes like hogwild seem to work with SGD. I believe TF has support for this too. It won't scale linearly but might be good enough. There's some excitement about synthetic gradients to allow less communication and further parallelism. The hpc community has certainly leveraged 400 GPU clusters. It seems like a fun problem, and if you want to focus the resources, there isn't anything insurmountable about utilizing 400 GPUs :)
- dharma1 10y ago
  The DeepMind paper on synthetic gradients is super interesting for training in clusters. I hope it gets built into TF soon. Also really hope there will be more options for GPU hardware on public clouds. Eventually the pieces will come together and it will be trivial to deploy cloud 400gpus for an hour to run the load your local 4gpu workstation would spend 100 hours on, but we are definitely not there yet. We are working on a packaging format called snappy (http://www.snapcraft.io http://www.snapcraft.io) and starting to talk to TF guys about packaging TF with it (kubernetes is already packaged as a snap) - hopefully this will take some pain away once it's working

by tlb 10y ago

deleted 10y ago

[deleted]

dragandj 10y ago

I am aware of that. The question was not whether more than 8 GPUs would be useful in ideal circumstances, it was about how many people actually use that functionality with other frameworks other than DL4J? BTW I do not have any connection with DL4J, nor do I use it.

emcq 10y ago

What is being shared between GPUs? Training data is easy to duplicate and share nothing. Large models with shared weights get tricky but less frequent asynchronous updates with schemes like hogwild seem to work with SGD. I believe TF has support for this too. It won't scale linearly but might be good enough. There's some excitement about synthetic gradients to allow less communication and further parallelism. The hpc community has certainly leveraged 400 GPU clusters. It seems like a fun problem, and if you want to focus the resources, there isn't anything insurmountable about utilizing 400 GPUs :)

dharma1 10y ago
The DeepMind paper on synthetic gradients is super interesting for training in clusters. I hope it gets built into TF soon. Also really hope there will be more options for GPU hardware on public clouds. Eventually the pieces will come together and it will be trivial to deploy cloud 400gpus for an hour to run the load your local 4gpu workstation would spend 100 hours on, but we are definitely not there yet. We are working on a packaging format called snappy (http://www.snapcraft.io http://www.snapcraft.io) and starting to talk to TF guys about packaging TF with it (kubernetes is already packaged as a snap) - hopefully this will take some pain away once it's working