4 ms·

I don't know, but I am curious about how many people, in percentages, outside Google and Facebook and the likes, need to scale their models to more than 8 GPUS?

euyyn 10y ago
If you're paying for CPU/GPU hours, the more you parallelize the faster you get your result for the same money. (Of course till you hit the parallelization limit of your network, but NNs are very parallelizable in general). And the larger your training dataset, the better your results.
- dragandj 10y ago
  I am aware of that. The question was not whether more than 8 GPUs would be useful in ideal circumstances, it was about how many people actually use that functionality with other frameworks other than DL4J?
  - euyyn 10y ago
    It'd be a nice statistic to know. Could be dangerous too, as in the infamous "640 kilobytes should be enough for everyone".
tlb 10y ago
Many of the models people are building here, such as generative image models, take a few days to train (say, 100 hours) on our 4 GPU boxes. Research would be faster if we could train on 400 GPUS in one hour, but the communication bandwidth required makes it hard to scale.
- deleted 10y ago
  [deleted]
- dragandj 10y ago
  I am aware of that. The question was not whether more than 8 GPUs would be useful in ideal circumstances, it was about how many people actually use that functionality with other frameworks other than DL4J? BTW I do not have any connection with DL4J, nor do I use it.
- emcq 10y ago
  What is being shared between GPUs? Training data is easy to duplicate and share nothing. Large models with shared weights get tricky but less frequent asynchronous updates with schemes like hogwild seem to work with SGD. I believe TF has support for this too. It won't scale linearly but might be good enough. There's some excitement about synthetic gradients to allow less communication and further parallelism. The hpc community has certainly leveraged 400 GPU clusters. It seems like a fun problem, and if you want to focus the resources, there isn't anything insurmountable about utilizing 400 GPUs :)
  - dharma1 10y ago
    The DeepMind paper on synthetic gradients is super interesting for training in clusters. I hope it gets built into TF soon. Also really hope there will be more options for GPU hardware on public clouds. Eventually the pieces will come together and it will be trivial to deploy cloud 400gpus for an hour to run the load your local 4gpu workstation would spend 100 hours on, but we are definitely not there yet. We are working on a packaging format called snappy (http://www.snapcraft.io http://www.snapcraft.io) and starting to talk to TF guys about packaging TF with it (kubernetes is already packaged as a snap) - hopefully this will take some pain away once it's working

euyyn 10y ago

If you're paying for CPU/GPU hours, the more you parallelize the faster you get your result for the same money. (Of course till you hit the parallelization limit of your network, but NNs are very parallelizable in general). And the larger your training dataset, the better your results.

dragandj 10y ago
I am aware of that. The question was not whether more than 8 GPUs would be useful in ideal circumstances, it was about how many people actually use that functionality with other frameworks other than DL4J?
- euyyn 10y ago
  It'd be a nice statistic to know. Could be dangerous too, as in the infamous "640 kilobytes should be enough for everyone".

tlb 10y ago

Many of the models people are building here, such as generative image models, take a few days to train (say, 100 hours) on our 4 GPU boxes. Research would be faster if we could train on 400 GPUS in one hour, but the communication bandwidth required makes it hard to scale.

deleted 10y ago
[deleted]
dragandj 10y ago
I am aware of that. The question was not whether more than 8 GPUs would be useful in ideal circumstances, it was about how many people actually use that functionality with other frameworks other than DL4J? BTW I do not have any connection with DL4J, nor do I use it.
emcq 10y ago
What is being shared between GPUs? Training data is easy to duplicate and share nothing. Large models with shared weights get tricky but less frequent asynchronous updates with schemes like hogwild seem to work with SGD. I believe TF has support for this too. It won't scale linearly but might be good enough. There's some excitement about synthetic gradients to allow less communication and further parallelism. The hpc community has certainly leveraged 400 GPU clusters. It seems like a fun problem, and if you want to focus the resources, there isn't anything insurmountable about utilizing 400 GPUs :)
- dharma1 10y ago
  The DeepMind paper on synthetic gradients is super interesting for training in clusters. I hope it gets built into TF soon. Also really hope there will be more options for GPU hardware on public clouds. Eventually the pieces will come together and it will be trivial to deploy cloud 400gpus for an hour to run the load your local 4gpu workstation would spend 100 hours on, but we are definitely not there yet. We are working on a packaging format called snappy (http://www.snapcraft.io http://www.snapcraft.io) and starting to talk to TF guys about packaging TF with it (kubernetes is already packaged as a snap) - hopefully this will take some pain away once it's working