4 ms·

How does DL4J training scale across >8 GPUs?

by tlb 10y ago

dragandj 10y ago
I don't know, but I am curious about how many people, in percentages, outside Google and Facebook and the likes, need to scale their models to more than 8 GPUS?
- euyyn 10y ago
  If you're paying for CPU/GPU hours, the more you parallelize the faster you get your result for the same money. (Of course till you hit the parallelization limit of your network, but NNs are very parallelizable in general). And the larger your training dataset, the better your results.
  - dragandj 10y ago
    I am aware of that. The question was not whether more than 8 GPUs would be useful in ideal circumstances, it was about how many people actually use that functionality with other frameworks other than DL4J?
    - euyyn 10y ago
      It'd be a nice statistic to know. Could be dangerous too, as in the infamous "640 kilobytes should be enough for everyone".
- tlb 10y ago
  Many of the models people are building here, such as generative image models, take a few days to train (say, 100 hours) on our 4 GPU boxes. Research would be faster if we could train on 400 GPUS in one hour, but the communication bandwidth required makes it hard to scale.
  - deleted 10y ago
    [deleted]
  - dragandj 10y ago
    I am aware of that. The question was not whether more than 8 GPUs would be useful in ideal circumstances, it was about how many people actually use that functionality with other frameworks other than DL4J? BTW I do not have any connection with DL4J, nor do I use it.
  - emcq 10y ago
    What is being shared between GPUs? Training data is easy to duplicate and share nothing. Large models with shared weights get tricky but less frequent asynchronous updates with schemes like hogwild seem to work with SGD. I believe TF has support for this too. It won't scale linearly but might be good enough. There's some excitement about synthetic gradients to allow less communication and further parallelism. The hpc community has certainly leveraged 400 GPU clusters. It seems like a fun problem, and if you want to focus the resources, there isn't anything insurmountable about utilizing 400 GPUs :)
    - dharma1 10y ago
      The DeepMind paper on synthetic gradients is super interesting for training in clusters. I hope it gets built into TF soon. Also really hope there will be more options for GPU hardware on public clouds. Eventually the pieces will come together and it will be trivial to deploy cloud 400gpus for an hour to run the load your local 4gpu workstation would spend 100 hours on, but we are definitely not there yet. We are working on a packaging format called snappy (http://www.snapcraft.io http://www.snapcraft.io) and starting to talk to TF guys about packaging TF with it (kubernetes is already packaged as a snap) - hopefully this will take some pain away once it's working
vonnik 10y ago
Hi Trevor - Nice to see you here. :) For data parallelism, we have a simple wrapper that covers as many GPUs as you want in one box. For more than one box, we use Spark, and the integration is explained here: http://deeplearning4j.org/spark http://deeplearning4j.org/spark The description for multi-GPUs is through this link: http://deeplearning4j.org/gpu http://deeplearning4j.org/gpu under the subhead "Multi-GPU data parallelism". The code is here: https://github.com/deeplearning4j/deeplearning4j/blob/77b836cd7daba5fbd4c7a77d341d489cf6e9a220/deeplearning4j-core/src/main/java/org/deeplearning4j/parallelism/ParallelWrapper.java https://github.com/deeplearning4j/deeplearning4j/blob/77b836... We're almost done testing on DGX's now, like the one you have at OpenAI, and once we work a bit more with RDMA-enabled hardware, we'll go Sparkless for that. That's how we're hoping to get academic HW acceleration into production environments. In September, we plan to come out with a Scala API inspired by Keras/Torch, which will also share a neural net config file with Keras and allow model imports into DL4J.
agibsonccc 10y ago
Hi Trevor, We actually are going to be doing some stuff with IBM/NVLink as well as some other neat things I can't announce. We'll be able to benchmark on this front similar to you guys though :). We'll be doing RDMA for this and plan on writing the code to match that using spark for orchestration and data storage/loading. Right now the main thing we do is data parallelism on partitions of data with intermittent averaging with data being trained on various spark partitions. Other than that, we have multi gpu settings. We've made it pretty configurable though: http://deeplearning4j.org/gpu http://deeplearning4j.org/gpu Admittedly, we'll continue to do more work in this area though. So far fp16 has been pretty nice though :).

How does DL4J training scale across >8 GPUs?

by tlb 10y ago

How does DL4J training scale across >8 GPUs?

I don't know, but I am curious about how many people, in percentages, outside Google and Facebook and the likes, need to scale their models to more than 8 GPUS?

euyyn 10y ago
If you're paying for CPU/GPU hours, the more you parallelize the faster you get your result for the same money. (Of course till you hit the parallelization limit of your network, but NNs are very parallelizable in general). And the larger your training dataset, the better your results.
- dragandj 10y ago
  I am aware of that. The question was not whether more than 8 GPUs would be useful in ideal circumstances, it was about how many people actually use that functionality with other frameworks other than DL4J?
  - euyyn 10y ago
    It'd be a nice statistic to know. Could be dangerous too, as in the infamous "640 kilobytes should be enough for everyone".
tlb 10y ago
Many of the models people are building here, such as generative image models, take a few days to train (say, 100 hours) on our 4 GPU boxes. Research would be faster if we could train on 400 GPUS in one hour, but the communication bandwidth required makes it hard to scale.
- deleted 10y ago
  [deleted]
- dragandj 10y ago
  I am aware of that. The question was not whether more than 8 GPUs would be useful in ideal circumstances, it was about how many people actually use that functionality with other frameworks other than DL4J? BTW I do not have any connection with DL4J, nor do I use it.
- emcq 10y ago
  What is being shared between GPUs? Training data is easy to duplicate and share nothing. Large models with shared weights get tricky but less frequent asynchronous updates with schemes like hogwild seem to work with SGD. I believe TF has support for this too. It won't scale linearly but might be good enough. There's some excitement about synthetic gradients to allow less communication and further parallelism. The hpc community has certainly leveraged 400 GPU clusters. It seems like a fun problem, and if you want to focus the resources, there isn't anything insurmountable about utilizing 400 GPUs :)
  - dharma1 10y ago
    The DeepMind paper on synthetic gradients is super interesting for training in clusters. I hope it gets built into TF soon. Also really hope there will be more options for GPU hardware on public clouds. Eventually the pieces will come together and it will be trivial to deploy cloud 400gpus for an hour to run the load your local 4gpu workstation would spend 100 hours on, but we are definitely not there yet. We are working on a packaging format called snappy (http://www.snapcraft.io http://www.snapcraft.io) and starting to talk to TF guys about packaging TF with it (kubernetes is already packaged as a snap) - hopefully this will take some pain away once it's working

vonnik 10y ago

Hi Trevor - Nice to see you here. :) For data parallelism, we have a simple wrapper that covers as many GPUs as you want in one box. For more than one box, we use Spark, and the integration is explained here: http://deeplearning4j.org/spark http://deeplearning4j.org/spark The description for multi-GPUs is through this link: http://deeplearning4j.org/gpu http://deeplearning4j.org/gpu under the subhead "Multi-GPU data parallelism". The code is here: https://github.com/deeplearning4j/deeplearning4j/blob/77b836cd7daba5fbd4c7a77d341d489cf6e9a220/deeplearning4j-core/src/main/java/org/deeplearning4j/parallelism/ParallelWrapper.java https://github.com/deeplearning4j/deeplearning4j/blob/77b836... We're almost done testing on DGX's now, like the one you have at OpenAI, and once we work a bit more with RDMA-enabled hardware, we'll go Sparkless for that. That's how we're hoping to get academic HW acceleration into production environments. In September, we plan to come out with a Scala API inspired by Keras/Torch, which will also share a neural net config file with Keras and allow model imports into DL4J.

agibsonccc 10y ago

Hi Trevor, We actually are going to be doing some stuff with IBM/NVLink as well as some other neat things I can't announce. We'll be able to benchmark on this front similar to you guys though :). We'll be doing RDMA for this and plan on writing the code to match that using spark for orchestration and data storage/loading. Right now the main thing we do is data parallelism on partitions of data with intermittent averaging with data being trained on various spark partitions. Other than that, we have multi gpu settings. We've made it pretty configurable though: http://deeplearning4j.org/gpu http://deeplearning4j.org/gpu Admittedly, we'll continue to do more work in this area though. So far fp16 has been pretty nice though :).