5 ms·

Tensorflow is actually pretty slow and problematic on large clusters outside the Google Cloud. Probably because that's not what it was designed for. For Java/S

by vonnik 10y ago

Tensorflow is actually pretty slow and problematic on large clusters outside the Google Cloud. Probably because that's not what it was designed for. For Java/Scala people, Deeplearning4j has a pretty sophisticated Spark + GPUs setup: http://deeplearning4j.org/gpu http://deeplearning4j.org/gpu http://deeplearning4j.org/spark http://deeplearning4j.org/spark http://deeplearning4j.org/spark-gpus http://deeplearning4j.org/spark-gpus [Disclosure: I help create DL4J, and it's supported by my startup, Skymind.]

minimaxir 10y ago
You should disclose that Deeplearning4j is your startup.
- vonnik 10y ago
  Done!
rryan 10y ago
> Probably because that's not what it was designed for. TF was explicitly designed with distributed training in mind (their initial whitepaper and the DistBelief paper that came before it make this clear) -- I don't know how you came to this conclusion. Usually when people say TF is slow, it turns out they've introduced a serious bottleneck somewhere.
- vonnik 10y ago
  The environment for your large clusters is almost certainly different from everyone using TF outside of Google. I'm speaking about the problems they'll run into.
  - nl 10y ago
    I'm speaking about the problems they'll run into. What are these mythical problems you speak of? I'd hear to hear some specifics, because I haven't hit them yet.
- turinturambar 10y ago
  Both points are true: TF is obviously designed for scalable, distributed training, but it also heavily tied to Google's compute infrastructure (less so all the time, of course, but now it's also being closely tied to Google Cloud). So while I disagree with my colleague* that TF is "slow" or "not designed for distributed training," I support the slightly different (and implicit) argument that there are some settings (often in enterprise, I am learning) where it might not be as good a fit as other frameworks (e.g., DL4J, Caffe, whatever). * Disclosure: I work with Skymind and contribute to DL4J, and I also use TensorFlow/Theano/keras heavily in my PhD research. I am an equal opportunity framework guy. ;)
  - Chronic9q 10y ago
    To all these Skymind kool-aid drinkers, I won't bother arguing with you. I'll let the tensorflow vs DL4J usage numbers tell the story. Spoiler: Tensorflow wins.
    - vonnik 10y ago
      Spoiler: 95% of them are Udacity students without experience or budgets.
tlb 10y ago
How does DL4J training scale across >8 GPUs?
- dragandj 10y ago
  I don't know, but I am curious about how many people, in percentages, outside Google and Facebook and the likes, need to scale their models to more than 8 GPUS?
  - euyyn 10y ago
    If you're paying for CPU/GPU hours, the more you parallelize the faster you get your result for the same money. (Of course till you hit the parallelization limit of your network, but NNs are very parallelizable in general). And the larger your training dataset, the better your results.
    - dragandj 10y ago
      I am aware of that. The question was not whether more than 8 GPUs would be useful in ideal circumstances, it was about how many people actually use that functionality with other frameworks other than DL4J?
      - euyyn 10y ago
        It'd be a nice statistic to know. Could be dangerous too, as in the infamous "640 kilobytes should be enough for everyone".
  - tlb 10y ago
    Many of the models people are building here, such as generative image models, take a few days to train (say, 100 hours) on our 4 GPU boxes. Research would be faster if we could train on 400 GPUS in one hour, but the communication bandwidth required makes it hard to scale.
    - deleted 10y ago
      [deleted]
nl 10y ago
As someone who runs a production Spark cluster, has a multi-node TF setup and had played with DL4J I'd say this is more untrue than true. Training TF on a cluster is hard because you need to write your model to be cluster-aware. Apart from that it works pretty well. From my understanding, DL4J can use Spark to coordinate training multiple models in parallel. This is interesting, but not the same thing at all. Also TF on a cluster is much simpler to get running than Spark+DL4J.
- vonnik 10y ago
  Curious: When did you play with DL4J and what kind of cluster are you running TF on? You're right to say those are two separate things, so let's compare apples to apples.
- agibsonccc 10y ago
  I'm assuming you're using pyspark? Also no - we use data paralellism. I'd maybe check out our new spark page we put up that explains how parameter averaging works: http://deeplearning4j.org/spark http://deeplearning4j.org/spark Basically we train on raw spark partitions. Basically because of the data parallelism we can also use hardware accel for our spark jobs, which means for training we'll train on gpus when present. We use a spark master for the "parameter server" to handle communication of the parameter updates with multiple ways of controlling the averaging threshold per iteration among other things to minimize communication. This allows us to do neat things like using cudnn on a spark cluster. I'll assume you're not interested in too many details there but happy to expand if needed. Edit: When you "played" with us - I'm assuming you just cloned our examples and ran us on the desktop? Likely more than a year ago before our c++ rewrite? I'd be curious to see if you ran us on spark as well. Running jvm based stuff is completely different than running a python job. I know jvm based stuff is a bit harder to get up and running because of the dependency clashes (with spark being jvm based etc) If you're ever in scala land I encourage you to take a look at our keras port to scala when its out. We'll also be running reinforcement learning workloads on spark as well.
  - agibsonccc 10y ago
    Not sure where the downvote came from - maybe I can explain a bit. If it was from my last part - I'm actually asking for clarification because we've often discovered this when people say they've tried us. A lot of folks don't spend too much time on a framework and I don't blame them. They are looking for a tool to get the job done and move on 99% of the time. That's more than fair. The bulk of these folks also tend to be people who are mildly curious coming from python, where most deep learning practitioners are. That being said: I was serious about each node being independent for spark. Our linear algebra library runs as a spark job on each executor/slave node. We've built in our own resource management via javacpp that handles memory allocation for the cuda buffers as well as cudnn. The driver for spark then tracks the coefficients across spark rdd partitions which allows us to synchronize updates. Look out for a blog post from us on parallelforall (nvidia's blog) here soon that explains how this works.

by vonnik 10y ago

minimaxir 10y ago

You should disclose that Deeplearning4j is your startup.

vonnik 10y ago
Done!

rryan 10y ago

> Probably because that's not what it was designed for. TF was explicitly designed with distributed training in mind (their initial whitepaper and the DistBelief paper that came before it make this clear) -- I don't know how you came to this conclusion. Usually when people say TF is slow, it turns out they've introduced a serious bottleneck somewhere.

vonnik 10y ago
The environment for your large clusters is almost certainly different from everyone using TF outside of Google. I'm speaking about the problems they'll run into.
- nl 10y ago
  I'm speaking about the problems they'll run into. What are these mythical problems you speak of? I'd hear to hear some specifics, because I haven't hit them yet.
turinturambar 10y ago
Both points are true: TF is obviously designed for scalable, distributed training, but it also heavily tied to Google's compute infrastructure (less so all the time, of course, but now it's also being closely tied to Google Cloud). So while I disagree with my colleague* that TF is "slow" or "not designed for distributed training," I support the slightly different (and implicit) argument that there are some settings (often in enterprise, I am learning) where it might not be as good a fit as other frameworks (e.g., DL4J, Caffe, whatever). * Disclosure: I work with Skymind and contribute to DL4J, and I also use TensorFlow/Theano/keras heavily in my PhD research. I am an equal opportunity framework guy. ;)
- Chronic9q 10y ago
  To all these Skymind kool-aid drinkers, I won't bother arguing with you. I'll let the tensorflow vs DL4J usage numbers tell the story. Spoiler: Tensorflow wins.
  - vonnik 10y ago
    Spoiler: 95% of them are Udacity students without experience or budgets.

tlb 10y ago

How does DL4J training scale across >8 GPUs?

dragandj 10y ago
I don't know, but I am curious about how many people, in percentages, outside Google and Facebook and the likes, need to scale their models to more than 8 GPUS?
- euyyn 10y ago
  If you're paying for CPU/GPU hours, the more you parallelize the faster you get your result for the same money. (Of course till you hit the parallelization limit of your network, but NNs are very parallelizable in general). And the larger your training dataset, the better your results.
  - dragandj 10y ago
    I am aware of that. The question was not whether more than 8 GPUs would be useful in ideal circumstances, it was about how many people actually use that functionality with other frameworks other than DL4J?
    - euyyn 10y ago
      It'd be a nice statistic to know. Could be dangerous too, as in the infamous "640 kilobytes should be enough for everyone".
- tlb 10y ago
  Many of the models people are building here, such as generative image models, take a few days to train (say, 100 hours) on our 4 GPU boxes. Research would be faster if we could train on 400 GPUS in one hour, but the communication bandwidth required makes it hard to scale.
  - deleted 10y ago
    [deleted]

nl 10y ago

As someone who runs a production Spark cluster, has a multi-node TF setup and had played with DL4J I'd say this is more untrue than true. Training TF on a cluster is hard because you need to write your model to be cluster-aware. Apart from that it works pretty well. From my understanding, DL4J can use Spark to coordinate training multiple models in parallel. This is interesting, but not the same thing at all. Also TF on a cluster is much simpler to get running than Spark+DL4J.

vonnik 10y ago
Curious: When did you play with DL4J and what kind of cluster are you running TF on? You're right to say those are two separate things, so let's compare apples to apples.
agibsonccc 10y ago
I'm assuming you're using pyspark? Also no - we use data paralellism. I'd maybe check out our new spark page we put up that explains how parameter averaging works: http://deeplearning4j.org/spark http://deeplearning4j.org/spark Basically we train on raw spark partitions. Basically because of the data parallelism we can also use hardware accel for our spark jobs, which means for training we'll train on gpus when present. We use a spark master for the "parameter server" to handle communication of the parameter updates with multiple ways of controlling the averaging threshold per iteration among other things to minimize communication. This allows us to do neat things like using cudnn on a spark cluster. I'll assume you're not interested in too many details there but happy to expand if needed. Edit: When you "played" with us - I'm assuming you just cloned our examples and ran us on the desktop? Likely more than a year ago before our c++ rewrite? I'd be curious to see if you ran us on spark as well. Running jvm based stuff is completely different than running a python job. I know jvm based stuff is a bit harder to get up and running because of the dependency clashes (with spark being jvm based etc) If you're ever in scala land I encourage you to take a look at our keras port to scala when its out. We'll also be running reinforcement learning workloads on spark as well.
- agibsonccc 10y ago
  Not sure where the downvote came from - maybe I can explain a bit. If it was from my last part - I'm actually asking for clarification because we've often discovered this when people say they've tried us. A lot of folks don't spend too much time on a framework and I don't blame them. They are looking for a tool to get the job done and move on 99% of the time. That's more than fair. The bulk of these folks also tend to be people who are mildly curious coming from python, where most deep learning practitioners are. That being said: I was serious about each node being independent for spark. Our linear algebra library runs as a spark job on each executor/slave node. We've built in our own resource management via javacpp that handles memory allocation for the cuda buffers as well as cudnn. The driver for spark then tracks the coefficients across spark rdd partitions which allows us to synchronize updates. Look out for a blog post from us on parallelforall (nvidia's blog) here soon that explains how this works.