4 ms·

I'm going to say something unpopular, but horizontally-scaled deep learning is overkill for most applications. Can anyone here present a use case where they ha

I'm going to say something unpopular, but horizontally-scaled deep learning is overkill for most applications. Can anyone here present a use case where they have personally needed horizontal scaling because a Titan X couldn't fit what they were trying to do? It's the thing I hear talked about the most and used the least. The biggest misunderstanding I've heard is "I have petabytes of data so I need multi-GPU", but NNs are trained on batches of data, so it doesn't matter. It's the model that must fit, but 12gb can fit a big model. The whole "who can build the biggest net" is like building the fastest car: it's for show, the slower ones are way more cost effective and get you there just as well. Keras is my present deep learning library recommendation--it's like torch 7 but in python and theano. Multi-GPU is roadmap because there's a lot more going on in deep learning that acts as a bigger differentiator than horizontal scaling. For the record, when I say scaling, I'm not talking about having multiple machines serve multiple requests (that's a good reason to purchase more machines), I'm talking about spreading your gradient computations and/or model weights across machines or GPUs.

freerobby 11y ago
Would image training for video recognition be a good example of this?
- dave_sullivan 11y ago
  It's a good example of something people think they need multi-GPU for, but no, I don't think it's needed. For the same reason as "I've got petabytes of data".
- yjxiong 11y ago
  Our experiments might support this. With 4 or 8 titan X, training time is shortened from more than one week to one or two days. code available at https://github.com/yjxiong/caffe https://github.com/yjxiong/caffe
  - dave_sullivan 11y ago
    Note: data parallel vs model parallel. Data parallel is easy to implement and leads to linear training time speed ups, as per 1 week to 2 days with 4x hardware. But not bigger models. Model parallelism leads to bigger models and is what I've been referring to. It is overkill and doesn't work well anyway. Frameworks should not be expected to support it because there are more interesting topics in deep learning they could support instead. Better research/methods will come out at some point, at which point this calculus will change, but not yet! Today it is the very definition of premature optimization in nearly all cases.
kastnerkyle 11y ago
For big LSTMs and long-ish sequences, the intermediate gradients can take up a huge amount of memory - often more than the model parameters themselves. In my experience it is mostly big LSTMs that need the 12GB+ GPUs. You can reduce the batch size to help this a bit, or train using trucated BPTT but RNN training is already a slow, sequential business. Of course, there are no clear wins (generally, losses) in computation by scaling RNNs horizontally on GPUs - but sometimes you really do need more than 12GB.
- dave_sullivan 11y ago
  All true things. But at the heart of the issue: No clear wins in horizontal scaling Reduce batch size Use truncated back prop *search for better hyper params* I usually do 2-4. After those, have you really seen scaling result in significant accuracy gains? And what percent of the time is that necessary? Genuinely interested--and you guys rock btw!
  - kastnerkyle 11y ago
    One case where I see 1 as being necessary is the softmax size / vocabulary boost in the seq2seq paper (8 GPUs IIRc, and 4 were dedicated to a softmax!). There are other ways to handle this (hierarchical softmax, sampled softmax, skip-thoughts trick of using word2vec vocabulary) but every time someone figures out how to have a larger vocabulary, neural machine translation results seem to improve. In general, 1 is a last resort for me - but maybe this is due to current tooling and availability of hardware as much as anything?
kajecounterhack 11y ago
> horizontally-scaled deep learning is overkill for most applications It's difficult to say what "most applications are" but many algos currently seem bottlenecked by how fast you can load data into your GFX card (hint: it's slow). So what you're saying about batching is correct, but it does matter.
- dave_sullivan 11y ago
  > many algos currently seem bottlenecked by how fast you can load data into your GFX card All of them are, but that's a PCI-e issue and horizontal scaling doesn't fix that (unless using nvlink or similar afaik, but then you face the fact that current horizontal scaling schemes aren't very effective at increasing model accuracy anyway) > It's difficult to say what "most applications are" Nevermind "most applications"--so far, all I've heard is one, that being the absolute bleeding edge of RNN research, assuming you're using a huge softmax instead of an alternative. My point remains: Multi-GPU is way down the list when it comes to features a DL framework should have. Because very very few people need it. kajecounterhack, do you use multi-GPU for your DL work? If so, how often?
  - varelse 11y ago
    >do you use multi-GPU for your DL work? If so, how often? I'm using it right now, to great effect. I can't really say what for, but methinks I'll be using it every hour of every day for the foreseeable future.
varelse 11y ago
I suspect this is a just-so-tale because the capability is so new and the GPU programming skill required to implement it efficiently is so rare. TLDR: just say no to parameter server (most of the time). But IMO the real obstacle to horizontal scaling is the communication between servers not the usefulness of doing so. Within a server, one can pack 8 TitanX/M40 GPUs with high speed 13 GB/s P2P communication between them (and up to 16 GPUs in various unproven science project servers). That's ~50 TFLOPs and 96 GB in a box. That rocks. Just ask the guys at mindori (if they ever ship that is). http://www.mindori.com http://www.mindori.com But between servers lies a sippy straw of 100 Gb/s Infiniband at best or worst-case, ~1 Gb/s on AWS GPU servers with freaky nearest neighbor weather. If you can't make efficient use of 8 GPUs in a single box, I agree, don't bother breaking out to the next server. That said, frameworks like mxnet have opened the floodgates to experimenting with larger models and more distributed training algorithms. Time will tell if this pans out. But 2 years ago, Andrew Ng's group showed 12 GTX 680s distributed across 3 servers kicking the crap out of Google Brain. I expect more of this, not less, in the near future.
moultano 11y ago
>Can anyone here present a use case where they have personally needed horizontal scaling The embeddings for text models with a real world vocabulary can get very very large, especially if multiple languages are involved.

freerobby 11y ago

Would image training for video recognition be a good example of this?

dave_sullivan 11y ago
It's a good example of something people think they need multi-GPU for, but no, I don't think it's needed. For the same reason as "I've got petabytes of data".
yjxiong 11y ago
Our experiments might support this. With 4 or 8 titan X, training time is shortened from more than one week to one or two days. code available at https://github.com/yjxiong/caffe https://github.com/yjxiong/caffe
- dave_sullivan 11y ago
  Note: data parallel vs model parallel. Data parallel is easy to implement and leads to linear training time speed ups, as per 1 week to 2 days with 4x hardware. But not bigger models. Model parallelism leads to bigger models and is what I've been referring to. It is overkill and doesn't work well anyway. Frameworks should not be expected to support it because there are more interesting topics in deep learning they could support instead. Better research/methods will come out at some point, at which point this calculus will change, but not yet! Today it is the very definition of premature optimization in nearly all cases.

kastnerkyle 11y ago

For big LSTMs and long-ish sequences, the intermediate gradients can take up a huge amount of memory - often more than the model parameters themselves. In my experience it is mostly big LSTMs that need the 12GB+ GPUs. You can reduce the batch size to help this a bit, or train using trucated BPTT but RNN training is already a slow, sequential business. Of course, there are no clear wins (generally, losses) in computation by scaling RNNs horizontally on GPUs - but sometimes you really do need more than 12GB.

dave_sullivan 11y ago
All true things. But at the heart of the issue: No clear wins in horizontal scaling Reduce batch size Use truncated back prop *search for better hyper params* I usually do 2-4. After those, have you really seen scaling result in significant accuracy gains? And what percent of the time is that necessary? Genuinely interested--and you guys rock btw!
- kastnerkyle 11y ago
  One case where I see 1 as being necessary is the softmax size / vocabulary boost in the seq2seq paper (8 GPUs IIRc, and 4 were dedicated to a softmax!). There are other ways to handle this (hierarchical softmax, sampled softmax, skip-thoughts trick of using word2vec vocabulary) but every time someone figures out how to have a larger vocabulary, neural machine translation results seem to improve. In general, 1 is a last resort for me - but maybe this is due to current tooling and availability of hardware as much as anything?

kajecounterhack 11y ago

> horizontally-scaled deep learning is overkill for most applications It's difficult to say what "most applications are" but many algos currently seem bottlenecked by how fast you can load data into your GFX card (hint: it's slow). So what you're saying about batching is correct, but it does matter.

dave_sullivan 11y ago
> many algos currently seem bottlenecked by how fast you can load data into your GFX card All of them are, but that's a PCI-e issue and horizontal scaling doesn't fix that (unless using nvlink or similar afaik, but then you face the fact that current horizontal scaling schemes aren't very effective at increasing model accuracy anyway) > It's difficult to say what "most applications are" Nevermind "most applications"--so far, all I've heard is one, that being the absolute bleeding edge of RNN research, assuming you're using a huge softmax instead of an alternative. My point remains: Multi-GPU is way down the list when it comes to features a DL framework should have. Because very very few people need it. kajecounterhack, do you use multi-GPU for your DL work? If so, how often?
- varelse 11y ago
  >do you use multi-GPU for your DL work? If so, how often? I'm using it right now, to great effect. I can't really say what for, but methinks I'll be using it every hour of every day for the foreseeable future.

varelse 11y ago

I suspect this is a just-so-tale because the capability is so new and the GPU programming skill required to implement it efficiently is so rare. TLDR: just say no to parameter server (most of the time). But IMO the real obstacle to horizontal scaling is the communication between servers not the usefulness of doing so. Within a server, one can pack 8 TitanX/M40 GPUs with high speed 13 GB/s P2P communication between them (and up to 16 GPUs in various unproven science project servers). That's ~50 TFLOPs and 96 GB in a box. That rocks. Just ask the guys at mindori (if they ever ship that is). http://www.mindori.com http://www.mindori.com But between servers lies a sippy straw of 100 Gb/s Infiniband at best or worst-case, ~1 Gb/s on AWS GPU servers with freaky nearest neighbor weather. If you can't make efficient use of 8 GPUs in a single box, I agree, don't bother breaking out to the next server. That said, frameworks like mxnet have opened the floodgates to experimenting with larger models and more distributed training algorithms. Time will tell if this pans out. But 2 years ago, Andrew Ng's group showed 12 GTX 680s distributed across 3 servers kicking the crap out of Google Brain. I expect more of this, not less, in the near future.

moultano 11y ago

>Can anyone here present a use case where they have personally needed horizontal scaling The embeddings for text models with a real world vocabulary can get very very large, especially if multiple languages are involved.