6 ms·
From Google's perspective it is probably more about how TensorFlow scales out horizontally. If a researcher fires off a Borg run (or whatever they use now) and
by mark_l_watson 11y ago
From Google's perspective it is probably more about how TensorFlow scales out horizontally. If a researcher fires off a Borg run (or whatever they use now) and the job takes a few thousand CPUs, no problem, at least for research.
They must have better optimization a for running in production, such as in place operations.
- mtw 11y agoThe rest of us don't have "a few thousand CPUs" though. Seen this way, TensorFlow is largely inaccessible and makes more sense to focus on existing librairies optimized for single cpus
- johanneskanybal 11y agoI don't think you understand what this library is for. You can pick up a graphic card for $100 and voila now you got 1000's of cores. To your defense the comment you answered seems to have mixed a few random things together too.
- varelse 11y agoTensorFlow is currently a single-server solution pthreads app. But assuming you're serious and not dabbling, you can build a ~27 TFLOP server for <$7K in parts or buy it assembled for ~$9K. http://exxactcorp.com/index.php/solution/solu_detail/225 http://exxactcorp.com/index.php/solution/solu_detail/225 Or if you just hate the thought of money burning a hole in your pocket, pay NVIDIA >$15K for the same thing with a pretty NVIDIA logo: https://developer.nvidia.com/devbox https://developer.nvidia.com/devbox If you're dabbling, just go buy a $1000 TitanX and you're instantly a minor deep learning superpower.
- zellyn 11y agoThis. Even if it takes 10,000 networked cores to replace the ultimate souped-up single machine loaded with GPUs, you can go from 10,000 to 20,000.
- andmarios 11y agoI wonder if this is the reason I can't get a preemptible VM from GCE to stay up for more than one hour in the last days.
- argonaut 11y agoWhich would be great, except for the fact that Google has not open-sourced the distributed version of TensorFlow. So this is just PR/speculation.
- aljones 11y agoSo how horizontally scalable is it?
- sanxiyn 11y agoAs far as I know, TensorFlow's scaling behavior has not been published. For its predecessor DistBelief, here are some quotes from NIPS 2012 paper: "The moderately sized speech model runs fastest on 8 machines, computing 2.2x faster than using a single machine. (Models were configured to use no more than 20 cores per machine.) Partitioning the model on more than 8 machines actually slows training, as network overhead starts to dominate in the fully-connected network structure and there is less work for each machine to perform with more partitions." ("The moderately sized" model here has 42 million model parameters. Check the paper for details.) "In contrast, the much larger, locally-connected image models can benefit from using many more machines per model replica. The largest model, with 1.7 billion parameters benefits the most, giving a speedup of more than 12x using 81 machines. For these large models using more machines continues to increase speed, but with diminishing returns." http://research.google.com/pubs/pub40565.html http://research.google.com/pubs/pub40565.html
- dave_sullivan 11y agoI'm going to say something unpopular, but horizontally-scaled deep learning is overkill for most applications. Can anyone here present a use case where they have personally needed horizontal scaling because a Titan X couldn't fit what they were trying to do? It's the thing I hear talked about the most and used the least. The biggest misunderstanding I've heard is "I have petabytes of data so I need multi-GPU", but NNs are trained on batches of data, so it doesn't matter. It's the model that must fit, but 12gb can fit a big model. The whole "who can build the biggest net" is like building the fastest car: it's for show, the slower ones are way more cost effective and get you there just as well. Keras is my present deep learning library recommendation--it's like torch 7 but in python and theano. Multi-GPU is roadmap because there's a lot more going on in deep learning that acts as a bigger differentiator than horizontal scaling. For the record, when I say scaling, I'm not talking about having multiple machines serve multiple requests (that's a good reason to purchase more machines), I'm talking about spreading your gradient computations and/or model weights across machines or GPUs.
- freerobby 11y agoWould image training for video recognition be a good example of this?
- dave_sullivan 11y agoIt's a good example of something people think they need multi-GPU for, but no, I don't think it's needed. For the same reason as "I've got petabytes of data".
- yjxiong 11y agoOur experiments might support this. With 4 or 8 titan X, training time is shortened from more than one week to one or two days. code available at https://github.com/yjxiong/caffe https://github.com/yjxiong/caffe
- dave_sullivan 11y agoNote: data parallel vs model parallel. Data parallel is easy to implement and leads to linear training time speed ups, as per 1 week to 2 days with 4x hardware. But not bigger models. Model parallelism leads to bigger models and is what I've been referring to. It is overkill and doesn't work well anyway. Frameworks should not be expected to support it because there are more interesting topics in deep learning they could support instead. Better research/methods will come out at some point, at which point this calculus will change, but not yet! Today it is the very definition of premature optimization in nearly all cases.