10 ms·
TensorFlow Benchmarks
- vonnik 11y agoIt takes 4x as long as CuDNN on AlexNet due to lack of in-place ops. What is up with that?
- mtgx 11y agoIt's almost like Google wanted everyone to use slow obsolete software and keep the really good stuff for itself, while still making it look like they're doing a great thing for the community.
- tfgg 11y agoI think it makes more sense that: a) It's Google, they can throw hardware at a problem such as out of memory b) TensorFlow makes methods development so much easier that it's worth the loss of performance c) It's early days and the compute graph scheduler has lots of opportunities, and is designed, for optimization, and in a more flexible fashion than other frameworks. When I work on developing methods for scientific code, I worry more about whether the code is bug free/easy to understand and that it's giving the right answer. I don't usually worry about performance unless I'm actually not able to run things. Especially since when developing stuff you waste way more time on runs with bugs - I dread to think what my (published / unpublished) CPU hour ratio is. If the new approach allows less buggy implementations then that's a resource win.
- IanCal 11y ago> b) TensorFlow makes methods development so much easier that it's worth the loss of performance Indeed, if TensorFlow means I can try out an idea with 1 day of coding and 2 days of training rather than 3 days of coding and 1 day of training then I can spend a day drinking cocktails and reading books and still be finished sooner. Based on the tutorials it seems like I'd be able to pretty quickly build a translation pipeline, and in fact there's an implementation of that I think I'll try. If it takes a week or two to train, that's fine by me, I've got other things to be getting on with.
- k__ 11y agoMaybe they got a pile of old software they give to their PR people. "Hey, we would throw this away, but maybe this could be used to shine up our brand a bit"
- abalos 11y agoSeeing as Google has one of the most powerful computing networks in the world, I bet it's more that they can just throw resources at it until it's fast enough. This helps justify the trade off between speed and ease of use.
- johanneskanybal 11y agoLike a factor of 4 matters much if your framework makes it easier to scale out horizontally. Also, I'm sure it will get optimized over time.
- IanCal 11y agoInteresting benchmarks. One hopefully constructive critique, if you say things go out of memory, it'd be really useful to know what your setup is. Maybe you've got a big array of massive GPUs or you're running it on a more normal consumer GPU+box.
- brianchu 11y agoAccording to: https://github.com/soumith/convnet-benchmarks https://github.com/soumith/convnet-benchmarks, it's an NVIDIA TitanX (12GB GPU memory), which is pretty much the top of the line GPU for training neural nets. From my limited experience, most deep learning in research - including much of the state of the art - is done on single GPUs. (My guess is that if your model doesn't fit in 12GB, your model has way too many parameters to practically train anyway).
- IanCal 11y agoAh thank you for that, I hadn't clocked on that this was opened by the person who ran the repo. I should have checked the main readme.
- programnature 11y agoIn the announcement they called this the "reference implementation".
- mark_l_watson 11y agoFrom Google's perspective it is probably more about how TensorFlow scales out horizontally. If a researcher fires off a Borg run (or whatever they use now) and the job takes a few thousand CPUs, no problem, at least for research. They must have better optimization a for running in production, such as in place operations.
- mtw 11y agoThe rest of us don't have "a few thousand CPUs" though. Seen this way, TensorFlow is largely inaccessible and makes more sense to focus on existing librairies optimized for single cpus
- johanneskanybal 11y agoI don't think you understand what this library is for. You can pick up a graphic card for $100 and voila now you got 1000's of cores. To your defense the comment you answered seems to have mixed a few random things together too.
- varelse 11y agoTensorFlow is currently a single-server solution pthreads app. But assuming you're serious and not dabbling, you can build a ~27 TFLOP server for <$7K in parts or buy it assembled for ~$9K. http://exxactcorp.com/index.php/solution/solu_detail/225 http://exxactcorp.com/index.php/solution/solu_detail/225 Or if you just hate the thought of money burning a hole in your pocket, pay NVIDIA >$15K for the same thing with a pretty NVIDIA logo: https://developer.nvidia.com/devbox https://developer.nvidia.com/devbox If you're dabbling, just go buy a $1000 TitanX and you're instantly a minor deep learning superpower.
- iraphael 11y agoI don't know how much this matters, but an issue similar to this had been raised here: https://github.com/tensorflow/tensorflow/issues/120 https://github.com/tensorflow/tensorflow/issues/120 The responses seem to show that the way you implement things can make a big difference in runtime. Perhaps the scripts used for benchmarking can be further optimized? That said, the lack of in-place operations might be surprising (although it has been said that they are coming)
- donthateyo 11y agoUntil now, I've seen two responses to Google's TensorFlow from Facebook employees. Yann Le Cunn seemed to really challenge Jeff Dean about TensorFlow's scalability [1] and this benchmark puts TensorFlow down there in all the measures it tested for. I can't ignore the possibility that this criticism of TensorFlow from Facebook employees (while factually correct and constructive) might be driven by some competition and jealousy. [1] https://www.youtube.com/watch?v=90-S1M7Ny_o&t=39m https://www.youtube.com/watch?v=90-S1M7Ny_o&t=39m
- kastnerkyle 11y agoThis benchmark basically shows that releasing TensorFlow with cudnn v2 backend support hurts - v2 is quite a bit slower than v3 (current) and v4 (upcoming). TF has announced that they will update to v4 support, which should help quite a bit - but when many hobbyists and researchers are developing on one or two GPUs performance on that scale is more important (for them) than infinite scalability. It is not surprising that a tool developed and focused on `Google scale` work has some imperfections in a wildly different setting. The question is - will they (or some dedicated contributor) speed up a use case the business itself may not have a use for? My gut feeling is that they will, but these things usually don`t happen overnight. Torch, Theano, and Caffe have years of work put into them and have largely been focused on the one to two GPU case.
- smhx 11y agoI've done an apples to apples, TensorFlow + CuDNN R2 vs Torch + CuDNN R2
- kastnerkyle 11y agoMy point is more that even if they were on the same footing from benchmark timings, v2 is still far behind what is supported in Torch, Caffe, and Theano right now (v3 in all IIRC). Your comparison is very fair, and it is good insight!
- varelse 11y agoYou might be seeing the final gasps of Google's longstanding and now-reversed anti-GPU stance in TF's initial GPU performance. NVIDIA support alone will make sure TF knocks it out of the park down the road. IMO it's crazy to consider the first release the final say on TF's GPU performance. Caffe, Torch, and Theano have a huge head start (and lots of pre-existing technical support from NVIDIA). The biggest limitation I see right now is that their multi-GPU algorithms are really simple and inefficient. That will change I'm sure now that they're getting benchmarked against everyone else.
- deleted 11y ago[deleted]
- emcq 11y agoIs this surprising? I think it's expected, when most published numbers from Google suggest that this is only 2x over DistBelief, which Adam Coates improved upon 6x (http://www.cs.stanford.edu/~acoates/papers/CoatesHuvalWangWuNgCatanzaro_icml2013.pdf http://www.cs.stanford.edu/~acoates/papers/CoatesHuvalWangWu...) in 2013. Coming from a HPC background, DistBelief (and sometimes MR in general) has always felt like a square being forced into a round hole. Sure it "scales" but it's not how you would build a system from scratch to meet the same computational needs. Moving data around is much slower than doing extra math. DNNs provide some interesting opportunities for domain specific optimization with lossy math because empirically it seems that the solutions dont require significant precision. Something like a general Tensor library holds you back from using some of the tricks like synchronization-free hogwild, where it wouldn't be appropriate in other scenarios where more exact computation is necessary. One interesting step towards this is their implementation of lossy compression with a custom 16-bit half float representation. However a lot of people are dialing down the precision even further, including one of the authors of TensorFlow (http://petewarden.com/2015/05/23/why-are-eight-bits-enough-for-deep-neural-networks/ http://petewarden.com/2015/05/23/why-are-eight-bits-enough-f...). Scott Gray, one of the principle engineers at Nervana Systems, has a very wise reply in the github thread - they aren't doing much to optimize for memory locality, and wont have amazing speedups until that happens.