6 ms·

Until now, I've seen two responses to Google's TensorFlow from Facebook employees. Yann Le Cunn seemed to really challenge Jeff Dean about TensorFlow's scalabil

Until now, I've seen two responses to Google's TensorFlow from Facebook employees. Yann Le Cunn seemed to really challenge Jeff Dean about TensorFlow's scalability [1] and this benchmark puts TensorFlow down there in all the measures it tested for. I can't ignore the possibility that this criticism of TensorFlow from Facebook employees (while factually correct and constructive) might be driven by some competition and jealousy. [1] https://www.youtube.com/watch?v=90-S1M7Ny_o&t=39m https://www.youtube.com/watch?v=90-S1M7Ny_o&t=39m

kastnerkyle 11y ago
This benchmark basically shows that releasing TensorFlow with cudnn v2 backend support hurts - v2 is quite a bit slower than v3 (current) and v4 (upcoming). TF has announced that they will update to v4 support, which should help quite a bit - but when many hobbyists and researchers are developing on one or two GPUs performance on that scale is more important (for them) than infinite scalability. It is not surprising that a tool developed and focused on `Google scale` work has some imperfections in a wildly different setting. The question is - will they (or some dedicated contributor) speed up a use case the business itself may not have a use for? My gut feeling is that they will, but these things usually don`t happen overnight. Torch, Theano, and Caffe have years of work put into them and have largely been focused on the one to two GPU case.
- smhx 11y ago
  I've done an apples to apples, TensorFlow + CuDNN R2 vs Torch + CuDNN R2
  - kastnerkyle 11y ago
    My point is more that even if they were on the same footing from benchmark timings, v2 is still far behind what is supported in Torch, Caffe, and Theano right now (v3 in all IIRC). Your comparison is very fair, and it is good insight!
- varelse 11y ago
  You might be seeing the final gasps of Google's longstanding and now-reversed anti-GPU stance in TF's initial GPU performance. NVIDIA support alone will make sure TF knocks it out of the park down the road. IMO it's crazy to consider the first release the final say on TF's GPU performance. Caffe, Torch, and Theano have a huge head start (and lots of pre-existing technical support from NVIDIA). The biggest limitation I see right now is that their multi-GPU algorithms are really simple and inefficient. That will change I'm sure now that they're getting benchmarked against everyone else.
  - kastnerkyle 11y ago
    I don't think anyone is claiming TF won't get much, much faster (probably very soon). But claiming that right now - today - TF kills existing toolkits like Torch, Caffe, and Theano (which I have seen here and elsewhere - though not claiming you are in this camp) is a bit premature. Even with v4 support which is coming soon, the general setup that users have is one to two GPUs in one machine. This means adding the right inplace operations can have a huge impact on performance, and is probably a usecase that Google has not focused on given their internal infrastructure. I am sure they will normalize to "at or slightly above" the performance of other toolkits, but the question is when? This benchmark doesn't have anything to do with multi-GPU as far as I am aware - these are single machine, single GPU results. I would wager 90% of the deep learning hobbyist and research communities run in this setting, so benchmarking this is really important. For people with huge amounts of networked and distributed resources - distributed support will be very amazing!
    - varelse 11y ago
      Inplace is a nice late phase optimization, but the root problem here appears to be that Google's convolution kernels are crap compared to those in cuDNN3 and Neon (I asked Scott Gray about this directly and I trust his wisdom). No surprises there whatsoever. The tensorflow engine is the most gold-plated POS I've seen in a long time. if it were running on 1000+ servers, I'd get the level of overengineering they've applied here. But single server pthreads? WTF? Also parameter server is dumb unless you sweat the implementation of the gather/reduction ops, but I digress.
argonaut 11y ago
I think I'll give the benefit of the doubt to, you know, the pioneer of deep learning, inventor of convolutional neural networks, and (co)-inventor of the backpropagation algorithm.
- discardorama 11y ago
  Just because you invent an algorithm, it doesn't mean you know how to implement it in the most efficient way possible. I think the key to TensorFlow is not how fast it runs on 1 machine; but how fast it runs on 10,000. Consider map-reduce (Hadoop). Sure, you can sort 1GB data on a single machine 10x faster (using /usr/bin/sort) than using Hadoop on that machine; but make the data 1TB and add 1000 machines, now lets see how fast you can sort with /usr/bin/sort!
  - mtw 11y ago
    Scenarios of machine learning on 1 machine is way more common than machine learning on 10,000 machines! I doubt even Google would have 10,000 for google translate It's more realistic to discuss librairies for 1 machine / GPU
  - argonaut 11y ago
    I'm just pointing out how ridiculous it is to cast aspersions on the judgment of one of the world's leading experts on deep learning. It's also getting out of hand because everyone has already decided TensorFlow must be amazing, so everyone is extolling the virtues of what right now we only know to be speculation and PR.
    - hueving 11y ago
      >I'm just pointing out how ridiculous it is to cast aspersions on the judgment of one of the world's leading experts on deep learning. A position of authority doesn't mean the person is immune to jealousy. Someone in a high status position is going to be much more likely to have a knee-jerk defensive reaction to news that threatens their image.
smhx 11y ago
I'm the one who runs the benchmarks. It's sad that you put such a twist to the whole thing. I've been running convnet-benchmarks forever now, and I've been running them independently on separate personal hardware. I do this as a hobby. I've done an apples-to-apples comparison, and my benchmark review only puts the facts forward, I dont attack them. If you read my other social media comments, I've been pretty positive about TensorFlow, and I've even put in some groundwork to write Torch bindings for it. Please stop spewing nonsense interpolated from like 2 super-weak data points.

kastnerkyle 11y ago

This benchmark basically shows that releasing TensorFlow with cudnn v2 backend support hurts - v2 is quite a bit slower than v3 (current) and v4 (upcoming). TF has announced that they will update to v4 support, which should help quite a bit - but when many hobbyists and researchers are developing on one or two GPUs performance on that scale is more important (for them) than infinite scalability. It is not surprising that a tool developed and focused on `Google scale` work has some imperfections in a wildly different setting. The question is - will they (or some dedicated contributor) speed up a use case the business itself may not have a use for? My gut feeling is that they will, but these things usually don`t happen overnight. Torch, Theano, and Caffe have years of work put into them and have largely been focused on the one to two GPU case.

smhx 11y ago
I've done an apples to apples, TensorFlow + CuDNN R2 vs Torch + CuDNN R2
- kastnerkyle 11y ago
  My point is more that even if they were on the same footing from benchmark timings, v2 is still far behind what is supported in Torch, Caffe, and Theano right now (v3 in all IIRC). Your comparison is very fair, and it is good insight!
varelse 11y ago
You might be seeing the final gasps of Google's longstanding and now-reversed anti-GPU stance in TF's initial GPU performance. NVIDIA support alone will make sure TF knocks it out of the park down the road. IMO it's crazy to consider the first release the final say on TF's GPU performance. Caffe, Torch, and Theano have a huge head start (and lots of pre-existing technical support from NVIDIA). The biggest limitation I see right now is that their multi-GPU algorithms are really simple and inefficient. That will change I'm sure now that they're getting benchmarked against everyone else.
- kastnerkyle 11y ago
  I don't think anyone is claiming TF won't get much, much faster (probably very soon). But claiming that right now - today - TF kills existing toolkits like Torch, Caffe, and Theano (which I have seen here and elsewhere - though not claiming you are in this camp) is a bit premature. Even with v4 support which is coming soon, the general setup that users have is one to two GPUs in one machine. This means adding the right inplace operations can have a huge impact on performance, and is probably a usecase that Google has not focused on given their internal infrastructure. I am sure they will normalize to "at or slightly above" the performance of other toolkits, but the question is when? This benchmark doesn't have anything to do with multi-GPU as far as I am aware - these are single machine, single GPU results. I would wager 90% of the deep learning hobbyist and research communities run in this setting, so benchmarking this is really important. For people with huge amounts of networked and distributed resources - distributed support will be very amazing!
  - varelse 11y ago
    Inplace is a nice late phase optimization, but the root problem here appears to be that Google's convolution kernels are crap compared to those in cuDNN3 and Neon (I asked Scott Gray about this directly and I trust his wisdom). No surprises there whatsoever. The tensorflow engine is the most gold-plated POS I've seen in a long time. if it were running on 1000+ servers, I'd get the level of overengineering they've applied here. But single server pthreads? WTF? Also parameter server is dumb unless you sweat the implementation of the gather/reduction ops, but I digress.

argonaut 11y ago

I think I'll give the benefit of the doubt to, you know, the pioneer of deep learning, inventor of convolutional neural networks, and (co)-inventor of the backpropagation algorithm.

discardorama 11y ago
Just because you invent an algorithm, it doesn't mean you know how to implement it in the most efficient way possible. I think the key to TensorFlow is not how fast it runs on 1 machine; but how fast it runs on 10,000. Consider map-reduce (Hadoop). Sure, you can sort 1GB data on a single machine 10x faster (using /usr/bin/sort) than using Hadoop on that machine; but make the data 1TB and add 1000 machines, now lets see how fast you can sort with /usr/bin/sort!
- mtw 11y ago
  Scenarios of machine learning on 1 machine is way more common than machine learning on 10,000 machines! I doubt even Google would have 10,000 for google translate It's more realistic to discuss librairies for 1 machine / GPU
- argonaut 11y ago
  I'm just pointing out how ridiculous it is to cast aspersions on the judgment of one of the world's leading experts on deep learning. It's also getting out of hand because everyone has already decided TensorFlow must be amazing, so everyone is extolling the virtues of what right now we only know to be speculation and PR.
  - hueving 11y ago
    >I'm just pointing out how ridiculous it is to cast aspersions on the judgment of one of the world's leading experts on deep learning. A position of authority doesn't mean the person is immune to jealousy. Someone in a high status position is going to be much more likely to have a knee-jerk defensive reaction to news that threatens their image.

smhx 11y ago

I'm the one who runs the benchmarks. It's sad that you put such a twist to the whole thing. I've been running convnet-benchmarks forever now, and I've been running them independently on separate personal hardware. I do this as a hobby. I've done an apples-to-apples comparison, and my benchmark review only puts the facts forward, I dont attack them. If you read my other social media comments, I've been pretty positive about TensorFlow, and I've even put in some groundwork to write Torch bindings for it. Please stop spewing nonsense interpolated from like 2 super-weak data points.