3 ms·

Just because you invent an algorithm, it doesn't mean you know how to implement it in the most efficient way possible. I think the key to TensorFlow is not how

Just because you invent an algorithm, it doesn't mean you know how to implement it in the most efficient way possible. I think the key to TensorFlow is not how fast it runs on 1 machine; but how fast it runs on 10,000. Consider map-reduce (Hadoop). Sure, you can sort 1GB data on a single machine 10x faster (using /usr/bin/sort) than using Hadoop on that machine; but make the data 1TB and add 1000 machines, now lets see how fast you can sort with /usr/bin/sort!

mtw 11y ago
Scenarios of machine learning on 1 machine is way more common than machine learning on 10,000 machines! I doubt even Google would have 10,000 for google translate It's more realistic to discuss librairies for 1 machine / GPU
argonaut 11y ago
I'm just pointing out how ridiculous it is to cast aspersions on the judgment of one of the world's leading experts on deep learning. It's also getting out of hand because everyone has already decided TensorFlow must be amazing, so everyone is extolling the virtues of what right now we only know to be speculation and PR.
- hueving 11y ago
  >I'm just pointing out how ridiculous it is to cast aspersions on the judgment of one of the world's leading experts on deep learning. A position of authority doesn't mean the person is immune to jealousy. Someone in a high status position is going to be much more likely to have a knee-jerk defensive reaction to news that threatens their image.
  - kastnerkyle 11y ago
    Yann LeCun has been working on hardware implementations of convnets almost since the beginning. LeNet 5 (the check reader of ATT, circa early 90s) needed a dedicated and specialized hardware implementation IIRC. It could be kneejerk, but he definitely has the background to make a fair assessment - and this benchmark seems to back his claims.
    - rryan 11y ago
      > and this benchmark seems to back his claims. These benchmarks evaluate single-node performance. LeCun's remarks were concerning distributed training (specifically that bandwidth between machines is a limiting factor to scalability) -- which we can't test yet since the current version of TF is single-node only. Dean's response in the video "it depends on your [computer] network" is an interesting response :).
      - kastnerkyle 11y ago
        Both Google and FB (from what I understand) have a ton of tricks to help this, but I expect Yann was speaking generally even with all these tricks, and he is right from what I have seen. The old paper on DistBelief talks about topping out at ~80 machines due to network overhead - it would be great if they talk more about distributed TF in an upcoming paper. If TF really has a way to make general, networked, distributed training efficient (more than 1 bit weight updates, low precision weights and all the other crazy tricks which already exist) - that is truly remarkable and they rightly deserve huge kudos. If they are faster in distributed training only on Google machines or with Google's network architecture, that isn't really a useful datapoint for the general public. We could wire everything with 10GB/s NiCs, change to jumbo frames, pull all these other bandwidth reducing tricks, trick out the Linux kernel, etc. and then your network probably won't matter - but that isn't really general or cheap. The key will be what is the minimum effort necessary to avoid network bottlenecking, and does TF improve that minimum level over existing solutions?
rspeer 11y ago
> make the data 1TB and add 1000 machines, now lets see how fast you can sort with /usr/bin/sort! Still faster than Hadoop, because sorting the data will be faster than sending it over a network. What should I do with the other 999 machines? (I know what you actually mean, by the way. Make it 100 TB and say that the data is already spread out across 1000 machines.)

mtw 11y ago

Scenarios of machine learning on 1 machine is way more common than machine learning on 10,000 machines! I doubt even Google would have 10,000 for google translate It's more realistic to discuss librairies for 1 machine / GPU

argonaut 11y ago

I'm just pointing out how ridiculous it is to cast aspersions on the judgment of one of the world's leading experts on deep learning. It's also getting out of hand because everyone has already decided TensorFlow must be amazing, so everyone is extolling the virtues of what right now we only know to be speculation and PR.

hueving 11y ago
>I'm just pointing out how ridiculous it is to cast aspersions on the judgment of one of the world's leading experts on deep learning. A position of authority doesn't mean the person is immune to jealousy. Someone in a high status position is going to be much more likely to have a knee-jerk defensive reaction to news that threatens their image.
- kastnerkyle 11y ago
  Yann LeCun has been working on hardware implementations of convnets almost since the beginning. LeNet 5 (the check reader of ATT, circa early 90s) needed a dedicated and specialized hardware implementation IIRC. It could be kneejerk, but he definitely has the background to make a fair assessment - and this benchmark seems to back his claims.
  - rryan 11y ago
    > and this benchmark seems to back his claims. These benchmarks evaluate single-node performance. LeCun's remarks were concerning distributed training (specifically that bandwidth between machines is a limiting factor to scalability) -- which we can't test yet since the current version of TF is single-node only. Dean's response in the video "it depends on your [computer] network" is an interesting response :).
    - kastnerkyle 11y ago
      Both Google and FB (from what I understand) have a ton of tricks to help this, but I expect Yann was speaking generally even with all these tricks, and he is right from what I have seen. The old paper on DistBelief talks about topping out at ~80 machines due to network overhead - it would be great if they talk more about distributed TF in an upcoming paper. If TF really has a way to make general, networked, distributed training efficient (more than 1 bit weight updates, low precision weights and all the other crazy tricks which already exist) - that is truly remarkable and they rightly deserve huge kudos. If they are faster in distributed training only on Google machines or with Google's network architecture, that isn't really a useful datapoint for the general public. We could wire everything with 10GB/s NiCs, change to jumbo frames, pull all these other bandwidth reducing tricks, trick out the Linux kernel, etc. and then your network probably won't matter - but that isn't really general or cheap. The key will be what is the minimum effort necessary to avoid network bottlenecking, and does TF improve that minimum level over existing solutions?

rspeer 11y ago

> make the data 1TB and add 1000 machines, now lets see how fast you can sort with /usr/bin/sort! Still faster than Hadoop, because sorting the data will be faster than sending it over a network. What should I do with the other 999 machines? (I know what you actually mean, by the way. Make it 100 TB and say that the data is already spread out across 1000 machines.)