4 ms·

It's nice to see Leaf coming along so well. Part of me would love to be able to build models in rust. For more benchmarks (including updated TensorFlow perform

It's nice to see Leaf coming along so well. Part of me would love to be able to build models in rust. For more benchmarks (including updated TensorFlow performance with cudnn v4) see https://github.com/soumith/convnet-benchmarks https://github.com/soumith/convnet-benchmarks

narrator 11y ago
Too bad Tensorflow Cuda only works with the top of the line NVIDIA cards that cost over 1k.
- jimfleming 11y ago
  Can you expand on that? As far as I'm aware that's not true (anymore). It runs quite fine on AWS which uses older NVIDIA cards and I know several people use it on older-gen GPU-enabled MBPs. EDIT: clarification
  - narrator 11y ago
    https://www.tensorflow.org/versions/r0.7/get_started/os_setup.html#optional-install-cuda-gpus-on-linux https://www.tensorflow.org/versions/r0.7/get_started/os_setu... I guess the cards listed there are not an all inclusive list?
    - vrv 11y ago
      Yeah, those are just cards we know that work: we don't have all possible GPUs to test our 20+ changes a day on, so we can't formally guarantee it will work on older cards, but in general we try our best to keep it all working for older and even newer cards.
      - nightski 11y ago
        Maybe a "reported to work but not guaranteed" section is in order.
        vrv 11y ago
        Keeping that up-to-date and useful is as hard as keeping benchmarks in Deep Learning up to date ;)
    - jimfleming 11y ago
      From that page: > Supported cards include but are not limited to[...]
    - barneso 11y ago
      In my experience, the architecture supports cards with shader model >= 3.0. Occasionally a commit will break the support (eg https://bitbucket.org/eigen/eigen/commits/a19653b8035d8ace53af184b02acb1ee9ab417c6 https://bitbucket.org/eigen/eigen/commits/a19653b8035d8ace53... was required earlier this year) but this is a function of the speed of development and usually straightforward to fix.
- rough-sea 11y ago
  Not true, works on GTX 750ti out of the box which is currently around $130
reality_hacker 11y ago
Just curious, does this benchmark include models quality somewhere?
- mjhirn 11y ago
  You mean, comparing the performance after training a model?
  - reality_hacker 11y ago
    Performance in terms of model accuracy. How accurate will be those models.
    - hobofan 11y ago
      No accuracy is not included, since that is something that should be constant for each model across frameworks. These benchmarks aim to highlight the performance differences in terms of speed/memory usage across frameworks and machine configurations. There is also the practical hurdle that training imagenet models to maximum accuracy takes 1 week+.
      - reality_hacker 11y ago
        So, are you saying that output of all frameworks are exactly and always the same? Sorry, if I am asking stupid questions.
        hobofan 11y ago
        Not exactly, no. But not even the output of the same framework will always be the same since you are usually randomly initalizing the weights in a network and randomly picking the samples used in SGD[1] (the seed for the RNG could of course be a fixed one to mitigate this somewhat). But in the end, if you are using the same model, the same solver and the same RNG, yes the output of all frameworks should be the same. In practice this also mostly holds true, since the stochastic processes involved are geared towards finding a good local minimum, which is the same given a model and a dataset. [1]: https://en.wikipedia.org/wiki/Stochastic_gradient_descent https://en.wikipedia.org/wiki/Stochastic_gradient_descent
mjhirn 11y ago
I just patched the link on the landing page to the repo [1] and updated benchmarks for Leaf 0.2 + cuDNN 4 for Overfeat and VGG. But I couldn't get Torch and Tensorflow running with cuDNN 4, yet. [1]: https://github.com/autumnai/deep-learning-benchmarks https://github.com/autumnai/deep-learning-benchmarks
- vrv 11y ago
  Feel free to ping us (TensorFlow) on github issues to get installation issues resolved -- on cudnn r4 we're doing much better, and we're soon to check in a series of changes to get us roughly on par with Torch on cudnn r4. We'd love to get a more up-to-date representation of the state of our own progress :)
  - mjhirn 11y ago
    Great, I will let you know. Looking forward to the benches.

narrator 11y ago

Too bad Tensorflow Cuda only works with the top of the line NVIDIA cards that cost over 1k.

jimfleming 11y ago
Can you expand on that? As far as I'm aware that's not true (anymore). It runs quite fine on AWS which uses older NVIDIA cards and I know several people use it on older-gen GPU-enabled MBPs. EDIT: clarification
- narrator 11y ago
  https://www.tensorflow.org/versions/r0.7/get_started/os_setup.html#optional-install-cuda-gpus-on-linux https://www.tensorflow.org/versions/r0.7/get_started/os_setu... I guess the cards listed there are not an all inclusive list?
  - vrv 11y ago
    Yeah, those are just cards we know that work: we don't have all possible GPUs to test our 20+ changes a day on, so we can't formally guarantee it will work on older cards, but in general we try our best to keep it all working for older and even newer cards.
    - nightski 11y ago
      Maybe a "reported to work but not guaranteed" section is in order.
      - vrv 11y ago
        Keeping that up-to-date and useful is as hard as keeping benchmarks in Deep Learning up to date ;)
  - jimfleming 11y ago
    From that page: > Supported cards include but are not limited to[...]
  - barneso 11y ago
    In my experience, the architecture supports cards with shader model >= 3.0. Occasionally a commit will break the support (eg https://bitbucket.org/eigen/eigen/commits/a19653b8035d8ace53af184b02acb1ee9ab417c6 https://bitbucket.org/eigen/eigen/commits/a19653b8035d8ace53... was required earlier this year) but this is a function of the speed of development and usually straightforward to fix.
rough-sea 11y ago
Not true, works on GTX 750ti out of the box which is currently around $130

reality_hacker 11y ago

Just curious, does this benchmark include models quality somewhere?

mjhirn 11y ago
You mean, comparing the performance after training a model?
- reality_hacker 11y ago
  Performance in terms of model accuracy. How accurate will be those models.
  - hobofan 11y ago
    No accuracy is not included, since that is something that should be constant for each model across frameworks. These benchmarks aim to highlight the performance differences in terms of speed/memory usage across frameworks and machine configurations. There is also the practical hurdle that training imagenet models to maximum accuracy takes 1 week+.
    - reality_hacker 11y ago
      So, are you saying that output of all frameworks are exactly and always the same? Sorry, if I am asking stupid questions.
      - hobofan 11y ago
        Not exactly, no. But not even the output of the same framework will always be the same since you are usually randomly initalizing the weights in a network and randomly picking the samples used in SGD[1] (the seed for the RNG could of course be a fixed one to mitigate this somewhat). But in the end, if you are using the same model, the same solver and the same RNG, yes the output of all frameworks should be the same. In practice this also mostly holds true, since the stochastic processes involved are geared towards finding a good local minimum, which is the same given a model and a dataset. [1]: https://en.wikipedia.org/wiki/Stochastic_gradient_descent https://en.wikipedia.org/wiki/Stochastic_gradient_descent

mjhirn 11y ago

I just patched the link on the landing page to the repo [1] and updated benchmarks for Leaf 0.2 + cuDNN 4 for Overfeat and VGG. But I couldn't get Torch and Tensorflow running with cuDNN 4, yet. [1]: https://github.com/autumnai/deep-learning-benchmarks https://github.com/autumnai/deep-learning-benchmarks

vrv 11y ago
Feel free to ping us (TensorFlow) on github issues to get installation issues resolved -- on cudnn r4 we're doing much better, and we're soon to check in a series of changes to get us roughly on par with Torch on cudnn r4. We'd love to get a more up-to-date representation of the state of our own progress :)
- mjhirn 11y ago
  Great, I will let you know. Looking forward to the benches.