3 ms·

Honestly, there are enough issues with TensorFlow right now due to CUDA 3.0 that using it with AWS is highly problematic. I appreciate the author's attempt, but

by Smerity 11y ago

Honestly, there are enough issues with TensorFlow right now due to CUDA 3.0 that using it with AWS is highly problematic. I appreciate the author's attempt, but there's no way the five lines of code he changed to allow CUDA 3.0 has fixed any of the issues found in [1], such as NaNs during training, equally slow training on a g2.2xlarge as a g2.8xlarge, etc ... If you're just interested in playing around, then your laptop will do fine - TensorFlow is happy with just about any hardware you throw at it. Hell, your modern Android phone will run it =] If you're interested in a more involved experiment, develop and debug your task locally on your laptop. By the time you're ready for large scale training, there might be a stable and battle tested AMI such that people are no longer reporting issues in [1] about it. Again, if you're interested, follow the CUDA 3.0 issue on GitHub[1] - this is nowhere near a solved problem and will only cause headaches if you're using it for education. [1]: https://github.com/tensorflow/tensorflow/issues/25 https://github.com/tensorflow/tensorflow/issues/25

alexkern 11y ago
Thanks for the feedback! I've added a note to the README that support is still experimental. I'll be tracking the issue and updating the repo + AMI as it develops. Will be compiling with the latest commit (72a5a60) for configurable CUDA Compute support soon.
- vrv 11y ago
  Thanks, would be good to have multiple sources of verification that HEAD now supports this natively without issues like unexpected NaNs. https://github.com/tensorflow/tensorflow/issues/25#issuecomment-156256071 https://github.com/tensorflow/tensorflow/issues/25#issuecomm... is one verification :)
  - alexkern 11y ago
    Updated to the latest commit, works for me! :)
erikbern 11y ago
I think the nan issue you are referring to was caused by some weird stuff with Bazel. I put together an AMI in virginia: ami-cf5028a5 and if you have a masochistic streak, here are the steps to do it yourself: https://gist.github.com/erikbern/78ba519b97b440e10640 https://gist.github.com/erikbern/78ba519b97b440e10640 The main issue I'm still seeing is that g2.8xlarge with my AMI doesn't run 4x faster than g2.2xlarge even if correctly detects the 4 GPU's. Haven't had time to identify the issue though.
- varelse 11y ago
  Xen virtualization disables P2P copies ergo GPUs have what we call a "failure to communicate and some GPUs you just can't reach (without going through the CPU that is)."

by Smerity 11y ago

alexkern 11y ago

Thanks for the feedback! I've added a note to the README that support is still experimental. I'll be tracking the issue and updating the repo + AMI as it develops. Will be compiling with the latest commit (72a5a60) for configurable CUDA Compute support soon.

vrv 11y ago
Thanks, would be good to have multiple sources of verification that HEAD now supports this natively without issues like unexpected NaNs. https://github.com/tensorflow/tensorflow/issues/25#issuecomment-156256071 https://github.com/tensorflow/tensorflow/issues/25#issuecomm... is one verification :)
- alexkern 11y ago
  Updated to the latest commit, works for me! :)

erikbern 11y ago

I think the nan issue you are referring to was caused by some weird stuff with Bazel. I put together an AMI in virginia: ami-cf5028a5 and if you have a masochistic streak, here are the steps to do it yourself: https://gist.github.com/erikbern/78ba519b97b440e10640 https://gist.github.com/erikbern/78ba519b97b440e10640 The main issue I'm still seeing is that g2.8xlarge with my AMI doesn't run 4x faster than g2.2xlarge even if correctly detects the 4 GPU's. Haven't had time to identify the issue though.

varelse 11y ago
Xen virtualization disables P2P copies ergo GPUs have what we call a "failure to communicate and some GPUs you just can't reach (without going through the CPU that is)."