5 ms·

Tensorflow 2.0.0-Alpha0

81 points by Gimpei 8y ago

runesoerensen 8y ago
Also https://medium.com/tensorflow/test-drive-tensorflow-2-0-alpha-b6dd1e522b01 https://medium.com/tensorflow/test-drive-tensorflow-2-0-alph...
- dynamicwebpaige 8y ago
  If you're interested in upgrading your models, also make sure to check out this Medium post: https://medium.com/tensorflow/upgrading-your-code-to-tensorflow-2-0-f72c3a4d83b5 https://medium.com/tensorflow/upgrading-your-code-to-tensorf...
rolleiflex 8y ago
One thing I've always wondered is, why is TensorFlow NVIDIA-only? I was hoping a new major release would fix that, but it doesn't appear to have. I've recently gotten an eGPU for my Macbook Pro for playing games in whatever little off-time I have on Windows. I also wanted to use it on Mac, though, and Macs only support AMD graphics. So I got a Vega 64. The other half of the reason was that I wanted to play with deep learning. Turns out TensorFlow just does not work on AMD. There is a fork of it maintained by AMD, but they only support it for Linux, and the code in question reaches deep enough to hardware layer that it cannot be run in a VM. It's also always at least a few versions behind as well. With NVIDIA GPUs more expensive and poorly performing per dollar than they have ever been, TF2 could have been the moment they made a major improvement. Sad to see that is not the case.
- vishvananda 8y ago
  I think this is primarily due to the immense effort that NVIDIA has put into CUDA. It works very well and it is extremely fast. The alternatives for AMD are OpenCL and ROCm which have seriously lagged behind CUDA in every respect. EDIT: lots of theories and discussion here: https://www.reddit.com/r/MachineLearning/comments/7tf4da/d_whats_the_best_option_right_now_for_amd/ https://www.reddit.com/r/MachineLearning/comments/7tf4da/d_w...
- minimaxir 8y ago
  CUDA is one hell of a drug. That said, I'm still surprised Apple gave up on NVIDIA support, especially since there's no it-just-works option for deep learning on a Mac.
  - rolleiflex 8y ago
    Apple and NVIDIA have been in each other's bad books since the 2012 Macbook Pro incident where discrete GPUs of those devices got hot beyond their thermal specs and melted themselves out of the soldered sockets, which NVIDIA refused to compensate Apple for. I know, because I had one of those. Apple took the hit to repair those Macbook Pros even while out of warranty, even if it was ultimately NVIDIA's poor engineering. They've not worked with each other since. 2012 Apple was pretty much the height of the quality apex.
  - m-p-3 8y ago
    At least AMD is working hard to upstream their drivers into the Linux kernel, which I don't doubt Apple appreciate when they merge the code into theirs.
  - tedivm 8y ago
    Apple and NVIDIA have spent the last six months fighting over whose fault it is that NVIDIA doesn't have working drivers for the latest OSX. https://create.pro/blog/nvidia-drivers-for-macos-mojave-timeline-of-events/ https://create.pro/blog/nvidia-drivers-for-macos-mojave-time...
  - dnautics 8y ago
    Keeping in mind that apple is primarily a consumer focused company, What's the it-just-works consumer machine learning use case that requires GPGPU on end user hardware?
    - minimaxir 8y ago
      Apple has been pushing machine learning at virtually every WWDC and its integrations from developers into product launches since. A developer can do the ugly hacking to get an NVIDIA eGPU working on macOS, but it's a hassle.
- endorphone 8y ago
  AMD dropped the ball by not dedicating the sort of resources that nvidia has on the deep learning space (and playing the long, but risky move of throwing all in with ROCm). Similarly, Apple dropped the ball by being so wed to AMD. Coincidentally I was just digging through the Tensorflow code to find where it delegates to CUDA -- an enormous task given that it is a huge volume of dependencies many layers deep -- because ultimately most neural nets entail a significant amount of simple math. It seems, and I say this in the nonsensical, overly confident way that we do when we look on from the outside, that it should be trivial making it work with OpenCL or even just a branch for AMD.
  - dragontamer 8y ago
    ROCm, or more specifically HCC, is comparable to CUDA: AMD needs to develop it more before anything else. Its the fundamental structure to GPU-coding. AMD can't even write a comparable library unless it is built on top of ROCm / HCC. Like, how would you even start to implement Tensorflow 2.0.0 for AMD Graphics cards otherwise? OpenCL was theoretically an option. But the old AMDGPU-pro driver had the OpenCL compiler as part of the driver stack. Which means OpenCL Compiler issues would appear and disappear as you updated your graphics card. Uhhhh... not good for deployment to say the least. Lots of OpenCL code with "This causes the compiler to enter an infinite loop on AMD Driver 12.5.whatever". There are serious issues about OpenCL's fundamental design. You really shouldn't have a full OpenCL compiler hidden inside of your device drivers. Its a nightmare to test and deploy, especially in a world of changing device drivers. -------- ROCm is a proper compiler stack and a proper development environment. If anything, AMD's mistake was taking so long before ROCm became usable. Even today, there are some issues (it doesn't work with Blender 2.79 OpenCL code yet), but at least its a proper development environment where you can build libraries on top of. AMD needs to get its compute stack in order if it wants to be taken seriously. Fortunately, it seems like they understand the issue. Hopefully the next year or two will be helpful, since they're ramping up on ROCm development.
  - zdw 8y ago
    AMD is a far smaller company than either nVidia or Intel, its two main competitors. There are a variety of reasons for this - anticompetitive behavior from those competitors, chip designs like Bulldozer that focused on many wide execution units at the expense of single threaded performance, etc., but what it comes down to is that they just don't have a comparable amount of resources dedicated to ROCm compared to what nVidia has invested in CUDA. I'm quite impressed with their current and near future designs in the CPU space, but GPU design is hard and beginning to hit a performance wall - witness the lackluster reaction to nVidia's current RTX chips in many areas.
- currymj 8y ago
  I'm not an expert on this and would love to hear from one, but as far as I know there's simply more work to be done on AMD's end, to provide fast implementations of all the operations TensorFlow, PyTorch, and others need. ROCm hasn't reached parity with CUDA. Theano used to work with OpenCL and therefore AMD, but the support was never that great, and I don't know that many people used it. And Theano is now EOL.
- nabla9 8y ago
  NVIDIA invested heavily into drivers and libraries (CUDA and cuDNN) and other tools like TensorRT. They created the ecosystem where deep learning can be tuned and optimized to work. AMD took minimal effort route. They just design graphics cards with some added functionality and if someone wants to write build libraries fro them, good.
- aseipp 8y ago
  Nvidia spends more money on marketing, developers, tooling, documentation, infrastructure, and publishing high quality, usable libraries for GPU numerical computing. They have been doing this for years (CUDA is +10 years old) and are very good at it, while HIP/HCC are just barely usable. This is basically all it comes down to, in a nutshell. I want to be clear that none of this is trivial in any sense; delivering a usable GPGPU software stack is extremely difficult work, and Nvidia is simply miles ahead of AMD in nearly every way, except the fact ROCm and the AMDKFD driver are going upstream to Linux. AMD is of course also a smaller company than Nvidia, as well, so their resources are more limited. Here's the thing: In reality, it doesn't matter if a Radeon VII or Vega64 has better theoretical TFLOPs than a 1080 Ti or whatever if the card runs 50W hotter(!!!) and the software is bad enough it runs at 1/2 the performance of the competitor. 50W hotter at a 10% perf loss and a 40% decrease in price is attractive. Hotter at 1/2 the perf, is a non-starter (almost entirely ignoring price factors). Free drivers do not make up for this. DL is a fast enough field that, for most practitioners, your time is better served as an enthusiast/participant by just paying the extra upcharge on an NVidia card and getting on with your life, so you can actually train models and do your work. There is only one viable solution if you want to "do your work" today -- not for better, and almost certainly for worst. Ceterum censeo. DL moves fast enough that it's always possible for new frameworks to come around, so perhaps a competitor with a multi-vendor design, from the ground up, can break though. But then you have to ask whether a lot of these frameworks can even succeed/tread water in such an environment without Google/Facebook propping them up, which is a separate problem... And before that, AMD has to make sure HIP/HCC have to work reliably, every time, everywhere, and they have the libraries to back it up. I'm hoping the succeed at this.
  - dragontamer 8y ago
    AMD's software stack is definitely holding them back. But the hardware in smaller tests I've done really can get close to the theoretical TFLOPs / Memory Bandwidth numbers that they tout. NVidia made the decision to hop on the Clang bandwagon years ago, and clearly that's allowed them to more easily optimize their code and move things forward. AMD finally has hopped on Clang with ROCm (but unfortunately still has some kinks to work out, esp with OpenCL)... and the compiler output seems to be superior as a result. ------------- It seems unlikely to get a value-proposition out of AMD if you're going for deep learning. NVidias libraries are simply superior at the moment, and they get superior performance as a result. But when you look at lower-level software that is written for both GPUs, AMD's compute girth really shows. Cryptocoins are perhaps the best example, although not every work-load looks like mining. Luxray and Blender are more pragmatic: software raytracers which demonstrate AMD's incredibly GPU girth and memory performance with HBM2. The AMD Vega chips can outperform NVidia's equivalent in these cases. ----------- All in all, AMD's ROCm environment seems like it is ready for programmers who are willing to stick to the C (or HCC) level, coding their own libraries and maybe using a bit of inline assembly here or there to maximize performance. CUDA has more libraries (BLAS: linear algebra, better Tensorflow support for Deep Learning, etc. etc.) and that's definitely a major selling point. Especially because of how well those libraries are written. I guess what I'm trying to say is... AMD's hardware is fine. Its their software thats clearly holding them back.
- ericd 8y ago
  It's just not worth trying to shoehorn a Mac into a deep learning role, and I say this as someone who works almost exclusively on OS X. If you don't want to build a dedicated deep learning box, it seems like there are some decent cloud options now (though I haven't tried them).
diminish 8y ago
What are you using in your projects? Is TF the dominant tool in commercial or startup DNN projects?
- minimaxir 8y ago
  TF (or Keras via TF) is definitely the most popular framework for newer projects, although some argue PyTorch is holistically better.
- async_dev 8y ago
  I think pytorch is more popular in research, but TF is the most popular for commercial applications right now
- p1esk 8y ago
  I have to use TF at work (small startup), but I use Pytorch for personal projects and research.
- FridgeSeal 8y ago
  I'm about to use Flux.jl for my next project. I have previously tried using TF but it was super painful so I used Keras instead.
p1esk 8y ago
It feels more like Pytorch now: https://github.com/tensorflow/docs/blob/master/site/en/r2/tutorials/eager/custom_layers.ipynb https://github.com/tensorflow/docs/blob/master/site/en/r2/tu...
DevX101 8y ago
Any thoughts on how this version now compares to PyTorch 1.0?
hgasimov 8y ago
Thanks god Tensorflow is doing something with global variables and sessions
truth_seeker 8y ago
Cool. I would love to see pypy support for this major version.

81 points by Gimpei 8y ago

Also https://medium.com/tensorflow/test-drive-tensorflow-2-0-alpha-b6dd1e522b01 https://medium.com/tensorflow/test-drive-tensorflow-2-0-alph...

dynamicwebpaige 8y ago
If you're interested in upgrading your models, also make sure to check out this Medium post: https://medium.com/tensorflow/upgrading-your-code-to-tensorflow-2-0-f72c3a4d83b5 https://medium.com/tensorflow/upgrading-your-code-to-tensorf...

rolleiflex 8y ago

One thing I've always wondered is, why is TensorFlow NVIDIA-only? I was hoping a new major release would fix that, but it doesn't appear to have. I've recently gotten an eGPU for my Macbook Pro for playing games in whatever little off-time I have on Windows. I also wanted to use it on Mac, though, and Macs only support AMD graphics. So I got a Vega 64. The other half of the reason was that I wanted to play with deep learning. Turns out TensorFlow just does not work on AMD. There is a fork of it maintained by AMD, but they only support it for Linux, and the code in question reaches deep enough to hardware layer that it cannot be run in a VM. It's also always at least a few versions behind as well. With NVIDIA GPUs more expensive and poorly performing per dollar than they have ever been, TF2 could have been the moment they made a major improvement. Sad to see that is not the case.

vishvananda 8y ago
I think this is primarily due to the immense effort that NVIDIA has put into CUDA. It works very well and it is extremely fast. The alternatives for AMD are OpenCL and ROCm which have seriously lagged behind CUDA in every respect. EDIT: lots of theories and discussion here: https://www.reddit.com/r/MachineLearning/comments/7tf4da/d_whats_the_best_option_right_now_for_amd/ https://www.reddit.com/r/MachineLearning/comments/7tf4da/d_w...
minimaxir 8y ago
CUDA is one hell of a drug. That said, I'm still surprised Apple gave up on NVIDIA support, especially since there's no it-just-works option for deep learning on a Mac.
- rolleiflex 8y ago
  Apple and NVIDIA have been in each other's bad books since the 2012 Macbook Pro incident where discrete GPUs of those devices got hot beyond their thermal specs and melted themselves out of the soldered sockets, which NVIDIA refused to compensate Apple for. I know, because I had one of those. Apple took the hit to repair those Macbook Pros even while out of warranty, even if it was ultimately NVIDIA's poor engineering. They've not worked with each other since. 2012 Apple was pretty much the height of the quality apex.
- m-p-3 8y ago
  At least AMD is working hard to upstream their drivers into the Linux kernel, which I don't doubt Apple appreciate when they merge the code into theirs.
- tedivm 8y ago
  Apple and NVIDIA have spent the last six months fighting over whose fault it is that NVIDIA doesn't have working drivers for the latest OSX. https://create.pro/blog/nvidia-drivers-for-macos-mojave-timeline-of-events/ https://create.pro/blog/nvidia-drivers-for-macos-mojave-time...
- dnautics 8y ago
  Keeping in mind that apple is primarily a consumer focused company, What's the it-just-works consumer machine learning use case that requires GPGPU on end user hardware?
  - minimaxir 8y ago
    Apple has been pushing machine learning at virtually every WWDC and its integrations from developers into product launches since. A developer can do the ugly hacking to get an NVIDIA eGPU working on macOS, but it's a hassle.
endorphone 8y ago
AMD dropped the ball by not dedicating the sort of resources that nvidia has on the deep learning space (and playing the long, but risky move of throwing all in with ROCm). Similarly, Apple dropped the ball by being so wed to AMD. Coincidentally I was just digging through the Tensorflow code to find where it delegates to CUDA -- an enormous task given that it is a huge volume of dependencies many layers deep -- because ultimately most neural nets entail a significant amount of simple math. It seems, and I say this in the nonsensical, overly confident way that we do when we look on from the outside, that it should be trivial making it work with OpenCL or even just a branch for AMD.
- dragontamer 8y ago
  ROCm, or more specifically HCC, is comparable to CUDA: AMD needs to develop it more before anything else. Its the fundamental structure to GPU-coding. AMD can't even write a comparable library unless it is built on top of ROCm / HCC. Like, how would you even start to implement Tensorflow 2.0.0 for AMD Graphics cards otherwise? OpenCL was theoretically an option. But the old AMDGPU-pro driver had the OpenCL compiler as part of the driver stack. Which means OpenCL Compiler issues would appear and disappear as you updated your graphics card. Uhhhh... not good for deployment to say the least. Lots of OpenCL code with "This causes the compiler to enter an infinite loop on AMD Driver 12.5.whatever". There are serious issues about OpenCL's fundamental design. You really shouldn't have a full OpenCL compiler hidden inside of your device drivers. Its a nightmare to test and deploy, especially in a world of changing device drivers. -------- ROCm is a proper compiler stack and a proper development environment. If anything, AMD's mistake was taking so long before ROCm became usable. Even today, there are some issues (it doesn't work with Blender 2.79 OpenCL code yet), but at least its a proper development environment where you can build libraries on top of. AMD needs to get its compute stack in order if it wants to be taken seriously. Fortunately, it seems like they understand the issue. Hopefully the next year or two will be helpful, since they're ramping up on ROCm development.
- zdw 8y ago
  AMD is a far smaller company than either nVidia or Intel, its two main competitors. There are a variety of reasons for this - anticompetitive behavior from those competitors, chip designs like Bulldozer that focused on many wide execution units at the expense of single threaded performance, etc., but what it comes down to is that they just don't have a comparable amount of resources dedicated to ROCm compared to what nVidia has invested in CUDA. I'm quite impressed with their current and near future designs in the CPU space, but GPU design is hard and beginning to hit a performance wall - witness the lackluster reaction to nVidia's current RTX chips in many areas.
currymj 8y ago
I'm not an expert on this and would love to hear from one, but as far as I know there's simply more work to be done on AMD's end, to provide fast implementations of all the operations TensorFlow, PyTorch, and others need. ROCm hasn't reached parity with CUDA. Theano used to work with OpenCL and therefore AMD, but the support was never that great, and I don't know that many people used it. And Theano is now EOL.
nabla9 8y ago
NVIDIA invested heavily into drivers and libraries (CUDA and cuDNN) and other tools like TensorRT. They created the ecosystem where deep learning can be tuned and optimized to work. AMD took minimal effort route. They just design graphics cards with some added functionality and if someone wants to write build libraries fro them, good.
aseipp 8y ago
Nvidia spends more money on marketing, developers, tooling, documentation, infrastructure, and publishing high quality, usable libraries for GPU numerical computing. They have been doing this for years (CUDA is +10 years old) and are very good at it, while HIP/HCC are just barely usable. This is basically all it comes down to, in a nutshell. I want to be clear that none of this is trivial in any sense; delivering a usable GPGPU software stack is extremely difficult work, and Nvidia is simply miles ahead of AMD in nearly every way, except the fact ROCm and the AMDKFD driver are going upstream to Linux. AMD is of course also a smaller company than Nvidia, as well, so their resources are more limited. Here's the thing: In reality, it doesn't matter if a Radeon VII or Vega64 has better theoretical TFLOPs than a 1080 Ti or whatever if the card runs 50W hotter(!!!) and the software is bad enough it runs at 1/2 the performance of the competitor. 50W hotter at a 10% perf loss and a 40% decrease in price is attractive. Hotter at 1/2 the perf, is a non-starter (almost entirely ignoring price factors). Free drivers do not make up for this. DL is a fast enough field that, for most practitioners, your time is better served as an enthusiast/participant by just paying the extra upcharge on an NVidia card and getting on with your life, so you can actually train models and do your work. There is only one viable solution if you want to "do your work" today -- not for better, and almost certainly for worst. Ceterum censeo. DL moves fast enough that it's always possible for new frameworks to come around, so perhaps a competitor with a multi-vendor design, from the ground up, can break though. But then you have to ask whether a lot of these frameworks can even succeed/tread water in such an environment without Google/Facebook propping them up, which is a separate problem... And before that, AMD has to make sure HIP/HCC have to work reliably, every time, everywhere, and they have the libraries to back it up. I'm hoping the succeed at this.
- dragontamer 8y ago
  AMD's software stack is definitely holding them back. But the hardware in smaller tests I've done really can get close to the theoretical TFLOPs / Memory Bandwidth numbers that they tout. NVidia made the decision to hop on the Clang bandwagon years ago, and clearly that's allowed them to more easily optimize their code and move things forward. AMD finally has hopped on Clang with ROCm (but unfortunately still has some kinks to work out, esp with OpenCL)... and the compiler output seems to be superior as a result. ------------- It seems unlikely to get a value-proposition out of AMD if you're going for deep learning. NVidias libraries are simply superior at the moment, and they get superior performance as a result. But when you look at lower-level software that is written for both GPUs, AMD's compute girth really shows. Cryptocoins are perhaps the best example, although not every work-load looks like mining. Luxray and Blender are more pragmatic: software raytracers which demonstrate AMD's incredibly GPU girth and memory performance with HBM2. The AMD Vega chips can outperform NVidia's equivalent in these cases. ----------- All in all, AMD's ROCm environment seems like it is ready for programmers who are willing to stick to the C (or HCC) level, coding their own libraries and maybe using a bit of inline assembly here or there to maximize performance. CUDA has more libraries (BLAS: linear algebra, better Tensorflow support for Deep Learning, etc. etc.) and that's definitely a major selling point. Especially because of how well those libraries are written. I guess what I'm trying to say is... AMD's hardware is fine. Its their software thats clearly holding them back.
ericd 8y ago
It's just not worth trying to shoehorn a Mac into a deep learning role, and I say this as someone who works almost exclusively on OS X. If you don't want to build a dedicated deep learning box, it seems like there are some decent cloud options now (though I haven't tried them).

diminish 8y ago

What are you using in your projects? Is TF the dominant tool in commercial or startup DNN projects?

minimaxir 8y ago
TF (or Keras via TF) is definitely the most popular framework for newer projects, although some argue PyTorch is holistically better.
async_dev 8y ago
I think pytorch is more popular in research, but TF is the most popular for commercial applications right now
p1esk 8y ago
I have to use TF at work (small startup), but I use Pytorch for personal projects and research.
FridgeSeal 8y ago
I'm about to use Flux.jl for my next project. I have previously tried using TF but it was super painful so I used Keras instead.

p1esk 8y ago

It feels more like Pytorch now: https://github.com/tensorflow/docs/blob/master/site/en/r2/tutorials/eager/custom_layers.ipynb https://github.com/tensorflow/docs/blob/master/site/en/r2/tu...

DevX101 8y ago

Any thoughts on how this version now compares to PyTorch 1.0?

hgasimov 8y ago

Thanks god Tensorflow is doing something with global variables and sessions

truth_seeker 8y ago

Cool. I would love to see pypy support for this major version.