5 ms·

We've been experimenting with TF 1.14 on a Radeon VII. When you manage to properly install it, it does work at about 90% the speed of an NVIDIA RTX 2080 Ti, but

We've been experimenting with TF 1.14 on a Radeon VII. When you manage to properly install it, it does work at about 90% the speed of an NVIDIA RTX 2080 Ti, but is much cheaper, which can be a good trade-off. However, the tooling and installation procedure is still rough around the edges and the documentation is lacking. Last time I tried to upgrade the rocm things and the drivers, and bumped up against an issue that one of the packages tried to write to /data/jenkins_workspace during installation. We have /data as an NFS mount and is not writable even for the root user, so the installation broke down. Worked around it, but there are other issues as well. The order of packages you install can matter as well, I had to purge packages and reinstall them a few times. It worked with TF 1.14.1 but not with 1.14.0 if I recall correctly. The first few iterations are very slow, it seems it needs to find the optimal conv implementation for all kernel sizes separately. Also, the card does not have anything like NVIDIA's tensor cores for half-precision (float16/FP16) computation. It can do FP16 but won't speed up and delivers worse results (in terms of prediction accuracy) than NVIDIA cards running FP16. There is no prediction accuracy difference in FP32. And some other small issues here and there. So be warned and test it yourself on one card before investing in big multi-GPU machines. But I do wish AMD entering the field will benefit us through increased competition in the long run. But at the moment ROCm seems like just a side project for a small team in AMD and they can't yet deliver the streamlined experience we're used to from CUDA.

danielscrubs 7y ago
float16 or bfloat16?
- bonoboTP 7y ago
  I wasn't aware of the difference, but now I looked. I've always used float16, not bfloat16 (tf.float16 is the usual float16), and it causes a large speed boost on Turing NVIDIA cards with tensor cores as opposed to float32. But certainly bfloat16 sounds attractive with its larger dynamic range, as float16 requires workarounds like loss scaling to avoid rounding down to zero when the exponent gets too small.
- bonoboTP 7y ago
  Seems like bfloat16 is only available for TPUs in TensorFlow[0-1]. [0] https://github.com/tensorflow/tensorflow/issues/21612 https://github.com/tensorflow/tensorflow/issues/21612 [1] https://github.com/tensorflow/tensorflow/issues/21317 https://github.com/tensorflow/tensorflow/issues/21317
mtw 7y ago
What card/brand are you using?
- bonoboTP 7y ago
  I'd have to look up the brand, someone else ordered it. Anyways, my complaints may be related partially to me using the xenial (Ubuntu 16.04) repo on bionic (18.04), because the there is no support for 18.04 LTS, even though it came out more than one and a half years ago. I also had to downgrade the kernel from 5.0 to 4.15 as 5.0 is not supported. I also had to force the installation of several 32-bit libraries on this 64-bit system, which took quite some googling and trial and error, otherwise installation didn't succeed. Little things like that.
rrss 7y ago
> it does work at about 90% the speed of an NVIDIA RTX 2080 Ti, but is much cheaper, which can be a good trade-off. Is this 90% of fp32 performance, of 90% of fp16 performance?
- bonoboTP 7y ago
  FP32 on both.
dchichkov 7y ago
Interesting. What benchmark are you running? Getting it within 10% on such drastically different hardware looks wrong to me. And sounds like GPU is not a bottleneck in that particular case. Could it be that the system is CPU bound (with some horrid thread contention in TensorFlow or something)?
- bonoboTP 7y ago
  LambdaLabs' benchmark: https://github.com/lambdal/lambda-tensorflow-benchmark https://github.com/lambdal/lambda-tensorflow-benchmark It's certainly around 10-15% and it's not CPU bound, as the benchmark uses synthetic images without the need for data loading and preprocessing. I cannot rerun them now for precise numbers because my ROCm packages are broken and I don't have the spare hours to fix things. Surely it can be fixed but needs fiddling.
  - dchichkov 7y ago
    You can still be CPU bound, even if the benchmark is using synthetic images. A quick check could be to look if your 2080 Ti is 100% utilized (with nvidia-smi). And calculate percentage of theoretical peak FP32 throughput that you are getting for both cards. It is just that "more normal" result that you would expect to get, would be a drastic differences in performance. Note, Cvikli comment on "May 12" on that GitHub page: * The result with RNN networks on 1 Radeon VII and 1080ti was close to the same * Comparing convolutional performance the 4AMD and 4Nvidia, difference got really huge because of cuDNN for Nvidia cards. We can get more than 10x performance from the 1080Ti than the Radeon VII card. We find this difference in speed a little too big at image recognition cuDNN, I can't believe that this should happen and the hardware shouldn't be able to achieve the same. Note also in that comment, it looks like 4xGPU system just didn't work. It is hard to fully utilize GPU resources. Getting same result normally means that the same bottleneck is being hit. Algorithmic differences could result in drastic performance differences (10x). And as a final point of small things making a difference, note it is 1080Ti in that comment.
    - bonoboTP 7y ago
allovernow 7y ago
>it does work at about 90% the speed of an NVIDIA RTX 2080 Ti Minor nitpick, but it looks like the Radeon vii is in the same tier as the 1080ti, and actually performs worse [1] in benchmarks. 1. https://gpu.userbenchmark.com/Compare/Nvidia-GTX-1080-Ti-vs-AMD-Radeon-VII/3918vs4035 https://gpu.userbenchmark.com/Compare/Nvidia-GTX-1080-Ti-vs-...
- threatripper 7y ago
  The two compare differently depending on the benchmark that is used.
  - allovernow 7y ago
    Do you have any benchmarks which show comparable performance? Two quick searches now but I can only find video game based benchmarks but in every case the 1080TI outperforms the VII, which is kind of surprising and IMO isn't a good sign for ML performance.
    - hgoel 7y ago
      Video games are never a great comparison for raw compute performance. AMD's GCN based architectures have all been very good at compute relative to their graphics performance (recall how it was nearly impossible to find an MSRP AMD GPU during the mining boom). The Radeon VII is pretty ridiculous for compute, aided by the 1TB/s bandwidth due to it's HBM2 stacks.
    - threatripper 7y ago
      https://www.phoronix.com/scan.php?page=article&item=radeon-vii-linux&num=9 https://www.phoronix.com/scan.php?page=article&item=radeon-v... Here it performs VERY well for machine learning. Look at PlaidML.
      - nl 7y ago
        That's using OpenCL on the NVidia card ("Once hitting PlaidML for various machine learning benchmarks with OpenCL"). No one does for machine learning that because the CUDA path is much better.

danielscrubs 7y ago

float16 or bfloat16?

bonoboTP 7y ago
I wasn't aware of the difference, but now I looked. I've always used float16, not bfloat16 (tf.float16 is the usual float16), and it causes a large speed boost on Turing NVIDIA cards with tensor cores as opposed to float32. But certainly bfloat16 sounds attractive with its larger dynamic range, as float16 requires workarounds like loss scaling to avoid rounding down to zero when the exponent gets too small.
bonoboTP 7y ago
Seems like bfloat16 is only available for TPUs in TensorFlow[0-1]. [0] https://github.com/tensorflow/tensorflow/issues/21612 https://github.com/tensorflow/tensorflow/issues/21612 [1] https://github.com/tensorflow/tensorflow/issues/21317 https://github.com/tensorflow/tensorflow/issues/21317

mtw 7y ago

What card/brand are you using?

bonoboTP 7y ago
I'd have to look up the brand, someone else ordered it. Anyways, my complaints may be related partially to me using the xenial (Ubuntu 16.04) repo on bionic (18.04), because the there is no support for 18.04 LTS, even though it came out more than one and a half years ago. I also had to downgrade the kernel from 5.0 to 4.15 as 5.0 is not supported. I also had to force the installation of several 32-bit libraries on this 64-bit system, which took quite some googling and trial and error, otherwise installation didn't succeed. Little things like that.

rrss 7y ago

> it does work at about 90% the speed of an NVIDIA RTX 2080 Ti, but is much cheaper, which can be a good trade-off. Is this 90% of fp32 performance, of 90% of fp16 performance?

bonoboTP 7y ago
FP32 on both.

dchichkov 7y ago

Interesting. What benchmark are you running? Getting it within 10% on such drastically different hardware looks wrong to me. And sounds like GPU is not a bottleneck in that particular case. Could it be that the system is CPU bound (with some horrid thread contention in TensorFlow or something)?

bonoboTP 7y ago
LambdaLabs' benchmark: https://github.com/lambdal/lambda-tensorflow-benchmark https://github.com/lambdal/lambda-tensorflow-benchmark It's certainly around 10-15% and it's not CPU bound, as the benchmark uses synthetic images without the need for data loading and preprocessing. I cannot rerun them now for precise numbers because my ROCm packages are broken and I don't have the spare hours to fix things. Surely it can be fixed but needs fiddling.
- dchichkov 7y ago
  You can still be CPU bound, even if the benchmark is using synthetic images. A quick check could be to look if your 2080 Ti is 100% utilized (with nvidia-smi). And calculate percentage of theoretical peak FP32 throughput that you are getting for both cards. It is just that "more normal" result that you would expect to get, would be a drastic differences in performance. Note, Cvikli comment on "May 12" on that GitHub page: * The result with RNN networks on 1 Radeon VII and 1080ti was close to the same * Comparing convolutional performance the 4AMD and 4Nvidia, difference got really huge because of cuDNN for Nvidia cards. We can get more than 10x performance from the 1080Ti than the Radeon VII card. We find this difference in speed a little too big at image recognition cuDNN, I can't believe that this should happen and the hardware shouldn't be able to achieve the same. Note also in that comment, it looks like 4xGPU system just didn't work. It is hard to fully utilize GPU resources. Getting same result normally means that the same bottleneck is being hit. Algorithmic differences could result in drastic performance differences (10x). And as a final point of small things making a difference, note it is 1080Ti in that comment.
  - bonoboTP 7y ago

allovernow 7y ago

>it does work at about 90% the speed of an NVIDIA RTX 2080 Ti Minor nitpick, but it looks like the Radeon vii is in the same tier as the 1080ti, and actually performs worse [1] in benchmarks. 1. https://gpu.userbenchmark.com/Compare/Nvidia-GTX-1080-Ti-vs-AMD-Radeon-VII/3918vs4035 https://gpu.userbenchmark.com/Compare/Nvidia-GTX-1080-Ti-vs-...

threatripper 7y ago
The two compare differently depending on the benchmark that is used.
- allovernow 7y ago
  Do you have any benchmarks which show comparable performance? Two quick searches now but I can only find video game based benchmarks but in every case the 1080TI outperforms the VII, which is kind of surprising and IMO isn't a good sign for ML performance.
  - hgoel 7y ago
    Video games are never a great comparison for raw compute performance. AMD's GCN based architectures have all been very good at compute relative to their graphics performance (recall how it was nearly impossible to find an MSRP AMD GPU during the mining boom). The Radeon VII is pretty ridiculous for compute, aided by the 1TB/s bandwidth due to it's HBM2 stacks.
  - threatripper 7y ago
    https://www.phoronix.com/scan.php?page=article&item=radeon-vii-linux&num=9 https://www.phoronix.com/scan.php?page=article&item=radeon-v... Here it performs VERY well for machine learning. Look at PlaidML.
    - nl 7y ago
      That's using OpenCL on the NVidia card ("Once hitting PlaidML for various machine learning benchmarks with OpenCL"). No one does for machine learning that because the CUDA path is much better.