5 ms·

A new GPU backend for the TVM stack

70 points by ziheng 9y ago

0xbear 9y ago
How’s the perf per dollar? It’s not enough to “bring” it to AMD, it must be competitive as well.
- make3 9y ago
  the post says they've only focused on coverage so far, not perf. so it's not good yet, I expect. "The current support on ROCm focuses on the functionality coverage. We have already seen promising performance results by simply adopting existing TVM schedules for CUDA backend. For example, you can try running the gemm test script in the TVM repository and see the result. For two types of cards we tested, the current gemm recipe for square matrix multiplication (not yet specifically optimized for AMD GPUs) already achieves 60% to 65% of peak performance. This is already a promising start, as it is very hard to optimize performance to get to peak and we did not yet apply AMD GPU specific optimizations. We are starting to look at performance optimization and we expect more improvement to come"
  - 0xbear 9y ago
    One thing I don’t get is why AMD is not paying attention to the deep learning market. They could easily turn things around within a year with a team of 10 solid engineers, yet they seem to be choosing not to. I’m pretty sure the chips are capable of delivering the goods, it’s just a pain to get to the goods right now.
    - DannyBee 9y ago
      There are way too many framework targets there be effective here yet. Any of these chips requires effective support from the frameworks, no matter what they do. Peanut buttering 10 engineers across all the frameworks to try to get market share is unlikely to work. You'd either need to dedicate an overwhelming team to shock and awe this (IE support and performance for AMD chips is best in class for all frameworks, period). This is 50 people, at least, who all work really well together, if you want to deliver it in the next year. Or, you can bide your time, dedicate those resources to better and better hardware, and when the frameworks field shakes out a bit, have a better shot. None of these people care about running on nvidia gpus (and in fact, the vendors pushing them don't want to be locked in either), so your main concern there is hand tuning and cuda kernel integration they do. The switching cost is something but not huge, and isn't increasing that much over time (unlike the x86 switching cost, for example). So waiting doesn't lose you a lot. So in the meantime, you make two bets: 1. You try to take over the intermediate IR of frameworks, and make it good enough that people stop writing hand tuned nvidia kernels. This is unlikely to work out, but worth a shot. 2. You slowly decide what customers you want, look at what they are writing hand-tuned kernels for, and try to tackle making the frameworks they use good enough to not need it. (in case #1 doesn't work out).
      - 0xbear 9y ago
        Don’t peanut butter then. Put 3 on TF, 3 on PyTorch, and the remaining 4 on shared library of some sort. You’re right that doing 10 things at once is a recipe for failure, but the reality is, majority of frameworks don’t really matter all that much, and if a couple of solid integrations existed, they could just reuse that work on their own.
        dragontamer 9y ago
        NVidia doesn't have just 3 or 4 engineers in these tasks. When you consider that NVidia just took Google's TPU concept, and basically cloned it into their upcoming GPU ("Tensor Cores"), you realize that NVidia is really pouring millions, maybe hundreds-of-millions, of dollars into AI. AMD has a number of initiatives that haven't panned out as well as NVidia's AI investment. AMD's "HSA" technology is actually quite interesting, although unpopular. AMD's push for HSA has made it faster at Video Rendering tasks (see Blender for instance) and random tasks like GPGPU-accelerated WinRAR decompression or LibreOffice Spreadsheets. IIRC, AMD Vega beats NVidia Titan XP in Creo and Solidworks benchmarks (CAD programs). So its not like AMD is sleeping on its laurels here, they're just focusing on other, still profitable, corners of the market. Of course, the current is behind AI, Tensors, and NVidia at the moment. AMD can't afford to fall further behind. The current trend is AI and Machine Learning, and it seems reasonable for AMD to at least get PyTorch running on AMD cards (if not "beating" NVidia, but at least they can play along).
        sitkack 9y ago
        HSA is also a longer term play. The pieces are in place for AMD literally jump 1-2 years ahead of Intel and Nvidia.
        DannyBee 9y ago
        I really don't think 3 people is going to cover this. Like literally, NVidia will just pay 3 people to do nothing but argue with those 3 people on mailing lists and keep them from landing patches at a reasonable rate. It would cost nothing compared to the benefit of slowing your only competitor down in a space worth this much. (and i've literally seen not-great companies do this to people's open source projects, so ...)
- dragontamer 9y ago
  Hmm, with the "Tensor Cores" of NVidia's next-generation Volta coming in, I'd bet that NVidia cards will be faster in machine learning tasks. But then again: AMD Vega 56 / 64 have HBM2 and are under $1000. IIRC, the Vega Frontier Edition is $999 and 16GB of HBM2 at 480GB/s theoretical bandwidth. NVidia also has an offering with high-speed HBM2 RAM: The Tesla P100, but its way more expensive: $7000 each. I dunno if there are major benefits of HBM2 over GDDR5x however. Just listing off numbers here. The Titan Xp apparently has more bandwidth from the GDDR5x RAM for example, although the Titan XP is still more expensive than the Vega 64. ------------ If there is some problem that is global memory-bandwidth constrained, then it might be better to run it on AMD Vega 64. After all, you can pretty much afford 7x AMD Vega Frontier editions than the NVidia P100. Obviously, this very much depends on your workload.
  - dharma1 9y ago
    Yep, Nvidia is quoting 125 TFLOPs mixed precision on V100, boosted by Tensor Cores. Vega 64 can in theory do 25 TFLOPs half precision. But as you say there's a large price difference too. For a market segment that needs 1-8 GPU rigs for ML on a low budget AMD could kill it if they invested in software support and kernel optimisation. For servers and large scale training, unless AMD has some ML specialised cores in the pipeline, Nvidia Volta and Google TPUs have a serious lead.
    - mtgx 9y ago
      Aren't the Tensor Cores mostly for inference, not training?
      - dragontamer 9y ago
        The Volta Tensor Cores aren't released yet, and I haven't played with Google's TPUs at all. But NVidia markets Tensor cores as: > New Tensor Cores designed specifically for deep learning deliver up to 12x higher peak TFLOP/ss for training, and 6x higher peak TFLOP/s for inference https://devblogs.nvidia.com/parallelforall/cuda-9-features-revealed/ https://devblogs.nvidia.com/parallelforall/cuda-9-features-r... I wouldn't be surprised if they were inflating the numbers slightly, as is common in a lot of marketing material.
railgun2space 9y ago
New usage for RX480/580 miner?
muli 9y ago
Some quick calculations about TFLOPS per dollar on GEMM. Since NVIDIA's volta consumer card is not out yet, I used Titan Xp as the reference card. I grabbed prices from wikipedia, and assume TVM reaches 64% peak perf on Vega and 90% peak perf on Titan Xp: Radeon RX Vega 64: 12.6TFLOPS * 65% / $499 = 0.01638 TFLOPS/$ Pascal Titan Xp: 12TFLOPS * 90% / $1200 = 0.009 TFLOPS/$ So Vega outperforms a lot here.
- dragontamer 9y ago
  I'd assume that Video RAM is important though. The Titan XP has 12GB of RAM, while the $500 Vega 64 only has 8GB of RAM. A better comparison is probably Vega Frontier with 16GB of RAM for $1000. If you're doing heavy compute, you're probably gonna need a ton of RAM to go with it.
  - jamilbk 9y ago
    That's a fair assumption I think. It will be interesting to see whether Vega's high-bandwidth cache controller (HBCC) will help nullify this difference if implemented in ML frameworks.
  - dharma1 9y ago
    Vega is meant to have someting called HBCC - that effectively allows you to use system RAM or even SSD as GPU memory for VERY large datasets. I have no idea if it would be fast enough for deep learning workloads over PCIe https://www.pcper.com/news/General-Tech/AMDs-HBCC-you-and-me https://www.pcper.com/news/General-Tech/AMDs-HBCC-you-and-me
  - microcolonel 9y ago
    Vega has half-precision floats as well though, which (with a seemingly negligible loss in precision) in combination with their HBCC (transparent main memory DMA) should more than make up for the lack of memory on the Vega 64, and in the case of Frontier Edition, all the more so.
- dharma1 9y ago
  Another thing to note is that you can in theory train with half precision without sacrificing much accuracy. There isn't that much support for it yet, for AMD anyway that I've seen, but Vega 64 has 2x speed half precision (25 TFLOPS) on paper. Out of Nvidia cards only Tesla and Volta have 2x (or more) speed mixed precision ops (in Voltas case actually much faster because of the Tensor Cores) but they are in a very different price bracket.

70 points by ziheng 9y ago

0xbear 9y ago

How’s the perf per dollar? It’s not enough to “bring” it to AMD, it must be competitive as well.

make3 9y ago
the post says they've only focused on coverage so far, not perf. so it's not good yet, I expect. "The current support on ROCm focuses on the functionality coverage. We have already seen promising performance results by simply adopting existing TVM schedules for CUDA backend. For example, you can try running the gemm test script in the TVM repository and see the result. For two types of cards we tested, the current gemm recipe for square matrix multiplication (not yet specifically optimized for AMD GPUs) already achieves 60% to 65% of peak performance. This is already a promising start, as it is very hard to optimize performance to get to peak and we did not yet apply AMD GPU specific optimizations. We are starting to look at performance optimization and we expect more improvement to come"
- 0xbear 9y ago
  One thing I don’t get is why AMD is not paying attention to the deep learning market. They could easily turn things around within a year with a team of 10 solid engineers, yet they seem to be choosing not to. I’m pretty sure the chips are capable of delivering the goods, it’s just a pain to get to the goods right now.
  - DannyBee 9y ago
    There are way too many framework targets there be effective here yet. Any of these chips requires effective support from the frameworks, no matter what they do. Peanut buttering 10 engineers across all the frameworks to try to get market share is unlikely to work. You'd either need to dedicate an overwhelming team to shock and awe this (IE support and performance for AMD chips is best in class for all frameworks, period). This is 50 people, at least, who all work really well together, if you want to deliver it in the next year. Or, you can bide your time, dedicate those resources to better and better hardware, and when the frameworks field shakes out a bit, have a better shot. None of these people care about running on nvidia gpus (and in fact, the vendors pushing them don't want to be locked in either), so your main concern there is hand tuning and cuda kernel integration they do. The switching cost is something but not huge, and isn't increasing that much over time (unlike the x86 switching cost, for example). So waiting doesn't lose you a lot. So in the meantime, you make two bets: 1. You try to take over the intermediate IR of frameworks, and make it good enough that people stop writing hand tuned nvidia kernels. This is unlikely to work out, but worth a shot. 2. You slowly decide what customers you want, look at what they are writing hand-tuned kernels for, and try to tackle making the frameworks they use good enough to not need it. (in case #1 doesn't work out).
    - 0xbear 9y ago
      Don’t peanut butter then. Put 3 on TF, 3 on PyTorch, and the remaining 4 on shared library of some sort. You’re right that doing 10 things at once is a recipe for failure, but the reality is, majority of frameworks don’t really matter all that much, and if a couple of solid integrations existed, they could just reuse that work on their own.
      - dragontamer 9y ago
        NVidia doesn't have just 3 or 4 engineers in these tasks. When you consider that NVidia just took Google's TPU concept, and basically cloned it into their upcoming GPU ("Tensor Cores"), you realize that NVidia is really pouring millions, maybe hundreds-of-millions, of dollars into AI. AMD has a number of initiatives that haven't panned out as well as NVidia's AI investment. AMD's "HSA" technology is actually quite interesting, although unpopular. AMD's push for HSA has made it faster at Video Rendering tasks (see Blender for instance) and random tasks like GPGPU-accelerated WinRAR decompression or LibreOffice Spreadsheets. IIRC, AMD Vega beats NVidia Titan XP in Creo and Solidworks benchmarks (CAD programs). So its not like AMD is sleeping on its laurels here, they're just focusing on other, still profitable, corners of the market. Of course, the current is behind AI, Tensors, and NVidia at the moment. AMD can't afford to fall further behind. The current trend is AI and Machine Learning, and it seems reasonable for AMD to at least get PyTorch running on AMD cards (if not "beating" NVidia, but at least they can play along).
        sitkack 9y ago
        HSA is also a longer term play. The pieces are in place for AMD literally jump 1-2 years ahead of Intel and Nvidia.
      - DannyBee 9y ago
        I really don't think 3 people is going to cover this. Like literally, NVidia will just pay 3 people to do nothing but argue with those 3 people on mailing lists and keep them from landing patches at a reasonable rate. It would cost nothing compared to the benefit of slowing your only competitor down in a space worth this much. (and i've literally seen not-great companies do this to people's open source projects, so ...)
dragontamer 9y ago
Hmm, with the "Tensor Cores" of NVidia's next-generation Volta coming in, I'd bet that NVidia cards will be faster in machine learning tasks. But then again: AMD Vega 56 / 64 have HBM2 and are under $1000. IIRC, the Vega Frontier Edition is $999 and 16GB of HBM2 at 480GB/s theoretical bandwidth. NVidia also has an offering with high-speed HBM2 RAM: The Tesla P100, but its way more expensive: $7000 each. I dunno if there are major benefits of HBM2 over GDDR5x however. Just listing off numbers here. The Titan Xp apparently has more bandwidth from the GDDR5x RAM for example, although the Titan XP is still more expensive than the Vega 64. ------------ If there is some problem that is global memory-bandwidth constrained, then it might be better to run it on AMD Vega 64. After all, you can pretty much afford 7x AMD Vega Frontier editions than the NVidia P100. Obviously, this very much depends on your workload.
- dharma1 9y ago
  Yep, Nvidia is quoting 125 TFLOPs mixed precision on V100, boosted by Tensor Cores. Vega 64 can in theory do 25 TFLOPs half precision. But as you say there's a large price difference too. For a market segment that needs 1-8 GPU rigs for ML on a low budget AMD could kill it if they invested in software support and kernel optimisation. For servers and large scale training, unless AMD has some ML specialised cores in the pipeline, Nvidia Volta and Google TPUs have a serious lead.
  - mtgx 9y ago
    Aren't the Tensor Cores mostly for inference, not training?
    - dragontamer 9y ago
      The Volta Tensor Cores aren't released yet, and I haven't played with Google's TPUs at all. But NVidia markets Tensor cores as: > New Tensor Cores designed specifically for deep learning deliver up to 12x higher peak TFLOP/ss for training, and 6x higher peak TFLOP/s for inference https://devblogs.nvidia.com/parallelforall/cuda-9-features-revealed/ https://devblogs.nvidia.com/parallelforall/cuda-9-features-r... I wouldn't be surprised if they were inflating the numbers slightly, as is common in a lot of marketing material.

railgun2space 9y ago

New usage for RX480/580 miner?

muli 9y ago

Some quick calculations about TFLOPS per dollar on GEMM. Since NVIDIA's volta consumer card is not out yet, I used Titan Xp as the reference card. I grabbed prices from wikipedia, and assume TVM reaches 64% peak perf on Vega and 90% peak perf on Titan Xp: Radeon RX Vega 64: 12.6TFLOPS * 65% / $499 = 0.01638 TFLOPS/$ Pascal Titan Xp: 12TFLOPS * 90% / $1200 = 0.009 TFLOPS/$ So Vega outperforms a lot here.

dragontamer 9y ago
I'd assume that Video RAM is important though. The Titan XP has 12GB of RAM, while the $500 Vega 64 only has 8GB of RAM. A better comparison is probably Vega Frontier with 16GB of RAM for $1000. If you're doing heavy compute, you're probably gonna need a ton of RAM to go with it.
- jamilbk 9y ago
  That's a fair assumption I think. It will be interesting to see whether Vega's high-bandwidth cache controller (HBCC) will help nullify this difference if implemented in ML frameworks.
- dharma1 9y ago
  Vega is meant to have someting called HBCC - that effectively allows you to use system RAM or even SSD as GPU memory for VERY large datasets. I have no idea if it would be fast enough for deep learning workloads over PCIe https://www.pcper.com/news/General-Tech/AMDs-HBCC-you-and-me https://www.pcper.com/news/General-Tech/AMDs-HBCC-you-and-me
- microcolonel 9y ago
  Vega has half-precision floats as well though, which (with a seemingly negligible loss in precision) in combination with their HBCC (transparent main memory DMA) should more than make up for the lack of memory on the Vega 64, and in the case of Frontier Edition, all the more so.
dharma1 9y ago
Another thing to note is that you can in theory train with half precision without sacrificing much accuracy. There isn't that much support for it yet, for AMD anyway that I've seen, but Vega 64 has 2x speed half precision (25 TFLOPS) on paper. Out of Nvidia cards only Tesla and Volta have 2x (or more) speed mixed precision ops (in Voltas case actually much faster because of the Tensor Cores) but they are in a very different price bracket.