8 ms·

AMD's Freshly-Baked MI350: An Interview with the Chief Architect

128 points by pella 1y ago

pella 1y ago
FP6: "Alan: Sure, yep, so one of the things that we felt like on MI350 in this timeframe, that it's going into the market and the current state of AI... we felt like that FP6 is a format that has potential to not only be used for inferencing, but potentially for training. And so we wanted to make sure that the capabilities for FP6 were class-leading relative to... what others maybe would have been implementing, or have implemented. And so, as you know, it's a long lead time to design hardware, so we were thinking about this years ago and wanted to make sure that MI350 had leadership in FP6 performance. So we made a decision to implement the FP6 data path at the same throughput as the FP4 data path. Of course, we had to take on a little bit more hardware in order to do that. FP6 has a few more bits, obviously, that's why it's called FP6. But we were able to do that within the area of constraints that we had in the matrix engine, and do that in a very power- and area-efficient way.
- kristianp 1y ago
  Will 1.58 bits be in the MI400? Or is it not established as a widely useful technology yet? See https://arxiv.org/abs/2402.17764 https://arxiv.org/abs/2402.17764
- treesciencebot 1y ago
  the main question is going to be software stack. NVIDIA is already shipping NVFP4 kernels and perf is looking good. It took a really long time after MI300X's that the FP8 kernels were OK (not even good, compared to almost perfect FP8 support in NVIDIA side of things). I will doubt that they will be able to reach %60-70 of the FLOPs in majority of the workloads (unless they hand craft and tune a specific GEMM kernel for their benchmark shape). But would be happy to be proven wrong, and go buy a bunch of them
  - pella 1y ago
    (related) Tinygrad: "We've been negotiating a $2M contract to get AMD on MLPerf, but one of the sticking points has been confidentiality. Perhaps posting the deliverables on X will help legal to get in the spirit of open source!" "Contract is signed! No confidentiality, AMD has leadership that's capable of acting. Let's make this training run happen, we work in public on our Discord. " https://x.com/__tinygrad__/status/1935364905949110532 https://x.com/__tinygrad__/status/1935364905949110532
    - LeonM 1y ago
      It still amazes me that George/Tinycorp somehow seems to get AMD on board every time, and being blissfully unaware that they are a very small player. See for example top comment here [0]. Don't get me wrong, I think it's impressive what he achieved so far, and I hope tiny can stay competitive in this market. [0] https://news.ycombinator.com/item?id=36193625 https://news.ycombinator.com/item?id=36193625
      - ryao 1y ago
        Most of those willing to work with AMD are very small players (with some notable exceptions). They are likely hopeful that the small players will grow.
      - imtringued 1y ago
        People get on board with George Hotz because they share the frustration of using ROCm on consumer GPUs, where the experience has been insultingly dreadful to the point where I decided to postpone buying new AMD GPUs for at least a decade. I'm not quite sure why he decided to pivot to datacenter GPUs where AMD has shown at least some commitment to ROCm. The intersection between users of tinygrad and people who use MI350s should essentially be George himself and no one else.
      - roenxi 1y ago
        That top comment doesn't seem to have engaged completely with the context here. AMD fumbled trillions of dollars of value creation by mis-identifying what their hardware was for. Or perhaps it is more correct to say by being too dogmatic about what their hardware is for. They weren't in a position to be picky. They had a choice - they could continue making trillion-dollar mistakes until their board got sacked and the exec team replaced. Or they could maybe listen to some of the people who were technically correct regardless of their size in the market. George is just some dude and I doubt AMD paid him much attention anywhere through this saga, but AMD had screwed up to the point where he could give some precise commentary about how they'd managed to duck and weave to avoid the overwhelming torrent of money trying to rush in and buy graphics hardware. They should make some time in their busy schedules to talk with people like that.
tedunangst 1y ago
A solid 40% of George's questions were deemed great. (Not counting some fluff like what's your job.)
deadbabe 1y ago
Will AMD catch up to Nvidia?
- wmf 1y ago
  Yes, if they can ship on time.
- mobilio 1y ago
  If they improve software quality and providing some low budget versions then - Yes.
- zombiwoof 1y ago
  They don’t care to catch up.
- AzzyHN 1y ago
  On the consumer side, almost certainly not. Nvidia is a HUGE brand name, it doesn't matter how good and cheap AMD makes their consumer GPUs, people will buy Nvidia GPUs for the brand and prebuilts will stick with Nvidia for the name. For AI chips... also probably not, unless AMD can compete with CUDA (or CUDA becomes irrelevant)
  - frje1400 1y ago
    > On the consumer side, almost certainly not. Nvidia is a HUGE brand name, it doesn't matter how good and cheap AMD makes their consumer GPUs, people will buy Nvidia GPUs for the brand and prebuilts will stick with Nvidia for the name. I think that AMD could do it, but they choose not to. If you look at their most recent lineup of cards (various SKUs of 9070 and 9060), they are not so much better than Nvidia at each price point that they are a must buy. They even released an outright bad card a few weeks ago (9060 8 GB). I assume that the rationale is that even if they could somehow dominate the gamer market, that is peanuts compared to the potential in AI.
  - jillesvangurp 1y ago
    Actually both Xbox and Playstation use AMD GPUs; and so does the Steam Deck. So there's that. For the narrow niche of gaming PCs, I think there are a lot of kids buying what they can afford and getting creative about what works. AMD isn't doing horrible in that market either. And for AI, CUDA is already becoming less relevant. Most of the big players use chips by their own designs: Google has its TPUs, Amazon has some in house designs, Apple has it's own CPU/GPU line and doesn't even support anything nvidia at this point, MS do their own thing for Azure, etc. You are basically making the Intel will stay big because Intel is big for Nvidia. Except of course that stopped being true for Intel. They are still largish. But a lot of data centers are transitioning to ARM CPUs. They lost Apple as a customer. And there are now some decent windows laptops using ARM CPUs as well.
behnamoh 1y ago
Does this also ship only in x8 batches? I really liked MI300 and could afford one of them for my research, but they only come in batches of x8 in a server rack, so I decided to buy an RTX Pro 6000.
- latchkey 1y ago
  It isn't 8x batches. It is 8 OAMs on a UBB. The UBB is what enables the 8x to communicate with each other over infinity fabric. If you don't need 8, then that's exactly why we offer 1xMI300x VM's.
  - ryao 1y ago
    A number of people want to purchase their own hardware, not rent cloud hardware. I recently purchased a RTX PRO 6000 for the same reason, despite having the option of renting a B200 VM for $1.49 an hour from DeepInfra until the end of June.
    - latchkey 1y ago
      True, but as time goes on, it will become a wider and wider gap between what is deployed in DC’s and what you can run at home. We see it now with 8x UBB and it will get worse with direct liquid cooling and larger power requirements. Mi300x is 700w. Mi355 is 1200w. Mi450 will be even more. Certainly amd should make some consumer grade stuff, but they won’t stop on the enterprise side either. Your only option to get super computer level compute, will be to rent it.
      - ryao 1y ago
        The higher power requirements of datacenter GPUs are mostly from running hardware well past a reasonable point on the efficiency curve to eke slightly higher generational increases in performance. Nvidia for example has three versions of the RTX PRO 6000. One runs at 600W while the other two run at 300W. The main differences are the power target and the cooling solutions. If I recall correctly, the performance difference between them is less than 20%, despite a 50% reduction in power consumption, for what is effectively the same hardware. This can be confirmed by changing the power target of the 600W version to 300W and benchmarking the before and after. Plenty of people have made similar observations of AMD’s hardware. That said, I am confident that Nvidia will continue serve those of us who want our own hardware.
teleforce 1y ago
This 8-combo MI350 is a beauty with 2304 GB VRAM of HMB3E memory on each UBB [1]. [1] This is the AMD Instinct MI350: https://www.servethehome.com/this-is-the-amd-instinct-mi350/ https://www.servethehome.com/this-is-the-amd-instinct-mi350/
jonfromsf 1y ago
NVDAs advantage is software, not just hardware. Would be amazing to have a competitive market but better hardware won't be enough to make it happen.
AbuAssar 1y ago
If MI350 employs CDNA, which is based on the VEGA (GCN) architecture, does that imply that MI400, when introduced next year, will skip the 2020 GCN and directly transition to RDNA 5 equivalent?
- pella 1y ago
  2026 - MI400X - CDNA 5 - UALink/IF - Helios - HBM Bandwidth: 1,400 TB/s https://www.tomshardware.com/pc-components/gpus/amd-says-instinct-mi400x-gpu-is-10x-faster-than-mi300x-will-power-helios-rack-scale-system-with-epyc-venice-cpus https://www.tomshardware.com/pc-components/gpus/amd-says-ins...
- almostgotcaught 1y ago
  it's all just called gcn now https://llvm.org/docs/AMDGPUUsage.html#id38 https://llvm.org/docs/AMDGPUUsage.html#id38
  - adrian_b 1y ago
    The identification of the AMD GPU architectures has always been extremely confusing, with tons of different names meaning the same thing and with some names, like GCN, used for several very different things. The table linked by you is good for revealing the meaning of a part of the many AMD code names.
- rfv6723 1y ago
  RDNA is a dead-end. AMD went down the wrong path by focusing on traditional rendering instead of machine learning. I think future AMD consumer GPUs would go back to GCN.
- adrian_b 1y ago
  There will be no RDNA 5, but a unified UDNA, replacing both CDNA and RDNA. AMD has not disclosed how they will achieve the unification, but it is far more likely that the unified architecture will be an evolution of CDNA 4, i.e. an evolution of the old GCN, than an evolution of RDNA, because basing the unified architecture on CDNA/GCN, will create less problems in software porting than basing it on RDNA 4 or 3. The unified architecture will probably take some features from RDNA only when they are hard to emulate on CDNA. While the first generation of RDNA has been acclaimed for having a good performance increase in games over the previous GCN-based Vega, it is not clear how much of that performance increase was due to RDNA being better for games and how much to the fact that the first RDNA GPUs happened to have double-width vector pipelines in comparison with the previous GCN GPUs, thus double throughput per clock cycle and per CU (32 FP32 operations/cycle vs. 16 FP32 operations/cycle). It is possible that RDNA was not really a better architecture, but omitting some of the hardware that was rarely used in games from GCN allowed the implementation of the wider pipelines that were more useful for games. So RDNA was a better compromise for the technology available at that time, not necessarily better in other circumstances.
latchkey 1y ago
I've got the MI300x and I can't wait to deploy a bunch of the MI355's.

128 points by pella 1y ago

FP6: "Alan: Sure, yep, so one of the things that we felt like on MI350 in this timeframe, that it's going into the market and the current state of AI... we felt like that FP6 is a format that has potential to not only be used for inferencing, but potentially for training. And so we wanted to make sure that the capabilities for FP6 were class-leading relative to... what others maybe would have been implementing, or have implemented. And so, as you know, it's a long lead time to design hardware, so we were thinking about this years ago and wanted to make sure that MI350 had leadership in FP6 performance. So we made a decision to implement the FP6 data path at the same throughput as the FP4 data path. Of course, we had to take on a little bit more hardware in order to do that. FP6 has a few more bits, obviously, that's why it's called FP6. But we were able to do that within the area of constraints that we had in the matrix engine, and do that in a very power- and area-efficient way.

kristianp 1y ago
Will 1.58 bits be in the MI400? Or is it not established as a widely useful technology yet? See https://arxiv.org/abs/2402.17764 https://arxiv.org/abs/2402.17764
treesciencebot 1y ago
the main question is going to be software stack. NVIDIA is already shipping NVFP4 kernels and perf is looking good. It took a really long time after MI300X's that the FP8 kernels were OK (not even good, compared to almost perfect FP8 support in NVIDIA side of things). I will doubt that they will be able to reach %60-70 of the FLOPs in majority of the workloads (unless they hand craft and tune a specific GEMM kernel for their benchmark shape). But would be happy to be proven wrong, and go buy a bunch of them
- pella 1y ago
  (related) Tinygrad: "We've been negotiating a $2M contract to get AMD on MLPerf, but one of the sticking points has been confidentiality. Perhaps posting the deliverables on X will help legal to get in the spirit of open source!" "Contract is signed! No confidentiality, AMD has leadership that's capable of acting. Let's make this training run happen, we work in public on our Discord. " https://x.com/__tinygrad__/status/1935364905949110532 https://x.com/__tinygrad__/status/1935364905949110532
  - LeonM 1y ago
    It still amazes me that George/Tinycorp somehow seems to get AMD on board every time, and being blissfully unaware that they are a very small player. See for example top comment here [0]. Don't get me wrong, I think it's impressive what he achieved so far, and I hope tiny can stay competitive in this market. [0] https://news.ycombinator.com/item?id=36193625 https://news.ycombinator.com/item?id=36193625
    - ryao 1y ago
      Most of those willing to work with AMD are very small players (with some notable exceptions). They are likely hopeful that the small players will grow.
    - imtringued 1y ago
      People get on board with George Hotz because they share the frustration of using ROCm on consumer GPUs, where the experience has been insultingly dreadful to the point where I decided to postpone buying new AMD GPUs for at least a decade. I'm not quite sure why he decided to pivot to datacenter GPUs where AMD has shown at least some commitment to ROCm. The intersection between users of tinygrad and people who use MI350s should essentially be George himself and no one else.
    - roenxi 1y ago
      That top comment doesn't seem to have engaged completely with the context here. AMD fumbled trillions of dollars of value creation by mis-identifying what their hardware was for. Or perhaps it is more correct to say by being too dogmatic about what their hardware is for. They weren't in a position to be picky. They had a choice - they could continue making trillion-dollar mistakes until their board got sacked and the exec team replaced. Or they could maybe listen to some of the people who were technically correct regardless of their size in the market. George is just some dude and I doubt AMD paid him much attention anywhere through this saga, but AMD had screwed up to the point where he could give some precise commentary about how they'd managed to duck and weave to avoid the overwhelming torrent of money trying to rush in and buy graphics hardware. They should make some time in their busy schedules to talk with people like that.

tedunangst 1y ago

A solid 40% of George's questions were deemed great. (Not counting some fluff like what's your job.)

deadbabe 1y ago

Will AMD catch up to Nvidia?

wmf 1y ago
Yes, if they can ship on time.
mobilio 1y ago
If they improve software quality and providing some low budget versions then - Yes.
zombiwoof 1y ago
They don’t care to catch up.
AzzyHN 1y ago
On the consumer side, almost certainly not. Nvidia is a HUGE brand name, it doesn't matter how good and cheap AMD makes their consumer GPUs, people will buy Nvidia GPUs for the brand and prebuilts will stick with Nvidia for the name. For AI chips... also probably not, unless AMD can compete with CUDA (or CUDA becomes irrelevant)
- frje1400 1y ago
  > On the consumer side, almost certainly not. Nvidia is a HUGE brand name, it doesn't matter how good and cheap AMD makes their consumer GPUs, people will buy Nvidia GPUs for the brand and prebuilts will stick with Nvidia for the name. I think that AMD could do it, but they choose not to. If you look at their most recent lineup of cards (various SKUs of 9070 and 9060), they are not so much better than Nvidia at each price point that they are a must buy. They even released an outright bad card a few weeks ago (9060 8 GB). I assume that the rationale is that even if they could somehow dominate the gamer market, that is peanuts compared to the potential in AI.
- jillesvangurp 1y ago
  Actually both Xbox and Playstation use AMD GPUs; and so does the Steam Deck. So there's that. For the narrow niche of gaming PCs, I think there are a lot of kids buying what they can afford and getting creative about what works. AMD isn't doing horrible in that market either. And for AI, CUDA is already becoming less relevant. Most of the big players use chips by their own designs: Google has its TPUs, Amazon has some in house designs, Apple has it's own CPU/GPU line and doesn't even support anything nvidia at this point, MS do their own thing for Azure, etc. You are basically making the Intel will stay big because Intel is big for Nvidia. Except of course that stopped being true for Intel. They are still largish. But a lot of data centers are transitioning to ARM CPUs. They lost Apple as a customer. And there are now some decent windows laptops using ARM CPUs as well.

behnamoh 1y ago

Does this also ship only in x8 batches? I really liked MI300 and could afford one of them for my research, but they only come in batches of x8 in a server rack, so I decided to buy an RTX Pro 6000.

latchkey 1y ago
It isn't 8x batches. It is 8 OAMs on a UBB. The UBB is what enables the 8x to communicate with each other over infinity fabric. If you don't need 8, then that's exactly why we offer 1xMI300x VM's.
- ryao 1y ago
  A number of people want to purchase their own hardware, not rent cloud hardware. I recently purchased a RTX PRO 6000 for the same reason, despite having the option of renting a B200 VM for $1.49 an hour from DeepInfra until the end of June.
  - latchkey 1y ago
    True, but as time goes on, it will become a wider and wider gap between what is deployed in DC’s and what you can run at home. We see it now with 8x UBB and it will get worse with direct liquid cooling and larger power requirements. Mi300x is 700w. Mi355 is 1200w. Mi450 will be even more. Certainly amd should make some consumer grade stuff, but they won’t stop on the enterprise side either. Your only option to get super computer level compute, will be to rent it.
    - ryao 1y ago
      The higher power requirements of datacenter GPUs are mostly from running hardware well past a reasonable point on the efficiency curve to eke slightly higher generational increases in performance. Nvidia for example has three versions of the RTX PRO 6000. One runs at 600W while the other two run at 300W. The main differences are the power target and the cooling solutions. If I recall correctly, the performance difference between them is less than 20%, despite a 50% reduction in power consumption, for what is effectively the same hardware. This can be confirmed by changing the power target of the 600W version to 300W and benchmarking the before and after. Plenty of people have made similar observations of AMD’s hardware. That said, I am confident that Nvidia will continue serve those of us who want our own hardware.

teleforce 1y ago

This 8-combo MI350 is a beauty with 2304 GB VRAM of HMB3E memory on each UBB [1]. [1] This is the AMD Instinct MI350: https://www.servethehome.com/this-is-the-amd-instinct-mi350/ https://www.servethehome.com/this-is-the-amd-instinct-mi350/

jonfromsf 1y ago

NVDAs advantage is software, not just hardware. Would be amazing to have a competitive market but better hardware won't be enough to make it happen.

AbuAssar 1y ago

If MI350 employs CDNA, which is based on the VEGA (GCN) architecture, does that imply that MI400, when introduced next year, will skip the 2020 GCN and directly transition to RDNA 5 equivalent?

pella 1y ago
2026 - MI400X - CDNA 5 - UALink/IF - Helios - HBM Bandwidth: 1,400 TB/s https://www.tomshardware.com/pc-components/gpus/amd-says-instinct-mi400x-gpu-is-10x-faster-than-mi300x-will-power-helios-rack-scale-system-with-epyc-venice-cpus https://www.tomshardware.com/pc-components/gpus/amd-says-ins...
almostgotcaught 1y ago
it's all just called gcn now https://llvm.org/docs/AMDGPUUsage.html#id38 https://llvm.org/docs/AMDGPUUsage.html#id38
- adrian_b 1y ago
  The identification of the AMD GPU architectures has always been extremely confusing, with tons of different names meaning the same thing and with some names, like GCN, used for several very different things. The table linked by you is good for revealing the meaning of a part of the many AMD code names.
rfv6723 1y ago
RDNA is a dead-end. AMD went down the wrong path by focusing on traditional rendering instead of machine learning. I think future AMD consumer GPUs would go back to GCN.
adrian_b 1y ago
There will be no RDNA 5, but a unified UDNA, replacing both CDNA and RDNA. AMD has not disclosed how they will achieve the unification, but it is far more likely that the unified architecture will be an evolution of CDNA 4, i.e. an evolution of the old GCN, than an evolution of RDNA, because basing the unified architecture on CDNA/GCN, will create less problems in software porting than basing it on RDNA 4 or 3. The unified architecture will probably take some features from RDNA only when they are hard to emulate on CDNA. While the first generation of RDNA has been acclaimed for having a good performance increase in games over the previous GCN-based Vega, it is not clear how much of that performance increase was due to RDNA being better for games and how much to the fact that the first RDNA GPUs happened to have double-width vector pipelines in comparison with the previous GCN GPUs, thus double throughput per clock cycle and per CU (32 FP32 operations/cycle vs. 16 FP32 operations/cycle). It is possible that RDNA was not really a better architecture, but omitting some of the hardware that was rarely used in games from GCN allowed the implementation of the wider pipelines that were more useful for games. So RDNA was a better compromise for the technology available at that time, not necessarily better in other circumstances.

latchkey 1y ago

I've got the MI300x and I can't wait to deploy a bunch of the MI355's.