4 ms·

Relevant, we deployed Lamini on hundreds of MI200 GPUs. Lisa tweet: https://x.com/LisaSu/status/1706707561809105331?s=20 https://x.com/LisaSu/status/1706707561

Relevant, we deployed Lamini on hundreds of MI200 GPUs. Lisa tweet: https://x.com/LisaSu/status/1706707561809105331?s=20 https://x.com/LisaSu/status/1706707561809105331?s=20 Lamini tweet: https://x.com/realSharonZhou/status/1706701693684154766?s=20 https://x.com/realSharonZhou/status/1706701693684154766?s=20 Blog: https://www.lamini.ai/blog/lamini-amd-paving-the-road-to-gpu-rich-enterprise-llms https://www.lamini.ai/blog/lamini-amd-paving-the-road-to-gpu... Register: https://www.theregister.com/2023/09/26/amd_instinct_ai_lamini/ https://www.theregister.com/2023/09/26/amd_instinct_ai_lamin... CRN: https://www.crn.com/news/components-peripherals/llm-startup-embraces-amd-gpus-says-rocm-has-parity-with-nvidia-s-cuda-platform https://www.crn.com/news/components-peripherals/llm-startup-... The hard part about using any AI Chips other than NVIDIA has been software. ROCm is finally at the point where it can train and deploy LLMs like Llama 2 in production. If you want to try this out, one big issue is that software support is hugely different on Instinct vs Radeon. I think AMD will fix this eventually, but today you need to use Instinct. We will post more information explaining how this works in the next few weeks. The middle section of the blog post above includes some details including GEMM/memcpy performance, and some of the software layers that we needed to write to run on AMD.

mardifoufs 3y ago
What's the cost benefit vs. Nvidia? Is it cheaper?
- gdiamos 3y ago
  Available in orders of up to 10,000 GPUs today - no shortage More than 10x cheaper than allocating machines on a tier 1 cloud - AWS, Azure, GCP, Oracle, etc More memory - 128GB HBM per GPU - means bigger models fit for training/inference without the nightmare of model parallelism over MPI/infiniband/etc Longer term - finetuning optimizations
  - mardifoufs 3y ago
    Ah! The memory sounds interesting. How would that compare to similar Nvidia hardware w.r.t cost assuming the hardware was available? Does AMD provide something similar to nvlink, and even libraries like cudnn? Also, last I checked none of the public clouds offered any of the latest gens MI GPUs, so I wasn't aware that it had good availability! Azure had a preview but I'll look more into it now. Thank you for your answer btw!
    - gdiamos 3y ago
      Yeah getting around the no public cloud thing was really annoying. We had to build our own datacenter. On the plus side, it was drastically cheaper and now we can just slot in machines. I would prefer that a tier 1 cloud made MI GPUs available though. It would make it so much more accessible.
      - bitcoinmoney 3y ago
        Are you releasing your software stack to the public?
        gdiamos 3y ago
        It's available now with an enterprise license, because we validate that it is running correctly on a system we help configure. We will open source pieces of it over time. Our strategy is open source functional core. Eventually we will have an open source dev environment that runs on a personal scale computer. We already have this for some configurations, but we don't do enough testing to ensure perf/functionality on many different systems. We are mainly bottlenecked by resources as a 12-person startup. We have released some open source SDKs here: https://github.com/orgs/lamini-ai https://github.com/orgs/lamini-ai This class has some training recipe code: https://www.deeplearning.ai/short-courses/finetuning-large-language-models/ https://www.deeplearning.ai/short-courses/finetuning-large-l... One thing I'd like to push back to open source is the scale out AMD SLURM support.
    - gdiamos 3y ago
      See the memory size comparison (GB) in this table: https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units#Tesla https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_proces...
      - QuarterReptile 3y ago
        It blows my mind that A100 and H100 are each safely below 1000W power draw.
- gardnr 3y ago
  The classic economic benefits of competition: * Drives down price * Enhances product features (I see them competing on VRAM first) * Helps to insulates buyers from supply issues Nvidia has kneecapped their consumer grade hardware to ensure the gaming market still has scraps to buy in spite of crypto mining and the AI gold rush. All AMD would have to do to eat into Nvidia marketshare is remove the hardware locks in low-end cards and ship one with 64GB+ of VRAM. This of course would only work if they have comparable/usable software support. Any improvements to ROCm will be a boon for any company that doesn't already have or can't afford huge farms of high-end Nvidia chips.
- light_hue_1 3y ago
  You simply cannot buy nvidia GPUs at scale at the moment. We're getting quotes that are many months out, sometimes even a year+ out.
  - gdiamos 3y ago
    We kept hearing 52 weeks for new shipments.
dotnet00 3y ago
It's nice to hear that there are actual results to show, since AMD execs simply saying that ROCm is a priority isn't really convincing anymore given their track record on claims regarding support on the consumer side.
- viewtransform 3y ago
  The difference this time is that the executive is from Xilinx. Xilinx has had an AI software development team for a while in the FPGA space. AMD has had poor management in the GPU computing space since Raja Koduri's time (he put the best engineering resources on VR during his tenure and ignored deep learning). Subsequent directors have not had a long term vision and left within a few years. Looks like Lisa Su has corrected this now - they seem to have moved AMD software engineers en masse to work under Xilinx management on AI. Remains to be seen if this new management hierarchy will have a better vision and customer focus.
  - varelse 3y ago
    [dead]
tbruckner 3y ago
I would really hope you could get decent utilization on ops as fundamental as GEMM/memcpy on a single device. Translating that to MFU is a completely different story.
- gdiamos 3y ago
  We get good utilization at scale as well. Typically 30-40% of peak at the full application level for training and inference. Perf isn't the biggest problem though, many AI chips can do this or a bit better on benchmarks, if you invest the engineering time to tune the benchmark. The really hard part is getting a complete software stack running. It took us over 3 years because many of the layers just didn't exist, e.g. scale out LLM inference service that supports multiple requests with fine-grained batching across models distributed over multiple GPUs. On Instinct, ROCm gets you the ability to run most pytorch models on one GPU assuming you get the right drivers, compilers, framework builds, etc. That's a good start, but you need more to serve a real application.
  - mgaunard 3y ago
    People have been using their GPGPUs for decades on a variety of scientific applications, and there are all kinds of hybrid and multi-device frameworks that exist (often supporting multiple backends). The difference is that it didn't get a lot of love as part of the overhyped python LLM movement.
    - gdiamos 3y ago
      Completely agree, I'd love to see some of the innovations from HPC move over into their LLM stack. We are working on it, but it takes time. Contributions to foundational layers like ROCBlas, pytorch, slurm, Tensile, huggingface, etc would help.
jauntywundrkind 3y ago
> If you want to try this out, one big issue is that software support is hugely different on Instinct vs Radeon. I think AMD will fix this eventually, but today you need to use Instinct. I'm really really worried about AMD, and whether they're going to care about anyone else. They might just care about Instinct, where margins are so high, and ignore consumer cards or making more friction and segmentation for consumer cards. Part of what made CUDA so successful was that the low hardware barrier to entry created such a popular offering. Everyone used it. I really hope AMD realizes that, and really hope AMD invests in consumer card software too. Just making it work on the high end doesn't seem enough to get the kind of mass-movement ecosystem success AMD really needs. I'm afraid they might go for a smaller win, try to compete only at the top.
- gdiamos 3y ago
  I completely agree. I wasted a lot of time just assuming that ROCm would work on Radeon, just like it does for CUDA.

mardifoufs 3y ago

What's the cost benefit vs. Nvidia? Is it cheaper?

gdiamos 3y ago
Available in orders of up to 10,000 GPUs today - no shortage More than 10x cheaper than allocating machines on a tier 1 cloud - AWS, Azure, GCP, Oracle, etc More memory - 128GB HBM per GPU - means bigger models fit for training/inference without the nightmare of model parallelism over MPI/infiniband/etc Longer term - finetuning optimizations
- mardifoufs 3y ago
  Ah! The memory sounds interesting. How would that compare to similar Nvidia hardware w.r.t cost assuming the hardware was available? Does AMD provide something similar to nvlink, and even libraries like cudnn? Also, last I checked none of the public clouds offered any of the latest gens MI GPUs, so I wasn't aware that it had good availability! Azure had a preview but I'll look more into it now. Thank you for your answer btw!
  - gdiamos 3y ago
    Yeah getting around the no public cloud thing was really annoying. We had to build our own datacenter. On the plus side, it was drastically cheaper and now we can just slot in machines. I would prefer that a tier 1 cloud made MI GPUs available though. It would make it so much more accessible.
    - bitcoinmoney 3y ago
      Are you releasing your software stack to the public?
      - gdiamos 3y ago
        It's available now with an enterprise license, because we validate that it is running correctly on a system we help configure. We will open source pieces of it over time. Our strategy is open source functional core. Eventually we will have an open source dev environment that runs on a personal scale computer. We already have this for some configurations, but we don't do enough testing to ensure perf/functionality on many different systems. We are mainly bottlenecked by resources as a 12-person startup. We have released some open source SDKs here: https://github.com/orgs/lamini-ai https://github.com/orgs/lamini-ai This class has some training recipe code: https://www.deeplearning.ai/short-courses/finetuning-large-language-models/ https://www.deeplearning.ai/short-courses/finetuning-large-l... One thing I'd like to push back to open source is the scale out AMD SLURM support.
  - gdiamos 3y ago
    See the memory size comparison (GB) in this table: https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units#Tesla https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_proces...
    - QuarterReptile 3y ago
      It blows my mind that A100 and H100 are each safely below 1000W power draw.
gardnr 3y ago
The classic economic benefits of competition: * Drives down price * Enhances product features (I see them competing on VRAM first) * Helps to insulates buyers from supply issues Nvidia has kneecapped their consumer grade hardware to ensure the gaming market still has scraps to buy in spite of crypto mining and the AI gold rush. All AMD would have to do to eat into Nvidia marketshare is remove the hardware locks in low-end cards and ship one with 64GB+ of VRAM. This of course would only work if they have comparable/usable software support. Any improvements to ROCm will be a boon for any company that doesn't already have or can't afford huge farms of high-end Nvidia chips.
light_hue_1 3y ago
You simply cannot buy nvidia GPUs at scale at the moment. We're getting quotes that are many months out, sometimes even a year+ out.
- gdiamos 3y ago
  We kept hearing 52 weeks for new shipments.

dotnet00 3y ago

It's nice to hear that there are actual results to show, since AMD execs simply saying that ROCm is a priority isn't really convincing anymore given their track record on claims regarding support on the consumer side.

viewtransform 3y ago
The difference this time is that the executive is from Xilinx. Xilinx has had an AI software development team for a while in the FPGA space. AMD has had poor management in the GPU computing space since Raja Koduri's time (he put the best engineering resources on VR during his tenure and ignored deep learning). Subsequent directors have not had a long term vision and left within a few years. Looks like Lisa Su has corrected this now - they seem to have moved AMD software engineers en masse to work under Xilinx management on AI. Remains to be seen if this new management hierarchy will have a better vision and customer focus.
- varelse 3y ago
  [dead]

tbruckner 3y ago

I would really hope you could get decent utilization on ops as fundamental as GEMM/memcpy on a single device. Translating that to MFU is a completely different story.

gdiamos 3y ago
We get good utilization at scale as well. Typically 30-40% of peak at the full application level for training and inference. Perf isn't the biggest problem though, many AI chips can do this or a bit better on benchmarks, if you invest the engineering time to tune the benchmark. The really hard part is getting a complete software stack running. It took us over 3 years because many of the layers just didn't exist, e.g. scale out LLM inference service that supports multiple requests with fine-grained batching across models distributed over multiple GPUs. On Instinct, ROCm gets you the ability to run most pytorch models on one GPU assuming you get the right drivers, compilers, framework builds, etc. That's a good start, but you need more to serve a real application.
- mgaunard 3y ago
  People have been using their GPGPUs for decades on a variety of scientific applications, and there are all kinds of hybrid and multi-device frameworks that exist (often supporting multiple backends). The difference is that it didn't get a lot of love as part of the overhyped python LLM movement.
  - gdiamos 3y ago
    Completely agree, I'd love to see some of the innovations from HPC move over into their LLM stack. We are working on it, but it takes time. Contributions to foundational layers like ROCBlas, pytorch, slurm, Tensile, huggingface, etc would help.

jauntywundrkind 3y ago

> If you want to try this out, one big issue is that software support is hugely different on Instinct vs Radeon. I think AMD will fix this eventually, but today you need to use Instinct. I'm really really worried about AMD, and whether they're going to care about anyone else. They might just care about Instinct, where margins are so high, and ignore consumer cards or making more friction and segmentation for consumer cards. Part of what made CUDA so successful was that the low hardware barrier to entry created such a popular offering. Everyone used it. I really hope AMD realizes that, and really hope AMD invests in consumer card software too. Just making it work on the high end doesn't seem enough to get the kind of mass-movement ecosystem success AMD really needs. I'm afraid they might go for a smaller win, try to compete only at the top.

gdiamos 3y ago
I completely agree. I wasted a lot of time just assuming that ROCm would work on Radeon, just like it does for CUDA.