4 ms·

I just want to remind everyone that El Capitan, Frontier and LUMI supercomputers are powered by AMD instinct cards. El Capitan is #1 in TOP500. Frontier is #2,

I just want to remind everyone that El Capitan, Frontier and LUMI supercomputers are powered by AMD instinct cards. El Capitan is #1 in TOP500. Frontier is #2, LUMI is #8. ROCm development is probably mainly driven by the needs of these supercompuers' users currently. So, we're seeing the tip of the iceberg. Also ROCm packages continue to land on Debian, so there's more than meets the eye. Note: Search "AMD Instinct" at https://top500.org/lists/top500/list/2024/11/ https://top500.org/lists/top500/list/2024/11/. There are way more systems.

brrrrrm 2y ago
Do super computers run in fp64 mostly? At fp8 an h100 hits 2 petaflops, and with only 1000 of them you’ve got more compute power than el capitan (in raw flop count)
- dhruvdh 2y ago
  El Capitan can also do FP8. HPC requires double precision generally but people are trying to make low precision work.
  - touisteur 2y ago
    I'm particularly fond of the Ozaki scheme https://arxiv.org/html/2306.11975v4 https://arxiv.org/html/2306.11975v4 and its recent refinements. Hopefully it trickles down to standard HPC libraries soon.
- bayindirh 2y ago
  Disclosure: I'm an HPC admin who developed a materials simulation framework for my Ph.D. Simulations run on FP64, and you have to since you're already approximating stuff with numerical algorithms (analytic solution of many things are impossible anyway). Even if you can do things with FP8, transferring everything to GPU is not trivially possible. A simulation contains tons of different algorithms, and not all of them can be modeled as a set of matrix operations effectively. Also, moving kernels in an out of GPU is not an instant affair, plus moving data to GPU is always more expensive. You have GPUDirect and MultiDMA engines in modern GPUs, but they need hardcore coding and knowing what you're doing if you're not solving popular stuff with established libraries and so on. Plus, if you don't prefer to be vendor locked, at least one of the vendors artificially limit the performance you can get from their cards. On the other hand, all of the prominent linear algebra libraries squeeze out the CPUs you have relatively easily, and you don't have to have matrices and vectors to get this performance from CPUs anyway. Lastly, I want to touch on that parallelization such problems are not always trivial even on CPUs. When you go multinode via MPI, things get fun. Getting GPUs into that mix is somewhat of a madness if you're not prepared.
- kllrnohj 2y ago
  It hits 2 petaflops on the tensor cores at fp8. If you want GPGPU, that plummets to 134 teraflops (for fp16, though)
slavik81 2y ago
> ROCm packages continue to land on Debian, so there's more than meets the eye I've been volunteering with Debian to help package ROCm for four years now, but today it officially became my full-time job. AMA.
- bayindirh 2y ago
  I have no questions, but congrats! It's great to hear good things like this as both an HPC admin, and a Debian user of 20+ years. Man, I'm old. :)
- latchkey 2y ago
  Congrats!
- Athas 2y ago
  Who do you work for? And is packaging ROCm for Debian really a full-time job, or is it just a part of your job? As messy as ROCm's packaging is, I can't imagine spending all day every day trying to fix it.
  - saagarjha 2y ago
    > I work for AMD
    - cma 2y ago
      from his profile, if anyone is looking for where that came from
  - slavik81 2y ago
    I work for AMD. To be clear, my new job is about integrating ROCm into the distribution not just about shipping ROCm packages that can run on Debian. I'll be doing things like creating new packages in main, helping to get support for the HIP language embedded into existing dpkg tooling, helping to get GPU architecture awareness integrated into the Debian CI infrastructure, helping to enable ROCm support in other libraries and applications packaged for Debian, and ensuring that everything in Debian is successfully imported into the Ubuntu universe repositories. Integrating HIP support into Debian so that it feels as natural as C or C++ and 'just works' across dozens of GPUs is a job for more than one person. That is why I'm glad there have been so many volunteers in the community stepping forward to help with various pieces.
saagarjha 2y ago
> ROCm development is probably mainly driven by the needs of these supercompuers' users currently. Seems like a problem since AMD wants to go after AI capex?
- roenxi 2y ago
  The AI capex is being invested into things that are, effectively, supercomputers.
  - saagarjha 2y ago
    Supercomputers have very different needs. They want 64-bit floating point which nobody has been focusing on for a while
    - arcanus 2y ago
      While FP64 is indeed important for supercomputers, the largest supercomputers have a great deal in common with AI infrastructure. For example, high bandwidth, low latency interconnects, supporting GPU direct network messaging and IO, are important. High memory bandwidth is also quite important. Debugging and performance profiling at scale also commonly uses similar tools.
    - bayindirh 2y ago
      No they do not, because supercomputers have different partitions to cater different needs. For example, a supercomputer's half the nodes might lack a GPU to cater for the users which really need FP64 on CPU, and the other half will have GPUs for users which needs them. They will be served from different queues, so their jobs do not block each other. OTOH, if you don't think nobody is focusing on FP64, look YoY performance gains on both CPUs and FPUs for high precision floating point performance. You'll be surprised.
wkat4242 2y ago
They support their workstation cards pretty poorly though. I have a Radeon VII Pro and it's already deprecated in ROCm, it's not even 3 years old. They can really learn a lesson from Nvidia that supports old cards going back far and supports every card, not just a few hand-picked business models.

brrrrrm 2y ago

Do super computers run in fp64 mostly? At fp8 an h100 hits 2 petaflops, and with only 1000 of them you’ve got more compute power than el capitan (in raw flop count)

dhruvdh 2y ago
El Capitan can also do FP8. HPC requires double precision generally but people are trying to make low precision work.
- touisteur 2y ago
  I'm particularly fond of the Ozaki scheme https://arxiv.org/html/2306.11975v4 https://arxiv.org/html/2306.11975v4 and its recent refinements. Hopefully it trickles down to standard HPC libraries soon.
bayindirh 2y ago
Disclosure: I'm an HPC admin who developed a materials simulation framework for my Ph.D. Simulations run on FP64, and you have to since you're already approximating stuff with numerical algorithms (analytic solution of many things are impossible anyway). Even if you can do things with FP8, transferring everything to GPU is not trivially possible. A simulation contains tons of different algorithms, and not all of them can be modeled as a set of matrix operations effectively. Also, moving kernels in an out of GPU is not an instant affair, plus moving data to GPU is always more expensive. You have GPUDirect and MultiDMA engines in modern GPUs, but they need hardcore coding and knowing what you're doing if you're not solving popular stuff with established libraries and so on. Plus, if you don't prefer to be vendor locked, at least one of the vendors artificially limit the performance you can get from their cards. On the other hand, all of the prominent linear algebra libraries squeeze out the CPUs you have relatively easily, and you don't have to have matrices and vectors to get this performance from CPUs anyway. Lastly, I want to touch on that parallelization such problems are not always trivial even on CPUs. When you go multinode via MPI, things get fun. Getting GPUs into that mix is somewhat of a madness if you're not prepared.
kllrnohj 2y ago
It hits 2 petaflops on the tensor cores at fp8. If you want GPGPU, that plummets to 134 teraflops (for fp16, though)

slavik81 2y ago

> ROCm packages continue to land on Debian, so there's more than meets the eye I've been volunteering with Debian to help package ROCm for four years now, but today it officially became my full-time job. AMA.

bayindirh 2y ago
I have no questions, but congrats! It's great to hear good things like this as both an HPC admin, and a Debian user of 20+ years. Man, I'm old. :)
latchkey 2y ago
Congrats!
Athas 2y ago
Who do you work for? And is packaging ROCm for Debian really a full-time job, or is it just a part of your job? As messy as ROCm's packaging is, I can't imagine spending all day every day trying to fix it.
- saagarjha 2y ago
  > I work for AMD
  - cma 2y ago
    from his profile, if anyone is looking for where that came from
- slavik81 2y ago
  I work for AMD. To be clear, my new job is about integrating ROCm into the distribution not just about shipping ROCm packages that can run on Debian. I'll be doing things like creating new packages in main, helping to get support for the HIP language embedded into existing dpkg tooling, helping to get GPU architecture awareness integrated into the Debian CI infrastructure, helping to enable ROCm support in other libraries and applications packaged for Debian, and ensuring that everything in Debian is successfully imported into the Ubuntu universe repositories. Integrating HIP support into Debian so that it feels as natural as C or C++ and 'just works' across dozens of GPUs is a job for more than one person. That is why I'm glad there have been so many volunteers in the community stepping forward to help with various pieces.

saagarjha 2y ago

> ROCm development is probably mainly driven by the needs of these supercompuers' users currently. Seems like a problem since AMD wants to go after AI capex?

roenxi 2y ago
The AI capex is being invested into things that are, effectively, supercomputers.
- saagarjha 2y ago
  Supercomputers have very different needs. They want 64-bit floating point which nobody has been focusing on for a while
  - arcanus 2y ago
    While FP64 is indeed important for supercomputers, the largest supercomputers have a great deal in common with AI infrastructure. For example, high bandwidth, low latency interconnects, supporting GPU direct network messaging and IO, are important. High memory bandwidth is also quite important. Debugging and performance profiling at scale also commonly uses similar tools.
  - bayindirh 2y ago
    No they do not, because supercomputers have different partitions to cater different needs. For example, a supercomputer's half the nodes might lack a GPU to cater for the users which really need FP64 on CPU, and the other half will have GPUs for users which needs them. They will be served from different queues, so their jobs do not block each other. OTOH, if you don't think nobody is focusing on FP64, look YoY performance gains on both CPUs and FPUs for high precision floating point performance. You'll be surprised.

wkat4242 2y ago

They support their workstation cards pretty poorly though. I have a Radeon VII Pro and it's already deprecated in ROCm, it's not even 3 years old. They can really learn a lesson from Nvidia that supports old cards going back far and supports every card, not just a few hand-picked business models.