5 ms·

If I recall correctly the AMD hardware is competitive or better in raw compute than their nVidia counterparts but Cuda makes developing software so easy so I’m

If I recall correctly the AMD hardware is competitive or better in raw compute than their nVidia counterparts but Cuda makes developing software so easy so I’m curious what ORNL’s take on that is.

vortico 7y ago
I'm wondering the same. I'd guess that 99% of science code that uses the GPU now depends on CUDA. The other 1% use other third-party GPU frontends which might also be tuned for Nvidia GPUs. I feel that it's good to use the AMD ecosystem to mix up the competition a bit, and the majority of other clusters will still use Nvidia if your application requires it, but I'm just not aware of any momentum at all (in the form of software ecosystems) that target AMD, so library-writers will have to basically start from scratch if this AMD cluster is confirmed.
- rl3 7y ago
  >... so library-writers will have to basically start from scratch if this AMD cluster is confirmed. I doubt library writers will care beyond anyone involved in the project. That said, it's possible to port CUDA code already with a bit of work, so not quite from scratch in any case: https://github.com/ROCm-Developer-Tools/HIP https://github.com/ROCm-Developer-Tools/HIP
  - minxomat 7y ago
    See also PlaidML
    - throw20102010 7y ago
      PlaidML is not what most people want. PlaidML is pretty much only for neural network based machine learning, plus it's primarily a backend for models written in Keras. But it is an option for the narrow use case of neural networks. Contrary to what buzzword happy SV types would have you believe, most "real" HPC work isn't machine learning (r/gatekeeping, I know). Particle physics, computational fluid dynamics, network simulations, etc. Lots of it is already written in CUDA. HIP, using hipify, can translate the already written CUDA code to HIP, which is GPU-agnostic.
      - minxomat 7y ago
        Plaid is just one of 01org's projects. There are dozens of other projects (and compilers), which when combined can be used to build very intricate HPC applications and pipelines. Though for some core projects, most of the tuning development is only available on Intel platforms.
        throw20102010 7y ago
        1. Your previous comment only mentioned PlaidML. 2. The parent comment was about translating existing code written in CUDA to be used on an AMD GPU, not about developing new software- if it were about developing new software, they'd be starting from scratch with 01.org tools, which is what everyone wants to avoid. 01.org doesn't have any translation tools for this.
- tgamblin 7y ago
  On the NNSA side of DOE, the codes very intentionally do not depend on CUDA. They use RAJA (https://github.com/llnl/raja https://github.com/llnl/raja) or Kokkos (https://github.com/kokkos/kokkos https://github.com/kokkos/kokkos) to ensure that they're not tied to a particular GPU. Using those frameworks you can easily switch between CUDA, OpenMP, or whatever in a single-source codebase.
jefft255 7y ago
I’m wondering the same thing. With the size of this lab, maybe this will drive the adoption of open alternatives to CUDA. Serious amount of software will be written for this supercomputer. In Canada, however, all public GPU clusters use nvidia...
petschge 7y ago
I don't work for ORNL and don't know what their take in this is, but a lot of codes that the DoE uses are being ported to the Kokkos framework out of the Sandia National Lab. The application developer at that point basically doesn't have to care if the code runs on KNL, on Nvidia cards using Cuda or on AMD cards using ROCm. The work of tuning Kokkos well on AMD cards is cheap compared to getting stuck on a single vendor.
- gigatexal 7y ago
  Is it open source? It’d be huge to have something compete with cuda so that the best hardware may win
  - petschge 7y ago
    Kokkos (as in the C++ template frame work and a lot of tools to get insights into bottle necks) is opens source. See https://github.com/kokkos/kokkos https://github.com/kokkos/kokkos and https://github.com/kokkos/kokkos-tools https://github.com/kokkos/kokkos-tools . They take pull requests and react to bug reports. They even plan to upstream some things from kokkos (such as multi-dimensional arrays) into future C++ standards. The mission related codes based on kokkos are very much not open source.
    - desertrider12 7y ago
      There are some non-weapons codes like Albany and NaluCFD based on Kokkos and Trilinos that are open source. [1] https://github.com/SNLComputation/Albany https://github.com/SNLComputation/Albany [2] https://github.com/NaluCFD/Nalu https://github.com/NaluCFD/Nalu
  - xbdobd 7y ago
    Most everything we develop (I work for the DOE) has to be open source.
- tgamblin 7y ago
  At LLNL the story is similar, but the framework is RAJA: https://github.com/llnl/raja https://github.com/llnl/raja. RAJA is a bit simpler than Kokkos in that it does not require you to adopt its data structures. The Kokkos and RAJA teams are also working together on some common utility libraries now.
stonogo 7y ago
Presumably they intend to expand CAAR[1] to help with this. They're going to need to, if they want this to succeed. 1 - https://www.olcf.ornl.gov/caar/ https://www.olcf.ornl.gov/caar/
noir_lord 7y ago
The Radeon VII is, all though it's AMD's high end 'gaming' card the suspicioun is that was a secondary use and they shoved it out to have something competitive with the higher end nvidia cards (for gaming nvidia holds the edge, my 2080 is about 5-7% faster than a VII but the VII has twice the RAM and it's faster RAM). > When looking at the geometric mean of all the OpenCL benchmarks carried out, the Radeon VII was 12% faster than the GeForce RTX 2080... https://www.phoronix.com/scan.php?page=article&item=radeon-vii-linux&num=11 https://www.phoronix.com/scan.php?page=article&item=radeon-v... It gets crushed by the Titans but then they cost massively more (more than double).
jcranmer 7y ago
A shocking fact I heard recently is that on these large GPGPU supercomputers, only about 10% of the applications actually use the GPU at all. The labs are not likely to be a big fan of CUDA, since the reality is that most scientists do not have the bandwidth to rewrite their software (even to use GPGPUs in the first place, see above), and Nvidia tries very hard to make sure that CUDA is impossible to use for other GPU vendors. The labs have a requirement to source from multiple vendors, so the CUDA lock-in is not something they are thrilled about.
- xbdobd 7y ago
  Sounds about right

vortico 7y ago

I'm wondering the same. I'd guess that 99% of science code that uses the GPU now depends on CUDA. The other 1% use other third-party GPU frontends which might also be tuned for Nvidia GPUs. I feel that it's good to use the AMD ecosystem to mix up the competition a bit, and the majority of other clusters will still use Nvidia if your application requires it, but I'm just not aware of any momentum at all (in the form of software ecosystems) that target AMD, so library-writers will have to basically start from scratch if this AMD cluster is confirmed.

rl3 7y ago
>... so library-writers will have to basically start from scratch if this AMD cluster is confirmed. I doubt library writers will care beyond anyone involved in the project. That said, it's possible to port CUDA code already with a bit of work, so not quite from scratch in any case: https://github.com/ROCm-Developer-Tools/HIP https://github.com/ROCm-Developer-Tools/HIP
- minxomat 7y ago
  See also PlaidML
  - throw20102010 7y ago
    PlaidML is not what most people want. PlaidML is pretty much only for neural network based machine learning, plus it's primarily a backend for models written in Keras. But it is an option for the narrow use case of neural networks. Contrary to what buzzword happy SV types would have you believe, most "real" HPC work isn't machine learning (r/gatekeeping, I know). Particle physics, computational fluid dynamics, network simulations, etc. Lots of it is already written in CUDA. HIP, using hipify, can translate the already written CUDA code to HIP, which is GPU-agnostic.
    - minxomat 7y ago
      Plaid is just one of 01org's projects. There are dozens of other projects (and compilers), which when combined can be used to build very intricate HPC applications and pipelines. Though for some core projects, most of the tuning development is only available on Intel platforms.
      - throw20102010 7y ago
        1. Your previous comment only mentioned PlaidML. 2. The parent comment was about translating existing code written in CUDA to be used on an AMD GPU, not about developing new software- if it were about developing new software, they'd be starting from scratch with 01.org tools, which is what everyone wants to avoid. 01.org doesn't have any translation tools for this.
tgamblin 7y ago
On the NNSA side of DOE, the codes very intentionally do not depend on CUDA. They use RAJA (https://github.com/llnl/raja https://github.com/llnl/raja) or Kokkos (https://github.com/kokkos/kokkos https://github.com/kokkos/kokkos) to ensure that they're not tied to a particular GPU. Using those frameworks you can easily switch between CUDA, OpenMP, or whatever in a single-source codebase.

jefft255 7y ago

I’m wondering the same thing. With the size of this lab, maybe this will drive the adoption of open alternatives to CUDA. Serious amount of software will be written for this supercomputer. In Canada, however, all public GPU clusters use nvidia...

petschge 7y ago

I don't work for ORNL and don't know what their take in this is, but a lot of codes that the DoE uses are being ported to the Kokkos framework out of the Sandia National Lab. The application developer at that point basically doesn't have to care if the code runs on KNL, on Nvidia cards using Cuda or on AMD cards using ROCm. The work of tuning Kokkos well on AMD cards is cheap compared to getting stuck on a single vendor.

gigatexal 7y ago
Is it open source? It’d be huge to have something compete with cuda so that the best hardware may win
- petschge 7y ago
  Kokkos (as in the C++ template frame work and a lot of tools to get insights into bottle necks) is opens source. See https://github.com/kokkos/kokkos https://github.com/kokkos/kokkos and https://github.com/kokkos/kokkos-tools https://github.com/kokkos/kokkos-tools . They take pull requests and react to bug reports. They even plan to upstream some things from kokkos (such as multi-dimensional arrays) into future C++ standards. The mission related codes based on kokkos are very much not open source.
  - desertrider12 7y ago
    There are some non-weapons codes like Albany and NaluCFD based on Kokkos and Trilinos that are open source. [1] https://github.com/SNLComputation/Albany https://github.com/SNLComputation/Albany [2] https://github.com/NaluCFD/Nalu https://github.com/NaluCFD/Nalu
- xbdobd 7y ago
  Most everything we develop (I work for the DOE) has to be open source.
tgamblin 7y ago
At LLNL the story is similar, but the framework is RAJA: https://github.com/llnl/raja https://github.com/llnl/raja. RAJA is a bit simpler than Kokkos in that it does not require you to adopt its data structures. The Kokkos and RAJA teams are also working together on some common utility libraries now.

stonogo 7y ago

Presumably they intend to expand CAAR[1] to help with this. They're going to need to, if they want this to succeed. 1 - https://www.olcf.ornl.gov/caar/ https://www.olcf.ornl.gov/caar/

noir_lord 7y ago

The Radeon VII is, all though it's AMD's high end 'gaming' card the suspicioun is that was a secondary use and they shoved it out to have something competitive with the higher end nvidia cards (for gaming nvidia holds the edge, my 2080 is about 5-7% faster than a VII but the VII has twice the RAM and it's faster RAM). > When looking at the geometric mean of all the OpenCL benchmarks carried out, the Radeon VII was 12% faster than the GeForce RTX 2080... https://www.phoronix.com/scan.php?page=article&item=radeon-vii-linux&num=11 https://www.phoronix.com/scan.php?page=article&item=radeon-v... It gets crushed by the Titans but then they cost massively more (more than double).

jcranmer 7y ago

A shocking fact I heard recently is that on these large GPGPU supercomputers, only about 10% of the applications actually use the GPU at all. The labs are not likely to be a big fan of CUDA, since the reality is that most scientists do not have the bandwidth to rewrite their software (even to use GPGPUs in the first place, see above), and Nvidia tries very hard to make sure that CUDA is impossible to use for other GPU vendors. The labs have a requirement to source from multiple vendors, so the CUDA lock-in is not something they are thrilled about.

xbdobd 7y ago
Sounds about right