4 ms·

A lot of HPC code is by its very nature pretty parallelizable (given that supercomputers these days are often just a large cluster of typical datacenter CPUs/GP

A lot of HPC code is by its very nature pretty parallelizable (given that supercomputers these days are often just a large cluster of typical datacenter CPUs/GPUs with very fast interconnects), so is theoretically portable to GPUs for some gain. The hangups mostly seem to be in that a lot of scientific code hasn't been written with GPU style parallelization in mind, so updating the code takes time. That said, since a lot of newer supercomputers are including a lot of GPUs, there is a lot more incentive for developers to attempt to port at least the parts that can benefit from GPUs. From my (admittedly limited) experience, the biggest hangup is convincing scientists that GPUs aren't just another tech fad and it's worth the effort to port. There also seems to be a bit of uncertainty over which platform to trust, OpenCL support has been shaky on most vendors and is fairly primitive in terms of features, CUDA is tied to one company and AMD doesn't have a great record of supporting HIP across their product stack, so choosing any one is a big decision (comparable to choosing to use C/C++ or Python, which are effectively guaranteed to be supported on every supercomputer for the next decade or two).

kxyvr 4y ago
I work as a computational mathematician. Part of the problem on my end is that while the libraries have gotten better, they have not contained the operations that I need for good algorithms in my field. As an example, when GPUs first came onto the field, cublas computed a subset of the BLAS routines, which are pretty fundamental for putting together known, good algorithms. Eventually, that got better, but then the operations needed to fit onto a single GPU. Eventually, cublas-xt came onto the market and that allowed parallelization across GPUs, which was great. However, it still didn't have all of the needed matrix operations. At the moment, I'm not sure where it currently is. Of course, matrix algebra was only one of the problems. BLAS supports the writing of LAPACK routines, which tend to deal more with dense factorizations and eigenvalue problems. I believe more factorizations have been implemented recently, but I'm not currently up to date. Nevertheless, that was absolutely a bottle neck for the longest time. Yes, iterative methods don't need factorizations, but a good fraction of the preconditioners do. Then, of course, this speaks nothing of the sparse linear algebra problem. There's multiple ways to do it, but many of the good sparse factorization routines need dense factorization routines, so these needed to come onto the market first and that took time. And, to be clear, I know that there are multiple, good teams working on this. It takes time. And, there's a huge number of operations. If you're bored, go look at the manual for Intel's MKL and see the number and variety of operations that it provides. Those operations are there because people like me need them to do our job. I'll also agree that the kinds of operations we use to do our job will evolve over time with hardware. However, matrix algebra, factorizations, and eigenvalue problems lie at the very core of applied mathematics and expecting the mathematics to rework the last several hundred years of practice to accommodate the lack of tooling from the GPU manufacturers isn't realistic either. Anyway, if someone knows the current state of what's possible, I'd love to hear. Selfishly, what it really boils down to is what operations (algebra, factorization, or eigenvalue), how big (how much memory or on multiple GPUs), and dense or sparse.
- p1esk 4y ago
  What prevents you from simply writing the operations you need in CUDA?
  - kxyvr 4y ago
    Honestly, a combination of time, money, and expertise. Getting a good, low level factorization that works well in parallel is hard. I can write them. Others write them better. However, outside of that, I don't have any financial support to write them and there's no way that I can convince my clients to fund that effort. Personally, I don't really care where the libraries come from, but I will contend that the hardware manufacturers are generally the best place for this work to occur. For many years, each of the chip makers published their own high performance BLAS and LAPACK routines, which worked really well. Intel had MKL, AMD had AMCL, Sun had sunperf, IBM had ESSL. NVIDIA does the same, and I'm hugely grateful for that, but it's not complete. Really, though, the comment is more to answer why more mathematicians don't use GPUs. My contention is not that I or my colleagues view it as a fad, but more that there's a lack of routines that we depend on. I currently work for myself, but I'll also mention that the manufacturers did go to management at places I've worked and management did put pressure on staff to just rewrite everything in CUDA. As staff does, they said, ok, fine, but it's going to cost you labor. Management didn't have the money, or didn't want to spend it, and some mild office conflict occurred. Anyway, mostly that's to say that I will gladly spend thousands on GPUs and recommend it to my clients as soon as I don't have to write all of the low level routines myself.
    - p1esk 4y ago
      Is execution speed important to you? How much of a speed up would make it worth your time to implement it yourself?
p1esk 4y ago
the biggest hangup is convincing scientists that GPUs aren't just another tech fad CUDA is 15 years old. All top supercomputers use GPUs and have been for a while. Mature software tools like cupy exist to make GPU programming easier. I don’t think anyone thinks today “GPUs are a fad”. The problem is a lot of code is simply hard to parallelize, and would need to be rewritten from scratch, likely in another language. For large old codebases this is a massive effort.
- dotnet00 4y ago
  CUDA is 15 years old, but in terms of significant presence in supercomputers, it's just a generation or two in. For example, NERSC only made its first system that's "GPU-first" available to all users two weeks ago, with the previous system (which was #5 in the world when commissioned in 2016) only having gotten some GPU nodes in 2020 to help smooth the transition. This was what led to the hesitance from the people I worked with. They were unsure if in another generation or two they'd have to look at another large effort to support another new programming paradigm. There was also hesitance in relying on CUDA since it locks them into a vendor. From an engineer's point of view, yeah, GPUs obviously aren't a fad, but a lot of this software is written by scientists, where the computer is simply a means of getting their result and not something they keep up with as they would with their own field and yet the software needs to remain relatively stable to be reliable for research. Thus anyone asking for a large modification of the code is going to be viewed with extreme skepticism. For reference, I had to spend around 2 months of weekly meetings and presentations showing test results and discussing the risk/reward tradeoffs in detail to convince a group that a limited port of the code would be worth going ahead with and that CUDA was effectively the only reliable option for now. They've only gotten serious about a more in-depth port from seeing the large speedups without breaking compatibility which we managed to get after a few iterations.
fffobar 4y ago
Most of the code ran in large datacenters isn't classic HPC. It's cloud workloads serving live traffic or running database backends or some analytical pipelines/queries running over the stored data. Deep ML is about the only thing that's both large and "classic parallelizable HPC".
- dotnet00 4y ago
  I was talking about scientific code, as that's what I have experience with. There a ton of work is large and parallelizable. Molecular dynamics, physical optics, magnetostatics, fluid simulations etc all benefit a lot from large scale parallelization/GPU support.
  - fffobar 4y ago
    Yeah, but that's the minority on a global scale.

kxyvr 4y ago

I work as a computational mathematician. Part of the problem on my end is that while the libraries have gotten better, they have not contained the operations that I need for good algorithms in my field. As an example, when GPUs first came onto the field, cublas computed a subset of the BLAS routines, which are pretty fundamental for putting together known, good algorithms. Eventually, that got better, but then the operations needed to fit onto a single GPU. Eventually, cublas-xt came onto the market and that allowed parallelization across GPUs, which was great. However, it still didn't have all of the needed matrix operations. At the moment, I'm not sure where it currently is. Of course, matrix algebra was only one of the problems. BLAS supports the writing of LAPACK routines, which tend to deal more with dense factorizations and eigenvalue problems. I believe more factorizations have been implemented recently, but I'm not currently up to date. Nevertheless, that was absolutely a bottle neck for the longest time. Yes, iterative methods don't need factorizations, but a good fraction of the preconditioners do. Then, of course, this speaks nothing of the sparse linear algebra problem. There's multiple ways to do it, but many of the good sparse factorization routines need dense factorization routines, so these needed to come onto the market first and that took time. And, to be clear, I know that there are multiple, good teams working on this. It takes time. And, there's a huge number of operations. If you're bored, go look at the manual for Intel's MKL and see the number and variety of operations that it provides. Those operations are there because people like me need them to do our job. I'll also agree that the kinds of operations we use to do our job will evolve over time with hardware. However, matrix algebra, factorizations, and eigenvalue problems lie at the very core of applied mathematics and expecting the mathematics to rework the last several hundred years of practice to accommodate the lack of tooling from the GPU manufacturers isn't realistic either. Anyway, if someone knows the current state of what's possible, I'd love to hear. Selfishly, what it really boils down to is what operations (algebra, factorization, or eigenvalue), how big (how much memory or on multiple GPUs), and dense or sparse.

p1esk 4y ago
What prevents you from simply writing the operations you need in CUDA?
- kxyvr 4y ago
  Honestly, a combination of time, money, and expertise. Getting a good, low level factorization that works well in parallel is hard. I can write them. Others write them better. However, outside of that, I don't have any financial support to write them and there's no way that I can convince my clients to fund that effort. Personally, I don't really care where the libraries come from, but I will contend that the hardware manufacturers are generally the best place for this work to occur. For many years, each of the chip makers published their own high performance BLAS and LAPACK routines, which worked really well. Intel had MKL, AMD had AMCL, Sun had sunperf, IBM had ESSL. NVIDIA does the same, and I'm hugely grateful for that, but it's not complete. Really, though, the comment is more to answer why more mathematicians don't use GPUs. My contention is not that I or my colleagues view it as a fad, but more that there's a lack of routines that we depend on. I currently work for myself, but I'll also mention that the manufacturers did go to management at places I've worked and management did put pressure on staff to just rewrite everything in CUDA. As staff does, they said, ok, fine, but it's going to cost you labor. Management didn't have the money, or didn't want to spend it, and some mild office conflict occurred. Anyway, mostly that's to say that I will gladly spend thousands on GPUs and recommend it to my clients as soon as I don't have to write all of the low level routines myself.
  - p1esk 4y ago
    Is execution speed important to you? How much of a speed up would make it worth your time to implement it yourself?

p1esk 4y ago

the biggest hangup is convincing scientists that GPUs aren't just another tech fad CUDA is 15 years old. All top supercomputers use GPUs and have been for a while. Mature software tools like cupy exist to make GPU programming easier. I don’t think anyone thinks today “GPUs are a fad”. The problem is a lot of code is simply hard to parallelize, and would need to be rewritten from scratch, likely in another language. For large old codebases this is a massive effort.

dotnet00 4y ago
CUDA is 15 years old, but in terms of significant presence in supercomputers, it's just a generation or two in. For example, NERSC only made its first system that's "GPU-first" available to all users two weeks ago, with the previous system (which was #5 in the world when commissioned in 2016) only having gotten some GPU nodes in 2020 to help smooth the transition. This was what led to the hesitance from the people I worked with. They were unsure if in another generation or two they'd have to look at another large effort to support another new programming paradigm. There was also hesitance in relying on CUDA since it locks them into a vendor. From an engineer's point of view, yeah, GPUs obviously aren't a fad, but a lot of this software is written by scientists, where the computer is simply a means of getting their result and not something they keep up with as they would with their own field and yet the software needs to remain relatively stable to be reliable for research. Thus anyone asking for a large modification of the code is going to be viewed with extreme skepticism. For reference, I had to spend around 2 months of weekly meetings and presentations showing test results and discussing the risk/reward tradeoffs in detail to convince a group that a limited port of the code would be worth going ahead with and that CUDA was effectively the only reliable option for now. They've only gotten serious about a more in-depth port from seeing the large speedups without breaking compatibility which we managed to get after a few iterations.

fffobar 4y ago

Most of the code ran in large datacenters isn't classic HPC. It's cloud workloads serving live traffic or running database backends or some analytical pipelines/queries running over the stored data. Deep ML is about the only thing that's both large and "classic parallelizable HPC".

dotnet00 4y ago
I was talking about scientific code, as that's what I have experience with. There a ton of work is large and parallelizable. Molecular dynamics, physical optics, magnetostatics, fluid simulations etc all benefit a lot from large scale parallelization/GPU support.
- fffobar 4y ago
  Yeah, but that's the minority on a global scale.