6 ms·

> If you switched all the CPU-only servers running AI and HPC worldwide to GPU-accelerated systems, you could save a whopping 11 trillion watt-hours of energy a

by rocqua 4y ago

> If you switched all the CPU-only servers running AI and HPC worldwide to GPU-accelerated systems, you could save a whopping 11 trillion watt-hours of energy a year. That’s like saving the energy more than 1.5 million homes consume in a year. That is a bold claim. It seems to me they simply assume all necessary code would trivially be converted to run on GPUs.

sprkwd 4y ago
If everything is GPUs, then nothing is GPUs?
- AYBABTME 4y ago
  Just make that a General Processing Unit.
dotnet00 4y ago
A lot of HPC code is by its very nature pretty parallelizable (given that supercomputers these days are often just a large cluster of typical datacenter CPUs/GPUs with very fast interconnects), so is theoretically portable to GPUs for some gain. The hangups mostly seem to be in that a lot of scientific code hasn't been written with GPU style parallelization in mind, so updating the code takes time. That said, since a lot of newer supercomputers are including a lot of GPUs, there is a lot more incentive for developers to attempt to port at least the parts that can benefit from GPUs. From my (admittedly limited) experience, the biggest hangup is convincing scientists that GPUs aren't just another tech fad and it's worth the effort to port. There also seems to be a bit of uncertainty over which platform to trust, OpenCL support has been shaky on most vendors and is fairly primitive in terms of features, CUDA is tied to one company and AMD doesn't have a great record of supporting HIP across their product stack, so choosing any one is a big decision (comparable to choosing to use C/C++ or Python, which are effectively guaranteed to be supported on every supercomputer for the next decade or two).
- kxyvr 4y ago
  I work as a computational mathematician. Part of the problem on my end is that while the libraries have gotten better, they have not contained the operations that I need for good algorithms in my field. As an example, when GPUs first came onto the field, cublas computed a subset of the BLAS routines, which are pretty fundamental for putting together known, good algorithms. Eventually, that got better, but then the operations needed to fit onto a single GPU. Eventually, cublas-xt came onto the market and that allowed parallelization across GPUs, which was great. However, it still didn't have all of the needed matrix operations. At the moment, I'm not sure where it currently is. Of course, matrix algebra was only one of the problems. BLAS supports the writing of LAPACK routines, which tend to deal more with dense factorizations and eigenvalue problems. I believe more factorizations have been implemented recently, but I'm not currently up to date. Nevertheless, that was absolutely a bottle neck for the longest time. Yes, iterative methods don't need factorizations, but a good fraction of the preconditioners do. Then, of course, this speaks nothing of the sparse linear algebra problem. There's multiple ways to do it, but many of the good sparse factorization routines need dense factorization routines, so these needed to come onto the market first and that took time. And, to be clear, I know that there are multiple, good teams working on this. It takes time. And, there's a huge number of operations. If you're bored, go look at the manual for Intel's MKL and see the number and variety of operations that it provides. Those operations are there because people like me need them to do our job. I'll also agree that the kinds of operations we use to do our job will evolve over time with hardware. However, matrix algebra, factorizations, and eigenvalue problems lie at the very core of applied mathematics and expecting the mathematics to rework the last several hundred years of practice to accommodate the lack of tooling from the GPU manufacturers isn't realistic either. Anyway, if someone knows the current state of what's possible, I'd love to hear. Selfishly, what it really boils down to is what operations (algebra, factorization, or eigenvalue), how big (how much memory or on multiple GPUs), and dense or sparse.
  - p1esk 4y ago
    What prevents you from simply writing the operations you need in CUDA?
stingraycharles 4y ago
Which is extra important because, if someone is still running AI/ML code on the CPU rather than GPU in 2022, I suspect there must be very good reasons for doing so. So that makes it even less likely that this code can be ported in an economical way.
- deleted 4y ago
  [deleted]
- rocqua 4y ago
  I can very well imagine there is lots of non AI/ML code that smart people could make a lot faster on a GPU. I doubt there are anywhere near enough smart people to do that. I also imagine Nvidea might like to overstate the total potential.
  - nobodyandproud 4y ago
    It’s not that difficult. The biggest hurdles are learning the dev tech stack and setting up the development environment. The last time I tried out CUDA, it took me a couple of days to correctly setup the environment; just to spend an hour playing around parallelizing math operations. The best way I could describe the first impression dev experience: It’s a huge hack, adding non-standard language extensions to C/C++ And so relying on their own compiler or linker (iirc), instead of the standard C++. Couple this with NVidia not playing nice with OpenCL to force you into their playground, and I lost interest pretty quickly.
    - rocqua 4y ago
      Besides that hack, getting your head around the way to optimize code for a GPU takes a while. Then the real challenge is taking code that is based on random access to gigabytes of RAM, and making that code work with any kind of performance on the GPU performance model. There are cases where the random access to RAM are very deep in the algorithm. Consider e.g. sparse matrix multiplication.
      - 4y ago

by rocqua 4y ago

sprkwd 4y ago

If everything is GPUs, then nothing is GPUs?

AYBABTME 4y ago
Just make that a General Processing Unit.

dotnet00 4y ago

A lot of HPC code is by its very nature pretty parallelizable (given that supercomputers these days are often just a large cluster of typical datacenter CPUs/GPUs with very fast interconnects), so is theoretically portable to GPUs for some gain. The hangups mostly seem to be in that a lot of scientific code hasn't been written with GPU style parallelization in mind, so updating the code takes time. That said, since a lot of newer supercomputers are including a lot of GPUs, there is a lot more incentive for developers to attempt to port at least the parts that can benefit from GPUs. From my (admittedly limited) experience, the biggest hangup is convincing scientists that GPUs aren't just another tech fad and it's worth the effort to port. There also seems to be a bit of uncertainty over which platform to trust, OpenCL support has been shaky on most vendors and is fairly primitive in terms of features, CUDA is tied to one company and AMD doesn't have a great record of supporting HIP across their product stack, so choosing any one is a big decision (comparable to choosing to use C/C++ or Python, which are effectively guaranteed to be supported on every supercomputer for the next decade or two).

kxyvr 4y ago
I work as a computational mathematician. Part of the problem on my end is that while the libraries have gotten better, they have not contained the operations that I need for good algorithms in my field. As an example, when GPUs first came onto the field, cublas computed a subset of the BLAS routines, which are pretty fundamental for putting together known, good algorithms. Eventually, that got better, but then the operations needed to fit onto a single GPU. Eventually, cublas-xt came onto the market and that allowed parallelization across GPUs, which was great. However, it still didn't have all of the needed matrix operations. At the moment, I'm not sure where it currently is. Of course, matrix algebra was only one of the problems. BLAS supports the writing of LAPACK routines, which tend to deal more with dense factorizations and eigenvalue problems. I believe more factorizations have been implemented recently, but I'm not currently up to date. Nevertheless, that was absolutely a bottle neck for the longest time. Yes, iterative methods don't need factorizations, but a good fraction of the preconditioners do. Then, of course, this speaks nothing of the sparse linear algebra problem. There's multiple ways to do it, but many of the good sparse factorization routines need dense factorization routines, so these needed to come onto the market first and that took time. And, to be clear, I know that there are multiple, good teams working on this. It takes time. And, there's a huge number of operations. If you're bored, go look at the manual for Intel's MKL and see the number and variety of operations that it provides. Those operations are there because people like me need them to do our job. I'll also agree that the kinds of operations we use to do our job will evolve over time with hardware. However, matrix algebra, factorizations, and eigenvalue problems lie at the very core of applied mathematics and expecting the mathematics to rework the last several hundred years of practice to accommodate the lack of tooling from the GPU manufacturers isn't realistic either. Anyway, if someone knows the current state of what's possible, I'd love to hear. Selfishly, what it really boils down to is what operations (algebra, factorization, or eigenvalue), how big (how much memory or on multiple GPUs), and dense or sparse.
- p1esk 4y ago
  What prevents you from simply writing the operations you need in CUDA?

stingraycharles 4y ago

Which is extra important because, if someone is still running AI/ML code on the CPU rather than GPU in 2022, I suspect there must be very good reasons for doing so. So that makes it even less likely that this code can be ported in an economical way.

deleted 4y ago
[deleted]
rocqua 4y ago
I can very well imagine there is lots of non AI/ML code that smart people could make a lot faster on a GPU. I doubt there are anywhere near enough smart people to do that. I also imagine Nvidea might like to overstate the total potential.
- nobodyandproud 4y ago
  It’s not that difficult. The biggest hurdles are learning the dev tech stack and setting up the development environment. The last time I tried out CUDA, it took me a couple of days to correctly setup the environment; just to spend an hour playing around parallelizing math operations. The best way I could describe the first impression dev experience: It’s a huge hack, adding non-standard language extensions to C/C++ And so relying on their own compiler or linker (iirc), instead of the standard C++. Couple this with NVidia not playing nice with OpenCL to force you into their playground, and I lost interest pretty quickly.
  - rocqua 4y ago
    Besides that hack, getting your head around the way to optimize code for a GPU takes a while. Then the real challenge is taking code that is based on random access to gigabytes of RAM, and making that code work with any kind of performance on the GPU performance model. There are cases where the random access to RAM are very deep in the algorithm. Consider e.g. sparse matrix multiplication.
    - 4y ago