5 ms·

It's so strange, when I went to school almost a decade ago, I took a GPGPU course - I assumed that with GPU compute becoming widely available, it would become u

It's so strange, when I went to school almost a decade ago, I took a GPGPU course - I assumed that with GPU compute becoming widely available, it would become ubiquitous, why sort an array on the CPU in nlogn time, when you can do it on the GPU in logn with bitonic sort? For some reason this future has never come to pass, GPGPU remained a computational curiosity, with it largely relegated to computer graphics and machine learning (although in the latter case, people rarely write CUDA themselves). I wonder why that is the case, and if it will ever become mainstream.

jahewson 4y ago
For most day-to-day (non-big) data the delay from copying the data to and from the GPU will outweigh the time saved. For large data, there’s usually an existing library for handing it - images, videos, ML weights, etc. It may take advantage of SIMD instructions, perhaps, or GPU, the end user doesn’t need to know. Another reason is that specialised devices beat general devices. Once everyone settles on H.264, then adding specialised decoding silicon to the die is a big win, especially where battery life and efficiency matter. Same for ray-tracing, neural/tensor cores.
- torginus 4y ago
  Yeah, but we got integrated GPUs which share memory memory with the CPU - consoles, mobiles, have done this forever, and it seems that sharing memory between the CPU and GPU is the way to go, considering stuff like the M1.
  - rocqua 4y ago
    M1 can pull off shared memory because it has ridiculous RAM speed. iGPU has to share memory. Thing is CPU is very memory latency sensitive, whilst GPU is much more memory throughout dependent. Hence they get memory with different tradeoffs.
majke 4y ago
In my humble opinon there just isn't much point to use GPGPU for normal computing. I'll give you an example. https://blog.cloudflare.com/computing-euclidean-distance-on-144-dimensions/ https://blog.cloudflare.com/computing-euclidean-distance-on-... during this work I was able to achieve: - 7.5kqps on 40 Xeon CPU's (optimized avx code) - 35kqps on V100 gpu V100 is say $10k, Xeon 6162 is $3k. That means $0.4 per request on intel, and $0.28 per request on GPU. This really is not much difference. I'd prefer to own more general purpose servers than hard to program proprietary GPU's. Obviousluy, getting GPU's sometimes makes sense. There are tasks that are a better fit for CUDA/GPGPU, stuff that requires high memory bandwidth, doesn't do branches. Furthermore, while V100 costs a lot, 3080 is supposed to be only say $700, which changes the equation in favor of gpu. But overally, modern x86 AVX is surprisingly good, and apart from very specialistic tasks, porting generic algorithms to GPU is not worth the effort.
- dahart 4y ago
  > 3080 is supposed to be only say $700, which changes the equation in favor of gpu. Yes, more than an order of magnitude cheaper, and also 2x higher peak teraflops. This makes a 20x-30x difference in your dollars per request calculation. So it’s somewhat confusing why at the end you ignore this and conclude GPUs aren’t worth it. V100 is a couple generations old, and comes in a half price model too, plus there are some other server grade options. The reasons to use the most expensive V100 are the mem capacity, or double precision, or to argue that it’s too expensive… ;)
  - wmf 4y ago
    GeForce GPUs aren't "allowed" to be used in servers so you "have to" buy the 10x more expensive ones.
    - dahart 4y ago
      Yeah true some configs are certified for servers and GeForce aren’t. Xeon’s in the same boat. So it does depend on your goals and requirements then. Using the V100 is still cherry picking in a price/perf argument since you don’t have to spend 10x, there are others cheaper options that are server qualified, right?
oriolid 4y ago
In my experience a big obstacle is that GPU programming is considered really difficult and a normal programmers shouldn't touch it on employer's time because it will just waste time and maybe produce difficult bugs if they get something done. The other obstacle is that any optimization is considered evil anyway, and since using GPGPU needs some architecture decisions, it's premature optimization which is extra evil. It's much cheaper to tell the users to buy a faster CPU. For machine learning and graphics GPGPU is allowed, because they would be more or less infeasible, not just slower, with just CPU.
- torginus 4y ago
  I think GPGPU is considered difficult because people aren't taught how to do it, and there's very little software support for it outside of esolangs, research projects, and single-person GitHub crusades, or vendor specific stuff like CUDA. There's other stuff that's difficult too, like farming out a compute workload to a bunch of unreliable servers through unreliable networks, but there's just so much tooling and expertise going around for it, that people do it regularly.
  - oriolid 4y ago
    If CUDA feels bad, there's this cross platform API called OpenCL. It's even possible to generate OpenCL from C++ without directly writing kernels with Boost.Compute, and I wouldn't call C++ an esolang. And if you're fine with nVidia, there's stuff like Thrust and cuBLAS. It's true that it's not taught, but again optimization isn't interesting for computer scientists, software engineers are taught it's evil and physicists are supposed to just read a book and get to work. I think distributed computing is OK because it again enables things that would be impossible with a single computer, no matter how huge.
    - pjmlp 4y ago
      Great printf debugging experience with C, and playing compiler with source code on the fly, meanwhile on CUDA side, graphical debugging with shader code and support for standard C++17, shipping bytecode.
nemothekid 4y ago
The number one issue is hardware; particularly memory latency. Unless you have a lot of compute that excels on GPU (like a gorillion matrix multiplications) or a lot of memory that you want to execute, then the execution time is dominated by memory transfer. Recently I had a task where I wanted to compute just cosine similarities between two vectors. For a couple hundred thousand floats my code spent something like ~1ms on CPU and ~25ms on the GPU. The GPU didn't start winning until I got to the millions of floats. For my use case a better solution was just taking advantage of a SIMD library.

jahewson 4y ago

For most day-to-day (non-big) data the delay from copying the data to and from the GPU will outweigh the time saved. For large data, there’s usually an existing library for handing it - images, videos, ML weights, etc. It may take advantage of SIMD instructions, perhaps, or GPU, the end user doesn’t need to know. Another reason is that specialised devices beat general devices. Once everyone settles on H.264, then adding specialised decoding silicon to the die is a big win, especially where battery life and efficiency matter. Same for ray-tracing, neural/tensor cores.

torginus 4y ago
Yeah, but we got integrated GPUs which share memory memory with the CPU - consoles, mobiles, have done this forever, and it seems that sharing memory between the CPU and GPU is the way to go, considering stuff like the M1.
- rocqua 4y ago
  M1 can pull off shared memory because it has ridiculous RAM speed. iGPU has to share memory. Thing is CPU is very memory latency sensitive, whilst GPU is much more memory throughout dependent. Hence they get memory with different tradeoffs.

majke 4y ago

In my humble opinon there just isn't much point to use GPGPU for normal computing. I'll give you an example. https://blog.cloudflare.com/computing-euclidean-distance-on-144-dimensions/ https://blog.cloudflare.com/computing-euclidean-distance-on-... during this work I was able to achieve: - 7.5kqps on 40 Xeon CPU's (optimized avx code) - 35kqps on V100 gpu V100 is say $10k, Xeon 6162 is $3k. That means $0.4 per request on intel, and $0.28 per request on GPU. This really is not much difference. I'd prefer to own more general purpose servers than hard to program proprietary GPU's. Obviousluy, getting GPU's sometimes makes sense. There are tasks that are a better fit for CUDA/GPGPU, stuff that requires high memory bandwidth, doesn't do branches. Furthermore, while V100 costs a lot, 3080 is supposed to be only say $700, which changes the equation in favor of gpu. But overally, modern x86 AVX is surprisingly good, and apart from very specialistic tasks, porting generic algorithms to GPU is not worth the effort.

dahart 4y ago
> 3080 is supposed to be only say $700, which changes the equation in favor of gpu. Yes, more than an order of magnitude cheaper, and also 2x higher peak teraflops. This makes a 20x-30x difference in your dollars per request calculation. So it’s somewhat confusing why at the end you ignore this and conclude GPUs aren’t worth it. V100 is a couple generations old, and comes in a half price model too, plus there are some other server grade options. The reasons to use the most expensive V100 are the mem capacity, or double precision, or to argue that it’s too expensive… ;)
- wmf 4y ago
  GeForce GPUs aren't "allowed" to be used in servers so you "have to" buy the 10x more expensive ones.
  - dahart 4y ago
    Yeah true some configs are certified for servers and GeForce aren’t. Xeon’s in the same boat. So it does depend on your goals and requirements then. Using the V100 is still cherry picking in a price/perf argument since you don’t have to spend 10x, there are others cheaper options that are server qualified, right?

oriolid 4y ago

In my experience a big obstacle is that GPU programming is considered really difficult and a normal programmers shouldn't touch it on employer's time because it will just waste time and maybe produce difficult bugs if they get something done. The other obstacle is that any optimization is considered evil anyway, and since using GPGPU needs some architecture decisions, it's premature optimization which is extra evil. It's much cheaper to tell the users to buy a faster CPU. For machine learning and graphics GPGPU is allowed, because they would be more or less infeasible, not just slower, with just CPU.

torginus 4y ago
I think GPGPU is considered difficult because people aren't taught how to do it, and there's very little software support for it outside of esolangs, research projects, and single-person GitHub crusades, or vendor specific stuff like CUDA. There's other stuff that's difficult too, like farming out a compute workload to a bunch of unreliable servers through unreliable networks, but there's just so much tooling and expertise going around for it, that people do it regularly.
- oriolid 4y ago
  If CUDA feels bad, there's this cross platform API called OpenCL. It's even possible to generate OpenCL from C++ without directly writing kernels with Boost.Compute, and I wouldn't call C++ an esolang. And if you're fine with nVidia, there's stuff like Thrust and cuBLAS. It's true that it's not taught, but again optimization isn't interesting for computer scientists, software engineers are taught it's evil and physicists are supposed to just read a book and get to work. I think distributed computing is OK because it again enables things that would be impossible with a single computer, no matter how huge.
  - pjmlp 4y ago
    Great printf debugging experience with C, and playing compiler with source code on the fly, meanwhile on CUDA side, graphical debugging with shader code and support for standard C++17, shipping bytecode.

nemothekid 4y ago

The number one issue is hardware; particularly memory latency. Unless you have a lot of compute that excels on GPU (like a gorillion matrix multiplications) or a lot of memory that you want to execute, then the execution time is dominated by memory transfer. Recently I had a task where I wanted to compute just cosine similarities between two vectors. For a couple hundred thousand floats my code spent something like ~1ms on CPU and ~25ms on the GPU. The GPU didn't start winning until I got to the millions of floats. For my use case a better solution was just taking advantage of a SIMD library.