4 ms·

Could you elaborate? I've only done a little work on CUDA, but I was pretty impressed with it and with their NSys tools. I'm curious what you wish was differe

Could you elaborate? I've only done a little work on CUDA, but I was pretty impressed with it and with their NSys tools. I'm curious what you wish was different.

saagarjha 1y ago
I actually really hate CUDA's programming model and feel like it's too low-level to actually get any productive work done. I don't really blame Nvidia because they basically invented the programmable GPU and it wouldn't be fair to have them also come up with the perfect programming model right out of the gate but at this point it's pretty clear that having independent threads work on their own programs makes no sense. High performance code requires scheduling across multiple threads in a way that is completely different if you are coming from CPUs. Of course, one might mention that GPUs are nothing like CPUs–but the programming model works super hard to try to hide this. So it's not really well designed in my book. I actually quite like the compilers that people are designing these days to write block-level code, because I feel like it better represents the work people want to do and then you pick which way you want it lowered. As for Nsight (Systems), it is…ok, I guess? It's fine for games and stuff I guess but for HPC or AI it doesn't really surface the information that you would want. People who are running their GPUs really hard know they have kernels running all the time and what the performance characteristics of them are. Nsight Compute is the thing that tells you that but it's kind of a mediocre profiler (some of this may be limitations of hardware performance counters) and to use it effectively you basically have to read a bunch of blog posts by people instead of official documentation. Despite not having used it much, my impression was that Nvidia's "moat" was that they have good networking libraries, that they are pretty good (relatively) and making sure all their tools work, and they have had consistent investment on this for a decade.
- electroglyph 1y ago
  i mean, it could be worse... it could be Vulkan
- jandrewrogers 1y ago
  GPUs are a type of barrel processor, which are optimized for workloads without cache locality. As a fundamental principle, they replace the CPU cache with latency hiding behavior. Consequently, you can't use algorithms and data structures designed for CPUs, since most of those assume the existence of a CPU cache. Some things are very cheap on a barrel processor that are very expensive on a CPU and vice versa, which changes the way you think about optimization. The wide vectors on GPUs are somewhat irrelevant. Scalar barrel processors exist and have the same issues. A scalar barrel processor feels deceptively CPU-like and will happily compile and run normal CPU code. The performance will nonetheless be poor unless the C++ code is designed to be a good fit for the nature of a barrel processor, code which will look weird and non-idiomatic to someone who has only written code for CPUs. There is no way to hide that a barrel processor is not a CPU even though they superficially have a lot of CPU-like properties. A barrel processor is extremely efficient once you learn to write code for them and exceptionally well-suited to HPC since they are not latency-sensitive. However, most people never learn how to write proper code for barrel processors. Ironically, barrel processor style code architecture is easy to translate into highly optimized CPU code, just not the reverse.
  - glitchc 1y ago
    I wanted to upvote you originally, but I'm afraid this is not correct. A GPU is not a barrel processor. In a barrel processor a single context is switched between multiple threads after each instruction. A barrel processor design has a singular instruction pipeline and a singular cache across all threads. In a GPU, due to the independence of the execution units, those threads will execute those instructions concurrently on all cores, as long as a program-based instruction dependency between threads is not introduced. It's true parallelism. Furthermore, each execution unit embeds its own instruction scheduler, it's own pipeline and its own L1 cache (see [1] for NVidia's architecture). [1] https://docs.nvidia.com/deeplearning/performance/dl-performance-gpu-background/index.html#gpu-arch https://docs.nvidia.com/deeplearning/performance/dl-performa...
    - jandrewrogers 1y ago
      Barrel processors are a spectrum and GPUs are on one end of it. Yes, the classic canonical barrel processors (e.g. Tera architecture) more or less work as you outline. That is a 40 year old microarchitecture, they haven't been designed that way for decades. Modern barrel processors implementations have complex microarchitectures that are much closer to a modern GPU in design. That is not accidental, the lineage is clearly there if you've worked on both. I will grant that vanishingly few people have ever seen or worked on a modern non-GPU barrel processor, since they are almost exclusively the domain of exotics built for government applications AFAICT.

Could you elaborate? I've only done a little work on CUDA, but I was pretty impressed with it and with their NSys tools. I'm curious what you wish was different.

saagarjha 1y ago

I actually really hate CUDA's programming model and feel like it's too low-level to actually get any productive work done. I don't really blame Nvidia because they basically invented the programmable GPU and it wouldn't be fair to have them also come up with the perfect programming model right out of the gate but at this point it's pretty clear that having independent threads work on their own programs makes no sense. High performance code requires scheduling across multiple threads in a way that is completely different if you are coming from CPUs. Of course, one might mention that GPUs are nothing like CPUs–but the programming model works super hard to try to hide this. So it's not really well designed in my book. I actually quite like the compilers that people are designing these days to write block-level code, because I feel like it better represents the work people want to do and then you pick which way you want it lowered. As for Nsight (Systems), it is…ok, I guess? It's fine for games and stuff I guess but for HPC or AI it doesn't really surface the information that you would want. People who are running their GPUs really hard know they have kernels running all the time and what the performance characteristics of them are. Nsight Compute is the thing that tells you that but it's kind of a mediocre profiler (some of this may be limitations of hardware performance counters) and to use it effectively you basically have to read a bunch of blog posts by people instead of official documentation. Despite not having used it much, my impression was that Nvidia's "moat" was that they have good networking libraries, that they are pretty good (relatively) and making sure all their tools work, and they have had consistent investment on this for a decade.

electroglyph 1y ago
i mean, it could be worse... it could be Vulkan
jandrewrogers 1y ago
GPUs are a type of barrel processor, which are optimized for workloads without cache locality. As a fundamental principle, they replace the CPU cache with latency hiding behavior. Consequently, you can't use algorithms and data structures designed for CPUs, since most of those assume the existence of a CPU cache. Some things are very cheap on a barrel processor that are very expensive on a CPU and vice versa, which changes the way you think about optimization. The wide vectors on GPUs are somewhat irrelevant. Scalar barrel processors exist and have the same issues. A scalar barrel processor feels deceptively CPU-like and will happily compile and run normal CPU code. The performance will nonetheless be poor unless the C++ code is designed to be a good fit for the nature of a barrel processor, code which will look weird and non-idiomatic to someone who has only written code for CPUs. There is no way to hide that a barrel processor is not a CPU even though they superficially have a lot of CPU-like properties. A barrel processor is extremely efficient once you learn to write code for them and exceptionally well-suited to HPC since they are not latency-sensitive. However, most people never learn how to write proper code for barrel processors. Ironically, barrel processor style code architecture is easy to translate into highly optimized CPU code, just not the reverse.
- glitchc 1y ago
  I wanted to upvote you originally, but I'm afraid this is not correct. A GPU is not a barrel processor. In a barrel processor a single context is switched between multiple threads after each instruction. A barrel processor design has a singular instruction pipeline and a singular cache across all threads. In a GPU, due to the independence of the execution units, those threads will execute those instructions concurrently on all cores, as long as a program-based instruction dependency between threads is not introduced. It's true parallelism. Furthermore, each execution unit embeds its own instruction scheduler, it's own pipeline and its own L1 cache (see [1] for NVidia's architecture). [1] https://docs.nvidia.com/deeplearning/performance/dl-performance-gpu-background/index.html#gpu-arch https://docs.nvidia.com/deeplearning/performance/dl-performa...
  - jandrewrogers 1y ago
    Barrel processors are a spectrum and GPUs are on one end of it. Yes, the classic canonical barrel processors (e.g. Tera architecture) more or less work as you outline. That is a 40 year old microarchitecture, they haven't been designed that way for decades. Modern barrel processors implementations have complex microarchitectures that are much closer to a modern GPU in design. That is not accidental, the lineage is clearly there if you've worked on both. I will grant that vanishingly few people have ever seen or worked on a modern non-GPU barrel processor, since they are almost exclusively the domain of exotics built for government applications AFAICT.