5 ms·

The main problem with GPU architectures is that the manufacturers hide their details from us. Hell, they don't even document the instruction sets! At least NVID

The main problem with GPU architectures is that the manufacturers hide their details from us. Hell, they don't even document the instruction sets! At least NVIDIA doesn't. And they often don't tell us what the hardware actually _does_, and rather offer a mental model of how we can think about it. Specifically, those structural diagrams of functional-units within SM's you see in the blog post? That comes from NVIDIA. And they explicitly state that they do _not_ guarantee that this is what the hardware is actually like. The hardware works "as-if" it were made up of this kind of units. And even more specifically - it's not clear whether there even is such a thing as a "tensor core", or whether it's just some hack for doing lower-precision FP math somewhat faster. ----- Anyway, if the architectures weren't mostly _hidden_, we would make them far less mysterious within a rather short period of time.

wolletd 3y ago
Just to be fair: as also stated at the beginning of the post, CPUs are far more complex nowadays than they represent themselves to the outside. They don't work like a von-Neumann machine, they act like one. Granted, we know a lot more about the inner workings of modern CPUs thn GPUs, but a lot of real-life work still assumes that the CPU works "as-if" it was a computer from the 70s, just really, really fast.
- WithinReason 3y ago
  A lot of effort goes into CPU design to ensure you don't have to care, unlike GPUs.
  - amelius 3y ago
    Is that really a smart choice, given that we have compilers that can deal with the gory details?
    - myrmidon 3y ago
      Relying on the compiler to compensate for an unwieldy instruction set architecture is something that was tried explicitly with the Intel Itanium and failed hard. \rant I feel that generally people often have a notion of the "compiler is unbeatable at optimizing", which is completely false: You can always start from compiler output when optimizing and you often have easy options that are inaccessible for the compiler because of language semantics/calling conventions etc. (=> but hand optimizing at instruction level is time consuming, time is expensive, and what the compiler does is basically free ;).
    - WithinReason 3y ago
      Compilers can absolutely not deal with the complexity of GPUs.
dist-epoch 3y ago
GPU instructions sets change every year/generation. Your software would not run next year if you directly targeted the instruction set. NVIDIA does document their PTX instruction set (a level above what the hardware actually runs): https://docs.nvidia.com/cuda/parallel-thread-execution/index.html https://docs.nvidia.com/cuda/parallel-thread-execution/index...
- Xeamek 3y ago
  ...But literally the exact same is the case for CPUs and yet we do have public and constant instruction set for decades now? Altough ofcourse CPU's instruction are also just a frontend api that behind the scenes is implemented using microcode, which probably is much less stable. But the point is, if we could move one level 'closer' on gpus, just like we have it on cpus, it would stop the big buisness gate-keeping that exsists when it comes to current day GPU apis/libraries
  - dist-epoch 3y ago
    That's not true, you can run a 30 years old binary on modern CPUs. The machine code you feed to the CPU didn't change much. That's not true for GPUs, the machine code changes very frequently. You feed your "binary" (PTX, ...) to the driver, and the driver compiles it to the actual machine code of your actual GPU.
    - Xeamek 3y ago
      But that's not inherent due to gpus completely revolutionizing their aproach every generation, is it? The main difference is that with cpu, the translation unit is hidden inside the cpu itself. With gpus, the translation is moved from the device to the driver. Old openGL code also will run in card that is newer then code itself. The only difference is that with the cpus, it's open standard what is the instruction set, while on gpus, instruction sets are defined by third parties (DX12,Vulkan,OpenGL) while it falls to nvidia to implement them.
      - dist-epoch 3y ago
xw30992 3y ago
I think you and fancyfredbot, the top commenter at the moment, need to have a conversation.
refulgentis 3y ago
4 hours after posting, this is currently the second comment. The first comment assures me "NVIDIA are really keen you understand their hardware, to the extent they will give you insanely detailed tutorials on things like avoiding shared memory bank conflicts." I don't know who to believe!
- einpoklum 3y ago
  I'd say NVIDIA is somewhat keen on you being able to utilize their hardware. So, they gave you some tutorials (absolutely not insanely detailed), documentation about the API, and a bunch of programming advice and recommendations. They also provide you with a mental model you could use to think about the GPU's work. But - that is all not the same as telling you what actually happens. They don't even publish instruction latency formulae, for example.
atq2119 3y ago
On the claim that instruction sets are undocumented: https://gpuopen.com/amd-isa-documentation/ https://gpuopen.com/amd-isa-documentation/ Of course this doesn't apply to Nvidia, but MI-300 seems to be pretty viable for machine learning, so if you want things to change across the industry, more people need to put their money where their mouths are.
Arelius 3y ago
Indeed, Nvidia has been awful about sharing documentation. The best docs I know of out Nvidia, are actually provided by Nintendo. AMD on the otherhand, while is far from perfect, does try to provide documentation. CDNA docs, for instance: https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/white-papers/amd-cdna-3-white-paper.pdf https://www.amd.com/content/dam/amd/en/documents/instinct-te...

wolletd 3y ago

Just to be fair: as also stated at the beginning of the post, CPUs are far more complex nowadays than they represent themselves to the outside. They don't work like a von-Neumann machine, they act like one. Granted, we know a lot more about the inner workings of modern CPUs thn GPUs, but a lot of real-life work still assumes that the CPU works "as-if" it was a computer from the 70s, just really, really fast.

WithinReason 3y ago
A lot of effort goes into CPU design to ensure you don't have to care, unlike GPUs.
- amelius 3y ago
  Is that really a smart choice, given that we have compilers that can deal with the gory details?
  - myrmidon 3y ago
    Relying on the compiler to compensate for an unwieldy instruction set architecture is something that was tried explicitly with the Intel Itanium and failed hard. \rant I feel that generally people often have a notion of the "compiler is unbeatable at optimizing", which is completely false: You can always start from compiler output when optimizing and you often have easy options that are inaccessible for the compiler because of language semantics/calling conventions etc. (=> but hand optimizing at instruction level is time consuming, time is expensive, and what the compiler does is basically free ;).
  - WithinReason 3y ago
    Compilers can absolutely not deal with the complexity of GPUs.

dist-epoch 3y ago

GPU instructions sets change every year/generation. Your software would not run next year if you directly targeted the instruction set. NVIDIA does document their PTX instruction set (a level above what the hardware actually runs): https://docs.nvidia.com/cuda/parallel-thread-execution/index.html https://docs.nvidia.com/cuda/parallel-thread-execution/index...

Xeamek 3y ago
...But literally the exact same is the case for CPUs and yet we do have public and constant instruction set for decades now? Altough ofcourse CPU's instruction are also just a frontend api that behind the scenes is implemented using microcode, which probably is much less stable. But the point is, if we could move one level 'closer' on gpus, just like we have it on cpus, it would stop the big buisness gate-keeping that exsists when it comes to current day GPU apis/libraries
- dist-epoch 3y ago
  That's not true, you can run a 30 years old binary on modern CPUs. The machine code you feed to the CPU didn't change much. That's not true for GPUs, the machine code changes very frequently. You feed your "binary" (PTX, ...) to the driver, and the driver compiles it to the actual machine code of your actual GPU.
  - Xeamek 3y ago
    But that's not inherent due to gpus completely revolutionizing their aproach every generation, is it? The main difference is that with cpu, the translation unit is hidden inside the cpu itself. With gpus, the translation is moved from the device to the driver. Old openGL code also will run in card that is newer then code itself. The only difference is that with the cpus, it's open standard what is the instruction set, while on gpus, instruction sets are defined by third parties (DX12,Vulkan,OpenGL) while it falls to nvidia to implement them.
    - dist-epoch 3y ago

xw30992 3y ago

I think you and fancyfredbot, the top commenter at the moment, need to have a conversation.

refulgentis 3y ago

4 hours after posting, this is currently the second comment. The first comment assures me "NVIDIA are really keen you understand their hardware, to the extent they will give you insanely detailed tutorials on things like avoiding shared memory bank conflicts." I don't know who to believe!

einpoklum 3y ago
I'd say NVIDIA is somewhat keen on you being able to utilize their hardware. So, they gave you some tutorials (absolutely not insanely detailed), documentation about the API, and a bunch of programming advice and recommendations. They also provide you with a mental model you could use to think about the GPU's work. But - that is all not the same as telling you what actually happens. They don't even publish instruction latency formulae, for example.

atq2119 3y ago

On the claim that instruction sets are undocumented: https://gpuopen.com/amd-isa-documentation/ https://gpuopen.com/amd-isa-documentation/ Of course this doesn't apply to Nvidia, but MI-300 seems to be pretty viable for machine learning, so if you want things to change across the industry, more people need to put their money where their mouths are.

Arelius 3y ago

Indeed, Nvidia has been awful about sharing documentation. The best docs I know of out Nvidia, are actually provided by Nintendo. AMD on the otherhand, while is far from perfect, does try to provide documentation. CDNA docs, for instance: https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/white-papers/amd-cdna-3-white-paper.pdf https://www.amd.com/content/dam/amd/en/documents/instinct-te...