4 ms·

Compiling from high-level lang to GPU is a huge problem, and we greatly appreciate efforts to solve it. If I understand correctly, this (CM) allows for C-style

Compiling from high-level lang to GPU is a huge problem, and we greatly appreciate efforts to solve it. If I understand correctly, this (CM) allows for C-style fine-level control over a GPU device as though it were a CPU. However, it does not appear to address data transit (critical for performance). Compilation and operator fusing to minimize transit is possibly more important. See Graphcore Poplar, Tensorflow XLA, Arrayfire, Pytorch Glow, etc. Further, this obviously only applies to Intel GPUs, so investing time in utilizing low-level control is possibly a hardware dead-end. Dream world for programmers is one where data transit and hardware architecture are taken into account without living inside a proprietary DSL Conversely, it is obviously against hardware manufacturers' interests to create this. Is MLIR / LLVM going to solve this? This list has been interesting to consider: https://github.com/merrymercy/awesome-tensor-compilers https://github.com/merrymercy/awesome-tensor-compilers

banachtarski 6y ago
I'm not a hardware engineer, but I am a GPU-focused graphics engineer. > C-style fine-level control over a GPU device as though it were a CPU. Personally, I think this is a fool's errand, and this has nothing to do with my desire for job security or anything. When I look at how code in the ML world is written for a GPU for example, it's really easy to see why it's so slow. The CPU and GPU architectures are fundamentally different. Different pipelining architecture, scalar instead of vector, 32/64-wide instruction dispatches, etc. HLSL/GLSL and other such shader languages are perfectly "high level" with other needed intrinsics needed to perform relevant warp level barriers, wave broadcasts/ballots/queries, use LDS storage, execute device level barriers, etc. This isn't to say that high level shader language improvements aren't welcome, but that trying to emulate a CPU is an unfortunate goal.
- mpweiher 6y ago
  What kinds of improvements would you like to see?
  - banachtarski 6y ago
    From an ergonomics point of view, it would be nice to have pointers directly into a descriptor heap, but this is coming already with SM 6.6 any day now. If I'm allowed to request hardware capabilities, I would like a non-divergent warp jmp instruction based on a function pointer table (much like is done with DXR-class raytracing hardware/shaders) to eliminate pipelining introduced only to mitigate shader permutation explosion.
baybal2 6y ago
> is possibly a hardware dead-end. I'm thinking the opposite, there been an unending succession of different accelerators for doing this, and that which eventually been obsoleted, and forgotten when general purpose CPU caught up to them in performance, or comp-sci learned how to do calculations more efficiently on mainstream hardware. Just by seeing how morbid are the sales of the new "NPUs," I can guess it's already happening. A number of cellphone brands experimented with such to run selfie filters, or do speech recognition, but later found that those work on CPUs not any worse at all if competent programmers are hired, and then threw the NPU hardware out, or disabled it.
- dragontamer 6y ago
  > I'm thinking the opposite, there been an unending succession of different accelerators for doing this, and that which eventually been obsoleted, and forgotten when general purpose CPU caught up to them in performance, or comp-sci learned how to do calculations more efficiently on mainstream hardware. There's only been two styles: * SIMD style compute (which has been around since Cray and CM-2, and continues today with AMD GPUs and NVidia GPUs). * Systolic-style compute -- which has been around since CD-ROM ASICs for decoding the CD-ROM realtime as it was spinning. Apparently Reed-Solomon decoding needed a ton of matrix-multiplications. Modern "NPU Hardware" is simply systolic compute. ------------ Traditional CPUs load data and then store data. The core itself can remain unchanging in the demands of different applications. Systolic Compute always loads/stores data in the same order, as it traverses across your systolic array. The systolic array designed for CD-ROM playback is quite different than the systolic array for NPUs. You gain faster speeds by building the hardware itself to handle the load/store pattern of the algorithm. That way the CPU-core itself doesn't waste time on loads/stores, nor do you have to worry about memory bandwidth issues or whatever. But by doing so: you lock yourself into a particular application. Modern neural-net NPUs can only do FP16 matrix multiplications. That's fine. If you want the fastest neural net in the world, systolic arrays will get you there. But that thing will ONLY be able to do neural net calculations, nothing else. Maybe you build the systolic array into an FPGA so that you can reconfigure the hardware later. Or maybe you just go with an ASIC because you're Google and can afford to build out clusters of ASICs of Systolic Arrays that can only do tensor-calculations.

banachtarski 6y ago

I'm not a hardware engineer, but I am a GPU-focused graphics engineer. > C-style fine-level control over a GPU device as though it were a CPU. Personally, I think this is a fool's errand, and this has nothing to do with my desire for job security or anything. When I look at how code in the ML world is written for a GPU for example, it's really easy to see why it's so slow. The CPU and GPU architectures are fundamentally different. Different pipelining architecture, scalar instead of vector, 32/64-wide instruction dispatches, etc. HLSL/GLSL and other such shader languages are perfectly "high level" with other needed intrinsics needed to perform relevant warp level barriers, wave broadcasts/ballots/queries, use LDS storage, execute device level barriers, etc. This isn't to say that high level shader language improvements aren't welcome, but that trying to emulate a CPU is an unfortunate goal.

mpweiher 6y ago
What kinds of improvements would you like to see?
- banachtarski 6y ago
  From an ergonomics point of view, it would be nice to have pointers directly into a descriptor heap, but this is coming already with SM 6.6 any day now. If I'm allowed to request hardware capabilities, I would like a non-divergent warp jmp instruction based on a function pointer table (much like is done with DXR-class raytracing hardware/shaders) to eliminate pipelining introduced only to mitigate shader permutation explosion.

baybal2 6y ago

> is possibly a hardware dead-end. I'm thinking the opposite, there been an unending succession of different accelerators for doing this, and that which eventually been obsoleted, and forgotten when general purpose CPU caught up to them in performance, or comp-sci learned how to do calculations more efficiently on mainstream hardware. Just by seeing how morbid are the sales of the new "NPUs," I can guess it's already happening. A number of cellphone brands experimented with such to run selfie filters, or do speech recognition, but later found that those work on CPUs not any worse at all if competent programmers are hired, and then threw the NPU hardware out, or disabled it.

dragontamer 6y ago
> I'm thinking the opposite, there been an unending succession of different accelerators for doing this, and that which eventually been obsoleted, and forgotten when general purpose CPU caught up to them in performance, or comp-sci learned how to do calculations more efficiently on mainstream hardware. There's only been two styles: * SIMD style compute (which has been around since Cray and CM-2, and continues today with AMD GPUs and NVidia GPUs). * Systolic-style compute -- which has been around since CD-ROM ASICs for decoding the CD-ROM realtime as it was spinning. Apparently Reed-Solomon decoding needed a ton of matrix-multiplications. Modern "NPU Hardware" is simply systolic compute. ------------ Traditional CPUs load data and then store data. The core itself can remain unchanging in the demands of different applications. Systolic Compute always loads/stores data in the same order, as it traverses across your systolic array. The systolic array designed for CD-ROM playback is quite different than the systolic array for NPUs. You gain faster speeds by building the hardware itself to handle the load/store pattern of the algorithm. That way the CPU-core itself doesn't waste time on loads/stores, nor do you have to worry about memory bandwidth issues or whatever. But by doing so: you lock yourself into a particular application. Modern neural-net NPUs can only do FP16 matrix multiplications. That's fine. If you want the fastest neural net in the world, systolic arrays will get you there. But that thing will ONLY be able to do neural net calculations, nothing else. Maybe you build the systolic array into an FPGA so that you can reconfigure the hardware later. Or maybe you just go with an ASIC because you're Google and can afford to build out clusters of ASICs of Systolic Arrays that can only do tensor-calculations.