5 ms·

Layers All the Way Down: The Untold Story of Shader Compilation

91 points by birdculture 1y ago

shmerl 1y ago
> Rendering, by comparison, is a huge can of worms. Every platform has their own unique support matrix. GPU APIs landscape is stuck in the dumb NIH mentality of the '90s because stuck up lock-in proponents refuse to support Vulkan on their walled gardens. The only recent somewhat positive development about that was MS deciding to use SPIR-V and eventually ditch DXIL. A small step in the right direction, but not really enough yet.
- alexk101 1y ago
  I have been playing around with slang, which is supposed to be more cross platform. They have a neural rendering slant, and I have yet to fully test on all platforms, but I think it's a welcome move to consolidate all these apis. https://shader-slang.org/ https://shader-slang.org/
  - raphlinus 1y ago
    Yup, I think slang is the future. Anyone on this thread willing to fund a Rust implementation?
    - gmueckl 1y ago
      Why? Unless you need to embed the slang transpiler (unlikely), the language it is written in literally doesn't matter.
      - shmerl 1y ago
        I think Rust itself as a language for GPU programming is another interesting alternative: https://github.com/Rust-GPU/rust-gpu/ https://github.com/Rust-GPU/rust-gpu/
      - raphlinus 1y ago
        This is a longer and deeper conversation, but I think on topic for the original article, so I'll go into it a bit. The tl;dr is developer friction. By all means if you're doing a game (or another app with similar build requirements), figure out a shader precompilation pipeline so you're able to compile down to the lowest portable IR for each target, and ship that in your app bundle. Slang is meant for that, and this pipeline will almost certainly contain other tools written in C++ or even without source available (DXC, the Apple shader compiler tools, etc). There are two main use cases where we want different pieces of shaders to come from different sources of truth, and link them together downstream. One is integrating samplers for (vello_hybrid) sparse strip textures so those can be combined with user paint sources in the user's 2D or 3D app. The other is that we're trying to make the renderer more modular so we have separate libraries for color space conversion and image filters (blur etc). To get maximal performance, you don't want to write out the blur result to a full-resolution texture, but rather have a function that can sample from an intermediate result. See [1] for more context and discussion of that point. Stitching together these separate pieces of shader is a major potential source of developer friction. There is a happy path in the Rust ecosystem, albeit with some compromises, which is to fully embrace WGSL as the source of truth. The pieces can be combined with string-pasting, though we're looking at WESL as a more systematic approach. With WGSL, you can either do all your shader compilation at runtime (using wgpu for native), or do a build.rs script invoking naga to precompile. See [2] for the main PR that implements the latter in vello_hybrid. In the former case, you can even have hot reloading of shaders; implemented in Vello main but not (yet) vello_hybrid. To get the same quality of developer experience with Slang, you'd need an implementation in Rust. I think this would be a good thing for Slang. I've consistently underestimated the importance of developer friction in the past. As a contrast, we're also doing a CPU-only version of Vello now, and it's absolutely night and day, both for development velocity and attracting users. I think it's possible the GPU world gets better, but at the moment it's quite painful. I personally believe doing a Rust implementation of the Slang compiler would be an important step in the right direction, and is worth funding. Whether the rest of the world agrees with me, we'll see. [1]: https://xi.zulipchat.com/#narrow/channel/197075-vello/topic/Fast.20CPU.20blur/near/515102470 https://xi.zulipchat.com/#narrow/channel/197075-vello/topic/... [2]: https://github.com/linebender/vello/pull/1011 https://github.com/linebender/vello/pull/1011
    - flohofwoe 1y ago
      AFAIK a very large part of Slang are massively big 3rd party libraries written in C++, the Slang-specific Rust code would just be a very thin layer on top of millions(?) of lines of C++ code that has been grown over decades and is maintained elsewhere. (fwiw I've been considering to write the custom parts of the Sokol shader compiler in Zig instead of C++, but that's just a couple of thousand lines of glue code on top of massive C++ libraries (SPIRVTools, SPIRVCross, glslang and Tint), and those C++ APIs are terrible to work with from non-C++ languages. As far as developer friction for integration into asset workflows goes, that's exactly where I would prefer Zig over Rust (but a simple build.zig already goes most of the way without porting any code to Zig).
    - pjmlp 1y ago
      It is according to Khronos anyway, for those that aren't already deeply invested into HLSL. Khronos has been quite vocal that there is no further development on GLSL, they see that as a community effort, they only provide SPIR-V. This is how vendor specific tooling eventually wins out, they kind of got lucky AMD decided to offer Mantle as basis for Vulkan, LunarG is doing the SDK, now NVidia contributed slang, otherwise they would still be arguing about OpenGL vNext.
  - socalgal2 1y ago
    https://compute.toys/view/1948 https://compute.toys/view/1948 recently added Slang support
    - deleted 1y ago
      [deleted]
- socalgal2 1y ago
  Vulkan is by far the worst API of all the modern graphics API. It's crap. It's also not even trying to be portable, expecting you to query a million things and then adapt to the platform. I for one am glad Vulkan is not "it" Also, it's not even portable on Windows. Sure, if you're making a game, you can expect your gamer audiences have it installed. If you're making an app and you expect businesses to use you it you'll find neither OpenGL nor Vulkan work on most business class machines view Remote Desktop. The point being, it's not portable by design nor in actuality
  - viraptor 1y ago
    Why would remote desktop make a difference for the API? I thought MS killed RemoteFX without any replacement? As in, everyone including DirectX is in the same bad state...
    - pjmlp 1y ago
      DirectX works just fine in RDP. How do you think those Azure VMs for game developers work? https://learn.microsoft.com/en-us/azure/virtual-desktop/graphics-enable-gpu-acceleration https://learn.microsoft.com/en-us/azure/virtual-desktop/grap...
      - viraptor 1y ago
        Sure it works. But the message before suggested that OpenGL and Vulkan don't for some reason. As far as I know they all work though and have worked for years now.
        pjmlp 1y ago
        Probably because the drivers have to support it as well, I guess. https://www.khronos.org/news/permalink/nvidia-provides-opengl-accelerated-remote-desktop-for-geforce-5e88fc2035e342.98417181 https://www.khronos.org/news/permalink/nvidia-provides-openg... Also on desktop side, Windows on ARM, and UWP don't have that great support for anything beyond DirectX. There is a compatibility pack and another one Angle based, but they haven't been updated in a while.
- flohofwoe 1y ago
  I would agree if Vulkan would be a good 3D API, but it turned out to be the worst of the modern 3D APIs because it repeats the same main mistake as GL: There is no design vision, instead it's a cobbled together mess of adhoc vendor extensions that are eventually promoted to 'core'.
raphlinus 1y ago
We tried something like this with piet-gpu-hal. One problem is that spirv-cross is lossy, though gaps in target language support are getting better. For example, a device scoped barrier is just dropped on the floor before Metal 3.2. Atomics are also a source of friction. But the main problem is not the shader language itself, but the binding model. It's a pretty big mess, and things change as you go in the direction of bindless (descriptor indexing). There are a few approaches to this, certainly reinventing WebGPU is one. Another intriguing approach is blade[1] by Dzmitry Malyshau. I wish the authors well, but this is a hard road, especially if the goal is to enable more advanced features including compute. [1]: https://github.com/kvark/blade https://github.com/kvark/blade
- pjmlp 1y ago
  As expected, they don't touch the issue of how shaders work in PlayStation (LibGNM, LibGNMX) and Switch (NVN).
  - gmueckl 1y ago
    These systems are highly proprietary and Inam reasonably certain that stating anything about them publicly would break some NDAs.
    - pjmlp 1y ago
      As someone still having an Nintendo Developer Portal account, holding SCEE content back when the London Soho office used to have a developer site (aka Team Soho), and PS2Linux owner, there is plenty of material that can be discussed publicly without breaking NDAs.
      - flohofwoe 1y ago
        Console specific information also is not all that interesting anymore these days since game consoles have switched to off-the-shelf GPU designs with only minor modifications.
riggsdk 1y ago
BGFX (https://github.com/bkaradzic/bgfx https://github.com/bkaradzic/bgfx) uses a different approach. You basically write your shader in a GLSL-like language but it's all just (either very clever or very horrible) macro expansions that handles all the platform differences. With that you get a very performant backend for OpenGL, WebGL, Vulkan, Metal, Direct3D 11 and 12 and Playstation. It's shader compiler program basically does minimal source level transformations before handing it over to the platforms' own shader compiler if available.
mvdtnz 1y ago
This is a very interesting article, it was fun to learn what shaders are and why I have to wait for them to compile every time I play Call of Duty. What I'd like to know is why game developers can't provide a service / daemon that will perform shader compilation when shaders or drivers change so I don't need to waste 25 minutes of my precious gaming time when I load CoD and see they need compiling. My computer has been sitting there unused all week. Steam was smart enough to download game updates in that time, why isn't your game smart enough to recompile my shaders on an idle machine?
- gmueckl 1y ago
  Imagine the outcry if every game came with a background process that just sits there in the background. Also - and this may sound a little bonkers - some renderers are so complex and flexible that the actual set of required shaders is only discovered when it tries to render a frame and the full set of possible shader configuration permutations is so large that compiling them all in advance is pointless or borderline infeasible.
  - mvdtnz 1y ago
    Lots of games come with background processes.
    - gmueckl 1y ago
      Can you name an example? I have a large library of games and can't name one that does.
- Negitivefrags 1y ago
  Steam actually shares compiled shaders between users with the same hardware / driver version, but only for Vulkan I believe.
  - ThatPlayer 1y ago
    The shared compiled shaders are only for the Steam Deck I believe. Other hardware will require a background compile or will compile at launch.
voidUpdate 1y ago
If you need to compile shaders at runtime, then why do I have to wait 10 mins for unreal to compile 30,000 shaders whenever it feels like it?
- flohofwoe 1y ago
  Because the input bytecode blobs that are passed into 3d APIs are only a gpu-vendor- and render-pipeline-agnostic bytecode format (eg SPIRV or DXBC) - this is not what's actually running on the GPU. There's a second compile step happening inside the driver which compiles into a proprietary "machine code" format, and specialized for a specific pipeline object. Normally that driver internal compile step is quite fast, but that doesn't matter if there's tens of thousands of shader variants.
pornel 1y ago
I really like the WebGPU API. That's the API where the major players, including Apple and Microsoft, are forced to collaborate. It has real-world implementations on all major platforms. With the wgpu and Google dawn implementations, the API isn't actually tied to the Web, and can be used in native applications.
- pjmlp 1y ago
  The only reason I like WebGL and WebGPU is that they are the only 3D APIs where major players take managed language runtimes into consideration, because they can't do otherwise. Had WebAssembly already been there without being forced to go through JavaScript for Web APIs, and most likely they would be C APIs where everyone and their dog are writing bindings insteads. Now, it is pretty much a ChromeOS only API still, and only available across macOS, Android and Windows. Safari and Firefox have it as preview, and who knows when it will ever it stable at a scale that doesn't require "Works best on Chrome" banners. Support on GNU/Linux, even from Chrome, is pretty much not there, at least for something to use in production. And then we have the whole drama that after 15 years, there are still no usable developer tools on browsers for 3D debugging, one is forced to either guess what rendering calls are from the browser or which are from the application, GPU printf debugging, or having a native version that can be plugged into Renderdoc or similar.
- flohofwoe 1y ago
  > Microsoft With the switch to Chromium as browser engine, I don't think that Microsoft has any people working on WebGPU.
  - pjmlp 1y ago
    They do, if you look at https://github.com/gpuweb/gpuweb/wiki/Implementation-Status https://github.com/gpuweb/gpuweb/wiki/Implementation-Status they do attend meetings.

91 points by birdculture 1y ago

> Rendering, by comparison, is a huge can of worms. Every platform has their own unique support matrix. GPU APIs landscape is stuck in the dumb NIH mentality of the '90s because stuck up lock-in proponents refuse to support Vulkan on their walled gardens. The only recent somewhat positive development about that was MS deciding to use SPIR-V and eventually ditch DXIL. A small step in the right direction, but not really enough yet.

alexk101 1y ago
I have been playing around with slang, which is supposed to be more cross platform. They have a neural rendering slant, and I have yet to fully test on all platforms, but I think it's a welcome move to consolidate all these apis. https://shader-slang.org/ https://shader-slang.org/
- raphlinus 1y ago
  Yup, I think slang is the future. Anyone on this thread willing to fund a Rust implementation?
  - gmueckl 1y ago
    Why? Unless you need to embed the slang transpiler (unlikely), the language it is written in literally doesn't matter.
    - shmerl 1y ago
      I think Rust itself as a language for GPU programming is another interesting alternative: https://github.com/Rust-GPU/rust-gpu/ https://github.com/Rust-GPU/rust-gpu/
    - raphlinus 1y ago
      This is a longer and deeper conversation, but I think on topic for the original article, so I'll go into it a bit. The tl;dr is developer friction. By all means if you're doing a game (or another app with similar build requirements), figure out a shader precompilation pipeline so you're able to compile down to the lowest portable IR for each target, and ship that in your app bundle. Slang is meant for that, and this pipeline will almost certainly contain other tools written in C++ or even without source available (DXC, the Apple shader compiler tools, etc). There are two main use cases where we want different pieces of shaders to come from different sources of truth, and link them together downstream. One is integrating samplers for (vello_hybrid) sparse strip textures so those can be combined with user paint sources in the user's 2D or 3D app. The other is that we're trying to make the renderer more modular so we have separate libraries for color space conversion and image filters (blur etc). To get maximal performance, you don't want to write out the blur result to a full-resolution texture, but rather have a function that can sample from an intermediate result. See [1] for more context and discussion of that point. Stitching together these separate pieces of shader is a major potential source of developer friction. There is a happy path in the Rust ecosystem, albeit with some compromises, which is to fully embrace WGSL as the source of truth. The pieces can be combined with string-pasting, though we're looking at WESL as a more systematic approach. With WGSL, you can either do all your shader compilation at runtime (using wgpu for native), or do a build.rs script invoking naga to precompile. See [2] for the main PR that implements the latter in vello_hybrid. In the former case, you can even have hot reloading of shaders; implemented in Vello main but not (yet) vello_hybrid. To get the same quality of developer experience with Slang, you'd need an implementation in Rust. I think this would be a good thing for Slang. I've consistently underestimated the importance of developer friction in the past. As a contrast, we're also doing a CPU-only version of Vello now, and it's absolutely night and day, both for development velocity and attracting users. I think it's possible the GPU world gets better, but at the moment it's quite painful. I personally believe doing a Rust implementation of the Slang compiler would be an important step in the right direction, and is worth funding. Whether the rest of the world agrees with me, we'll see. [1]: https://xi.zulipchat.com/#narrow/channel/197075-vello/topic/Fast.20CPU.20blur/near/515102470 https://xi.zulipchat.com/#narrow/channel/197075-vello/topic/... [2]: https://github.com/linebender/vello/pull/1011 https://github.com/linebender/vello/pull/1011
  - flohofwoe 1y ago
    AFAIK a very large part of Slang are massively big 3rd party libraries written in C++, the Slang-specific Rust code would just be a very thin layer on top of millions(?) of lines of C++ code that has been grown over decades and is maintained elsewhere. (fwiw I've been considering to write the custom parts of the Sokol shader compiler in Zig instead of C++, but that's just a couple of thousand lines of glue code on top of massive C++ libraries (SPIRVTools, SPIRVCross, glslang and Tint), and those C++ APIs are terrible to work with from non-C++ languages. As far as developer friction for integration into asset workflows goes, that's exactly where I would prefer Zig over Rust (but a simple build.zig already goes most of the way without porting any code to Zig).
  - pjmlp 1y ago
    It is according to Khronos anyway, for those that aren't already deeply invested into HLSL. Khronos has been quite vocal that there is no further development on GLSL, they see that as a community effort, they only provide SPIR-V. This is how vendor specific tooling eventually wins out, they kind of got lucky AMD decided to offer Mantle as basis for Vulkan, LunarG is doing the SDK, now NVidia contributed slang, otherwise they would still be arguing about OpenGL vNext.
- socalgal2 1y ago
  https://compute.toys/view/1948 https://compute.toys/view/1948 recently added Slang support
  - deleted 1y ago
    [deleted]
socalgal2 1y ago
Vulkan is by far the worst API of all the modern graphics API. It's crap. It's also not even trying to be portable, expecting you to query a million things and then adapt to the platform. I for one am glad Vulkan is not "it" Also, it's not even portable on Windows. Sure, if you're making a game, you can expect your gamer audiences have it installed. If you're making an app and you expect businesses to use you it you'll find neither OpenGL nor Vulkan work on most business class machines view Remote Desktop. The point being, it's not portable by design nor in actuality
- viraptor 1y ago
  Why would remote desktop make a difference for the API? I thought MS killed RemoteFX without any replacement? As in, everyone including DirectX is in the same bad state...
  - pjmlp 1y ago
    DirectX works just fine in RDP. How do you think those Azure VMs for game developers work? https://learn.microsoft.com/en-us/azure/virtual-desktop/graphics-enable-gpu-acceleration https://learn.microsoft.com/en-us/azure/virtual-desktop/grap...
    - viraptor 1y ago
      Sure it works. But the message before suggested that OpenGL and Vulkan don't for some reason. As far as I know they all work though and have worked for years now.
      - pjmlp 1y ago
        Probably because the drivers have to support it as well, I guess. https://www.khronos.org/news/permalink/nvidia-provides-opengl-accelerated-remote-desktop-for-geforce-5e88fc2035e342.98417181 https://www.khronos.org/news/permalink/nvidia-provides-openg... Also on desktop side, Windows on ARM, and UWP don't have that great support for anything beyond DirectX. There is a compatibility pack and another one Angle based, but they haven't been updated in a while.
flohofwoe 1y ago
I would agree if Vulkan would be a good 3D API, but it turned out to be the worst of the modern 3D APIs because it repeats the same main mistake as GL: There is no design vision, instead it's a cobbled together mess of adhoc vendor extensions that are eventually promoted to 'core'.

raphlinus 1y ago

We tried something like this with piet-gpu-hal. One problem is that spirv-cross is lossy, though gaps in target language support are getting better. For example, a device scoped barrier is just dropped on the floor before Metal 3.2. Atomics are also a source of friction. But the main problem is not the shader language itself, but the binding model. It's a pretty big mess, and things change as you go in the direction of bindless (descriptor indexing). There are a few approaches to this, certainly reinventing WebGPU is one. Another intriguing approach is blade[1] by Dzmitry Malyshau. I wish the authors well, but this is a hard road, especially if the goal is to enable more advanced features including compute. [1]: https://github.com/kvark/blade https://github.com/kvark/blade

pjmlp 1y ago
As expected, they don't touch the issue of how shaders work in PlayStation (LibGNM, LibGNMX) and Switch (NVN).
- gmueckl 1y ago
  These systems are highly proprietary and Inam reasonably certain that stating anything about them publicly would break some NDAs.
  - pjmlp 1y ago
    As someone still having an Nintendo Developer Portal account, holding SCEE content back when the London Soho office used to have a developer site (aka Team Soho), and PS2Linux owner, there is plenty of material that can be discussed publicly without breaking NDAs.
    - flohofwoe 1y ago
      Console specific information also is not all that interesting anymore these days since game consoles have switched to off-the-shelf GPU designs with only minor modifications.

riggsdk 1y ago

BGFX (https://github.com/bkaradzic/bgfx https://github.com/bkaradzic/bgfx) uses a different approach. You basically write your shader in a GLSL-like language but it's all just (either very clever or very horrible) macro expansions that handles all the platform differences. With that you get a very performant backend for OpenGL, WebGL, Vulkan, Metal, Direct3D 11 and 12 and Playstation. It's shader compiler program basically does minimal source level transformations before handing it over to the platforms' own shader compiler if available.

mvdtnz 1y ago

This is a very interesting article, it was fun to learn what shaders are and why I have to wait for them to compile every time I play Call of Duty. What I'd like to know is why game developers can't provide a service / daemon that will perform shader compilation when shaders or drivers change so I don't need to waste 25 minutes of my precious gaming time when I load CoD and see they need compiling. My computer has been sitting there unused all week. Steam was smart enough to download game updates in that time, why isn't your game smart enough to recompile my shaders on an idle machine?

gmueckl 1y ago
Imagine the outcry if every game came with a background process that just sits there in the background. Also - and this may sound a little bonkers - some renderers are so complex and flexible that the actual set of required shaders is only discovered when it tries to render a frame and the full set of possible shader configuration permutations is so large that compiling them all in advance is pointless or borderline infeasible.
- mvdtnz 1y ago
  Lots of games come with background processes.
  - gmueckl 1y ago
    Can you name an example? I have a large library of games and can't name one that does.
Negitivefrags 1y ago
Steam actually shares compiled shaders between users with the same hardware / driver version, but only for Vulkan I believe.
- ThatPlayer 1y ago
  The shared compiled shaders are only for the Steam Deck I believe. Other hardware will require a background compile or will compile at launch.

voidUpdate 1y ago

If you need to compile shaders at runtime, then why do I have to wait 10 mins for unreal to compile 30,000 shaders whenever it feels like it?

flohofwoe 1y ago
Because the input bytecode blobs that are passed into 3d APIs are only a gpu-vendor- and render-pipeline-agnostic bytecode format (eg SPIRV or DXBC) - this is not what's actually running on the GPU. There's a second compile step happening inside the driver which compiles into a proprietary "machine code" format, and specialized for a specific pipeline object. Normally that driver internal compile step is quite fast, but that doesn't matter if there's tens of thousands of shader variants.

pornel 1y ago

I really like the WebGPU API. That's the API where the major players, including Apple and Microsoft, are forced to collaborate. It has real-world implementations on all major platforms. With the wgpu and Google dawn implementations, the API isn't actually tied to the Web, and can be used in native applications.

pjmlp 1y ago
The only reason I like WebGL and WebGPU is that they are the only 3D APIs where major players take managed language runtimes into consideration, because they can't do otherwise. Had WebAssembly already been there without being forced to go through JavaScript for Web APIs, and most likely they would be C APIs where everyone and their dog are writing bindings insteads. Now, it is pretty much a ChromeOS only API still, and only available across macOS, Android and Windows. Safari and Firefox have it as preview, and who knows when it will ever it stable at a scale that doesn't require "Works best on Chrome" banners. Support on GNU/Linux, even from Chrome, is pretty much not there, at least for something to use in production. And then we have the whole drama that after 15 years, there are still no usable developer tools on browsers for 3D debugging, one is forced to either guess what rendering calls are from the browser or which are from the application, GPU printf debugging, or having a native version that can be plugged into Renderdoc or similar.
flohofwoe 1y ago
> Microsoft With the switch to Chromium as browser engine, I don't think that Microsoft has any people working on WebGPU.
- pjmlp 1y ago
  They do, if you look at https://github.com/gpuweb/gpuweb/wiki/Implementation-Status https://github.com/gpuweb/gpuweb/wiki/Implementation-Status they do attend meetings.