12 ms·

Building the DirectX shader compiler better than Microsoft?

327 points by emidoots 3y ago

mcraiha 3y ago
This is also related to Godot "The reason to make it optional is that Direct3D 12 support currently relies on the proprietary dxil.dll library from the DirectX Shader Compiler being shipped together with Godot, and shipping proprietary software goes against the mission of the Godot project." https://godotengine.org/article/dev-snapshot-godot-4-3-dev-3/ https://godotengine.org/article/dev-snapshot-godot-4-3-dev-3...
- moffkalast 3y ago
  One of those things that the end users won't ever give half a shit about. Same thing when installing some linux distros: "dO yoU wANT To doWNloAd AND iNStAll 3Rd paRty driVers" Yes. Yes I want my GPU, wifi and bluetooth to work. Get over yourselves and leave that checkbox checked by default.
  - matheusmoreira 3y ago
    Huh? I absolutely do not want proprietary drivers tainting my kernel. I barely tolerate firmware blobs. So far only nvidia has required such drivers.
  - dvngnt_ 3y ago
    some people do
  - AeroNotix 3y ago
    Speak for yourself.
  - gkbrk 3y ago
    GPU, Wifi and Bluetooth commonly work with open source drivers
    - comex 3y ago
      More often the issue is firmware rather than drivers.
  - TillE 3y ago
    It's not an arbitrary ideological decision, it's a legal one. Blame Microsoft for their dumb license on dxil.dll which, as I read it, would require Godot to add some kind of click-through agreement.
    - sterlind 3y ago
      Reminds me of the baffling license on redistributing MS's C runtime as a DLL rather than an MSI, leading to everyone having 12 different versions installed, and not being able to ship programs that run without being installed first. I have no clue why they did this. I work for MS and I'm as annoyed about it as anyone.
- malkia 3y ago
  dxil.dll is managed ("dotnet") code, right?
  - yazzku 3y ago
    No, it's C++, based on LLVM. https://github.com/microsoft/DirectXShaderCompiler https://github.com/microsoft/DirectXShaderCompiler Edit: apparently dxil.dll is not part of DXC (the classic move to make "open source" software dependent on external proprietary garbage, apparently.) But I'd still doubt it's a managed DLL.
- yazzku 3y ago
  Which part of dxil/dxc is proprietary exactly? Trying to make sense of the license barf at https://github.com/microsoft/DirectXShaderCompiler https://github.com/microsoft/DirectXShaderCompiler
  - charcircuit 3y ago
    The source code for dxil.dll is not part of that repo.
    - yazzku 3y ago
      Looks like this is another case of "Microsoft loves Open Source" then.
      - charcircuit 3y ago
        Are any libraries from the Windows SDK open source? Windows application code calling into libraries that are not open source is nothing new.
      - Dylan16807 3y ago
        If someone says they love pizza, do you call them a faker every time they eat something else?
  - zerocrates 3y ago
    The license (and the code) for dxil.dll/libdxil.so isn't in that repo, they just include the blob in releases. If you look at a release you'll see an additional LICENSE-MS.txt that just covers that dxil signing library.
    - yazzku 3y ago
      How is that compatible with the GPL licence from autoconf?
      - Dylan16807 3y ago
flohofwoe 3y ago
A great overview of the terrible mess that underlies cross-3D-API shader compilation (and even though it focuses on D3D and Microsoft, it's not much better on other 3D APIs - for instance you simply can't cross-compile Metal shaders from a Linux host - only from macOS and - somewhat recently - Windows). If the Mach team can pull off this whole "Zig as a cross-3D-API shader compiler" and make it work as smoothly as "Zig as a cross-compilation toolchain", then this would be pretty much the biggest thing in computer graphics since 1995 or so :)
- lr1970 3y ago
  > If the Mach team can pull off this whole "Zig as a cross-3D-API shader compiler" and make it work as smoothly as "Zig as a cross-compilation toolchain", then this would be pretty much the biggest thing in computer graphics since 1995 or so :) And a major boost for Zig -- both as a language and a toolchain.
- astrange 3y ago
  If you can do it on Windows doesn't that mean you can do it on Linux with Wine?
  - flohofwoe 3y ago
    It probably works, but Jenga tower hacks like that is exactly the problem ;)
- matheusmoreira 3y ago
  > for instance you simply can't cross-compile Metal shaders from a Linux host Why not? Doesn't that just mean that nobody has implemented it yet?
  - flohofwoe 3y ago
    The Metal shader compiler is only distributed as binary blobs for Mac and Windows. Reverse engineering of the undocumented output is possible of course, but that also means keeping track of all the changes Apple does to the output format in new versions.
lastmartyr 3y ago
[flagged]
Voultapher 3y ago
What a joy to read. It's this sort of infrastructure work that unlocks so many doors. Thank you Stephen!
mouse_ 3y ago
Microsoft has no incentive to make good software. Most people will use it no matter what.
- lukan 3y ago
  That is not true. People use it as long as the pain of using it, is lower than the pain of switching. Assuming they have even something to switch for.
  - npteljes 3y ago
    >Assuming they have even something to switch for. This one hits the nail on the head, and the reason why not just Microsoft, but a lot of large software players are not incentivized to create better software. At the end of the day, people like power, to make money, and the people at Microsoft are no exception. And businesses are businesses, enterprises to make money, not altruistic benefactors of humanity, or optimizers of a specific domain, like software. So what business will do are their original thing AND business tactics, and the larger the business, the more tactics they have to employ, otherwise they won't be as large, or even simply won't be, at all. So, on the top, it's all ruthless business tactics. As Microsoft is a large player for a long time, they have quite the rep sheet[0], but they are not unique in doing this. It's the name of the game. [0] https://en.wikipedia.org/wiki/Criticism_of_Microsoft https://en.wikipedia.org/wiki/Criticism_of_Microsoft
  - pjmlp 3y ago
    Thankfully Valve is doing the good work to keep game developers targeting Windows and DirectX without caring about alternatives on the PC space.
    - timlatim 3y ago
      Apologies if I'm misreading your intention, but are you suggesting that Valve's work on Wine is somehow worse than asking game developers to target Linux/other OSes natively? As a Linux desktop enthusiast, I much prefer the Valve's approach: the library of existing Windows-only games that are unlikely to be ever ported is too vast, and the benefits of targeting a disjointed[1] platform with <2% market share[2] for new games are not at all clear. It's only thanks to Valve that I (and hopefully many other Linux users) do not need to maintain a second Windows system for fun, as the majority of games run perfectly fine on Linux and require nothing more than clicking Install then Play in the Steam client. [1] Case in point: glibc's compatibility guarantees are weaker than what you get on Windows. (For instance, your system's glibc cannot be older than what a game is built against, which may present problems for devs using Fedora/Arch and players on Debian/LTS Ubuntu, something I've experienced first-hand for my apps.) The X11 to Wayland migration is also still underway. (Though things are getting better, the attitudes of some Wayland maintainers are a bit concerning: "I don't [care] what you think is normal behavior for games. You get certain guarantees with wayland. Deal with it. If clients decide to do exactly what they do on windows or X11 they won't work correctly." [3] I'm not sure game developers would enjoy such reception.) [2] https://store.steampowered.com/hwsurvey https://store.steampowered.com/hwsurvey [3] https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18359#note_1832873 https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18...
Emigre_ 3y ago
This is excellent. Bravo, mister! Really interesting stuff.
msk-lywenn 3y ago
So the signing[1] DXIL.dll does is just a modified MD5? 1: https://github.com/hexops/DirectXShaderCompiler/blob/4190bb0c90d374c6b4d0b0f2c7b45b604eda24b6/tools/clang/tools/dxcompiler/MachSiegbertVogtDXCSA.cpp#L178 https://github.com/hexops/DirectXShaderCompiler/blob/4190bb0...
- atq2119 3y ago
  That "signing" has always been bullshit security theater. You'll note that every other graphics API manages just fine without it. I'm glad somebody posted the "signing" algorithm out in the open.
  - TillE 3y ago
    Putting a pseudo-MD5 hash in your file header sounds like someone got confused and wound up caught between over-elaborate integrity check and half-hearted security measure. It sort of works if your signing tool is part of a private console SDK, but the DirectX SDK was always freely available.
- Culonavirus 3y ago
  This is the part of HN I really like. People who can look at random C code working with memory and be like "yep this looks like modified X". Pretty amazing (to someone like me who mostly works in high level languages).
  - msk-lywenn 3y ago
    With the phrasing of the article, I immediately thought it must have been something simple/dumb. I quickly thought MD5 so I looked at the magic numbers in the C code and looked them up on wikipedia. Then I noticed the four F G H I functions. Still I’m not sure it is (maybe it’s something common in hashes?) and I still don’t know what the change is
    - EMIRELADERO 3y ago
      From the commit message[1]: "dxil.dll is closed-source, so we cannot simply patch it in the same way. To fix this, we outright disable runtime loading of dxil.dll and silence warnings related to it not being present / the final binary not being code signed. Instead, once the compiler would emit a compiled shader blob, we perform our own code signing algorithm (Mach Siegbert Vogt DXCSA) which results in a bitwise identical compiled shader, and thus dxil.dll is no longer needed to perform code signing of shaders." [1] https://github.com/hexops/DirectXShaderCompiler/commit/7a0138d6eab5ce712e6dc70d3dc200eb2193574f https://github.com/hexops/DirectXShaderCompiler/commit/7a013...
Franscoben 3y ago
[dead]
nraynaud 3y ago
Somebody went to take a big one for the team Open Source, thank you.
sylware 3y ago
I guess the less troublesome would be something like HLSL/GLSL->SPIR-V<->DXIL, or shaders could be authored directly in SPIR-V. In vkd3d (from wine), I think you have a DXIL->SPIR-V translater (much more robust than any high level shading language converter since it is a much more simple intermediate language). That said, apart from this abomination of llvm, is there a HLSL->DXIL compiler written in plain and simple C99 (namely which require NOT gcc or clang to compile)?
- noahwhygodwhy 3y ago
  "Shaders could be authored directly in SPIRV" oh god please no lmao Also to answer your question, no. The hlsl to dxil translation is basically owned by microsoft. There's been little effort to move away from that
  - NekkoDroid 3y ago
    Well... they are rewriting & upstreaming DXC to LLVM main (https://github.com/microsoft/DirectXShaderCompiler/wiki/Contributing-HLSL-to-LLVM-main https://github.com/microsoft/DirectXShaderCompiler/wiki/Cont...) and want to kinda deprecate DXC. IIRC currently only compute shaders are supported but I may be very wrong.
    - sylware 3y ago
      llvm (apple) is a mistake, but I am not expecting microsoft to do anything else than mistakes anyway.
  - sylware 3y ago
    I am really serious about direct authoring of shaders using SPIR-V (with probably a little SSA checker on the side). We can reasonably think it will significantly increase the compatibility and bug avoidance of shaders among various drivers (and that is priceless when you think QA of big video games spaning many different drivers). I am perfectly aware this will require a bit more upfront work to write shaders, but due to their life cycle, it should be benign and we should get all the benefits (not even mentioning to free the SDK dependency from a massive and complex high level shading language compiler). I am going to give it a try. I need first a SPIR-V assembler, the one from the khronos spriv tools and the one from llvm are c++ diarrhea then a definitive nono, have to write my own. Let's think long run here: we don't have a _REAL_ standard very high level language yet (python? lua? javascript? perl5? ruby? so_many_others?), I'll go rv64 assembly then (I'll write a mini rv64 interpreter for x86_64).
detay 3y ago
Very promising and deserves support.
a1o 3y ago
SDL people are working in creating some shader language in the form of SDL_gpu (not the old one), to launch in SDL3, so it could have some cross platform way to work with 3D graphics there for games. I have been looking at it closely for a little while.
- jsheard 3y ago
  Is SDL_gpu meaningfully different from WebGPU? Looking at their goals they seem to be more or less the same, building an accessible lowest-common-denominator abstraction over DX12/Vulkan/Metal, except SDL_gpu is an early work in progress and WebGPU already has two good open implementations.
  - stephc_int13 3y ago
    Not sure about WebGPU. Is it really good? Have you checked with some benchmarks? From my experience, it's been disappointing. Big, complex to setup, not convinced about many tradeoffs.
    - jsheard 3y ago
      WebGPU running on the web necessarily has overhead compared to native due to all the extra validation it has to do for security, but there's nothing stopping native implementations from offering unsafe escape hatches as extensions for those who want them.
      - kvark 3y ago
        And wgpu has been doing this for years. Things like descriptor indexing are not exposed to the web but used by Rust (mostly) engines on native. https://wgpu.rs/ https://wgpu.rs/
cactusplant7374 3y ago
Playing Halo CE on Mac M1 with wine is an abject failure. I am assuming it is because Direct3D support is implemented poorly?
- Cu3PO42 3y ago
  Maybe. If you haven't tried it, you could look into Apple's own Direct3D to Metal translation layer from Game Porting Toolkit. It generally gives better results than DXVK chained with MoltenVK.
steeve 3y ago
I highly advise people to look into the Mach ecosytem, especially mach-sysgpu, which is a complete reimplementation of WebGPU, written (mostly) by Ali Chraghi, who is 17
fifteen1506 3y ago
Microsoft <3 Open Source
delta_p_delta_x 3y ago
> However, it doesn’t require any additional .dlls to be shipped with the application. Many video games already do this with all the proprietary middleware they use (Bink, SpeedTree, PhysX, etc). Most launchers (Steam, GOG, Epic, etc) also require their respective .DLLs. Many games also use D3D11On12. Many shipping games in my list have dxil.dll amongst their installed files. Therefore, an honest question: what's the problem with shipping an additional DLL? The work done here to reverse-engineer and re-implement the code-signing is fantastic—especially the fact that it is bitwise equal to dxil.dll's output. But I am ridiculously lazy and prefer to take the easier way out, and would've just shipped the DLL.
- msk-lywenn 3y ago
  As you pointed out, it’s really not a problem. However, the article points out at the end the real interesting bit: cross building from any OS/arch.
- chris37879 3y ago
  This opensource implementation can be included in existing game engines without requiring the programmer to know or understand what dxil.dll even is. The Mach engine, for instance, can use this to create a compiler that compiles zig code into shaders across all its targeted platforms in a way that doesn't require any additional setup or dependencies for the end user, it's just built into Mach's core.
- badsectoracula 3y ago
  There are more games than AAA games and there are more graphics applications than games. Something that is fine for your average 100GB AAA game may not be for a 30MB game (of which 24MB are DXC) on itch.io or a 3D model viewer/converter/animator/whatever on someone's website. In addition, even for AAA games (but also for other games and software, this adds extra dependencies that may be broken outside your control. At a company i worked several years ago, we spent more effort upgrading Visual Studio because for some middle we only had DLLs (and their libraries) to link against and had to obtain new versions (part of the effort was that the company that wrote said middleware hadn't upgraded their own so we also became QA for their compatibility with the newer VS, though i don't remember if there were any issues in practice). Of course having the code doesn't mean updates will be frictionless, but there will certainly be way less friction and you wont need to wait for someone else. (btw i think Visual Studio has tried to be backwards compatible in recent versions with binary C++ libraries, but i don't think this is something you can rely on in the long term) Also all that are with the assumption that the requirement is that your code remain on the same platform and targets. But at some point you may want to work with another platform (be it as host or target) and not having source code can make that from incredibly difficult to impossible. This may not seem much of an issue for something as platform-specific as DXIL - but actually notice the article mentioning that the binary blobbiness of DXIL made it impossible to precompile shaders on platforms outside specific Windows and Linux architectures for which the DLL was provided by Microsoft.
nightowl_games 3y ago
Using zig itself as a shading language is awesome. Zig is the one true language. It's a build system! It's a shading language! Wow!
unnouinceput 3y ago
Quote: "All that was left was that pesky dxil.dll - what sort of magic might Microsoft be employing in that library to “sign shaders”? How can they prevent unsigned shaders from running on Windows machines that aren’t in developer mode? How are they able to distribute that binary on Linux, too? I won’t comment on any of those questions, but will say that you’ll find dxil.dll is NOT a dependency of mach-dxcompiler in any form. You can compile an HLSL shader on a macOS machine using mach-dxcompiler, without the proprietary dxil.dll blob - and end up with a DXIL bytecode file that is byte-for-byte equal to one which runs it on a standard Windows box. Enjoy!" That above is the real magic. Since he won't comment on the how, I guess he took a swing at poping the hood underneath, and did exactly what Wine developers did 2 decades ago. Any old timers here remembering that scandal? Smart kid to not comment on the how, this way Microsoft won't have any legal leg and since times have changed with all that "Microsoft loves Linux" shit they yell at all corners (not that I believe that for a single second), then it will be swept all under the rug and FOSS wins. For now.
- rofrol 3y ago
  > So the signing[1] DXIL.dll does is just a modified MD5? [1] https://github.com/hexops/DirectXShaderCompiler/blob/4190bb0c90d374c6b4d0b0f2c7b45b604eda24b6/tools/clang/tools/dxcompiler/MachSiegbertVogtDXCSA.cpp#L178 https://github.com/hexops/DirectXShaderCompiler/blob/4190bb0... https://news.ycombinator.com/item?id=39325654 https://news.ycombinator.com/item?id=39325654
- charcircuit 3y ago
  >Since he won't comment on the how, I guess he took a swing at poping the hood underneath The way signing is done is already public knowledge that can be found using google as other open source projects have implemented it.
deleted 3y ago
[deleted]
trevortheblack 3y ago
DXIL is pronounced "dixel", like "pixel" but replace the p with a d
chrsig 3y ago
> DXC’s fork of LLVM removed and/or damaged much of the code generation layer and infrastructure [of LLVM]. Given that, supporting DXBC generation in DXC would be a massive task to fix and restore broken LLVM functionality. Due to the large scale of this issue and resource constraints on our team we’re not going to address this issue in [the new] DXC [compiler] ever. > > We may support DXBC generation in Clang in the future (we mentioned that in the original proposal to LLVM). That work is unlikely to begin for a few years as our focus will be on supporting DXIL and SPIR-V generation first. I appreciate this quote[0] from the microsoft camp. Setting clear expectations that something will not be done is a nice bit of fresh air. [0] https://github.com/microsoft/DirectXShaderCompiler/issues/5773#issuecomment-1735794551 https://github.com/microsoft/DirectXShaderCompiler/issues/57...

327 points by emidoots 3y ago

This is also related to Godot "The reason to make it optional is that Direct3D 12 support currently relies on the proprietary dxil.dll library from the DirectX Shader Compiler being shipped together with Godot, and shipping proprietary software goes against the mission of the Godot project." https://godotengine.org/article/dev-snapshot-godot-4-3-dev-3/ https://godotengine.org/article/dev-snapshot-godot-4-3-dev-3...

moffkalast 3y ago
One of those things that the end users won't ever give half a shit about. Same thing when installing some linux distros: "dO yoU wANT To doWNloAd AND iNStAll 3Rd paRty driVers" Yes. Yes I want my GPU, wifi and bluetooth to work. Get over yourselves and leave that checkbox checked by default.
- matheusmoreira 3y ago
  Huh? I absolutely do not want proprietary drivers tainting my kernel. I barely tolerate firmware blobs. So far only nvidia has required such drivers.
- dvngnt_ 3y ago
  some people do
- AeroNotix 3y ago
  Speak for yourself.
- gkbrk 3y ago
  GPU, Wifi and Bluetooth commonly work with open source drivers
  - comex 3y ago
    More often the issue is firmware rather than drivers.
- TillE 3y ago
  It's not an arbitrary ideological decision, it's a legal one. Blame Microsoft for their dumb license on dxil.dll which, as I read it, would require Godot to add some kind of click-through agreement.
  - sterlind 3y ago
    Reminds me of the baffling license on redistributing MS's C runtime as a DLL rather than an MSI, leading to everyone having 12 different versions installed, and not being able to ship programs that run without being installed first. I have no clue why they did this. I work for MS and I'm as annoyed about it as anyone.
malkia 3y ago
dxil.dll is managed ("dotnet") code, right?
- yazzku 3y ago
  No, it's C++, based on LLVM. https://github.com/microsoft/DirectXShaderCompiler https://github.com/microsoft/DirectXShaderCompiler Edit: apparently dxil.dll is not part of DXC (the classic move to make "open source" software dependent on external proprietary garbage, apparently.) But I'd still doubt it's a managed DLL.
yazzku 3y ago
Which part of dxil/dxc is proprietary exactly? Trying to make sense of the license barf at https://github.com/microsoft/DirectXShaderCompiler https://github.com/microsoft/DirectXShaderCompiler
- charcircuit 3y ago
  The source code for dxil.dll is not part of that repo.
  - yazzku 3y ago
    Looks like this is another case of "Microsoft loves Open Source" then.
    - charcircuit 3y ago
      Are any libraries from the Windows SDK open source? Windows application code calling into libraries that are not open source is nothing new.
    - Dylan16807 3y ago
      If someone says they love pizza, do you call them a faker every time they eat something else?
- zerocrates 3y ago
  The license (and the code) for dxil.dll/libdxil.so isn't in that repo, they just include the blob in releases. If you look at a release you'll see an additional LICENSE-MS.txt that just covers that dxil signing library.
  - yazzku 3y ago
    How is that compatible with the GPL licence from autoconf?
    - Dylan16807 3y ago

flohofwoe 3y ago

A great overview of the terrible mess that underlies cross-3D-API shader compilation (and even though it focuses on D3D and Microsoft, it's not much better on other 3D APIs - for instance you simply can't cross-compile Metal shaders from a Linux host - only from macOS and - somewhat recently - Windows). If the Mach team can pull off this whole "Zig as a cross-3D-API shader compiler" and make it work as smoothly as "Zig as a cross-compilation toolchain", then this would be pretty much the biggest thing in computer graphics since 1995 or so :)

lr1970 3y ago
> If the Mach team can pull off this whole "Zig as a cross-3D-API shader compiler" and make it work as smoothly as "Zig as a cross-compilation toolchain", then this would be pretty much the biggest thing in computer graphics since 1995 or so :) And a major boost for Zig -- both as a language and a toolchain.
astrange 3y ago
If you can do it on Windows doesn't that mean you can do it on Linux with Wine?
- flohofwoe 3y ago
  It probably works, but Jenga tower hacks like that is exactly the problem ;)
matheusmoreira 3y ago
> for instance you simply can't cross-compile Metal shaders from a Linux host Why not? Doesn't that just mean that nobody has implemented it yet?
- flohofwoe 3y ago
  The Metal shader compiler is only distributed as binary blobs for Mac and Windows. Reverse engineering of the undocumented output is possible of course, but that also means keeping track of all the changes Apple does to the output format in new versions.

lastmartyr 3y ago

[flagged]

Voultapher 3y ago

What a joy to read. It's this sort of infrastructure work that unlocks so many doors. Thank you Stephen!

mouse_ 3y ago

Microsoft has no incentive to make good software. Most people will use it no matter what.

lukan 3y ago
That is not true. People use it as long as the pain of using it, is lower than the pain of switching. Assuming they have even something to switch for.
- npteljes 3y ago
  >Assuming they have even something to switch for. This one hits the nail on the head, and the reason why not just Microsoft, but a lot of large software players are not incentivized to create better software. At the end of the day, people like power, to make money, and the people at Microsoft are no exception. And businesses are businesses, enterprises to make money, not altruistic benefactors of humanity, or optimizers of a specific domain, like software. So what business will do are their original thing AND business tactics, and the larger the business, the more tactics they have to employ, otherwise they won't be as large, or even simply won't be, at all. So, on the top, it's all ruthless business tactics. As Microsoft is a large player for a long time, they have quite the rep sheet[0], but they are not unique in doing this. It's the name of the game. [0] https://en.wikipedia.org/wiki/Criticism_of_Microsoft https://en.wikipedia.org/wiki/Criticism_of_Microsoft
- pjmlp 3y ago
  Thankfully Valve is doing the good work to keep game developers targeting Windows and DirectX without caring about alternatives on the PC space.
  - timlatim 3y ago
    Apologies if I'm misreading your intention, but are you suggesting that Valve's work on Wine is somehow worse than asking game developers to target Linux/other OSes natively? As a Linux desktop enthusiast, I much prefer the Valve's approach: the library of existing Windows-only games that are unlikely to be ever ported is too vast, and the benefits of targeting a disjointed[1] platform with <2% market share[2] for new games are not at all clear. It's only thanks to Valve that I (and hopefully many other Linux users) do not need to maintain a second Windows system for fun, as the majority of games run perfectly fine on Linux and require nothing more than clicking Install then Play in the Steam client. [1] Case in point: glibc's compatibility guarantees are weaker than what you get on Windows. (For instance, your system's glibc cannot be older than what a game is built against, which may present problems for devs using Fedora/Arch and players on Debian/LTS Ubuntu, something I've experienced first-hand for my apps.) The X11 to Wayland migration is also still underway. (Though things are getting better, the attitudes of some Wayland maintainers are a bit concerning: "I don't [care] what you think is normal behavior for games. You get certain guarantees with wayland. Deal with it. If clients decide to do exactly what they do on windows or X11 they won't work correctly." [3] I'm not sure game developers would enjoy such reception.) [2] https://store.steampowered.com/hwsurvey https://store.steampowered.com/hwsurvey [3] https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18359#note_1832873 https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18...

Emigre_ 3y ago

This is excellent. Bravo, mister! Really interesting stuff.

msk-lywenn 3y ago

So the signing[1] DXIL.dll does is just a modified MD5? 1: https://github.com/hexops/DirectXShaderCompiler/blob/4190bb0c90d374c6b4d0b0f2c7b45b604eda24b6/tools/clang/tools/dxcompiler/MachSiegbertVogtDXCSA.cpp#L178 https://github.com/hexops/DirectXShaderCompiler/blob/4190bb0...

atq2119 3y ago
That "signing" has always been bullshit security theater. You'll note that every other graphics API manages just fine without it. I'm glad somebody posted the "signing" algorithm out in the open.
- TillE 3y ago
  Putting a pseudo-MD5 hash in your file header sounds like someone got confused and wound up caught between over-elaborate integrity check and half-hearted security measure. It sort of works if your signing tool is part of a private console SDK, but the DirectX SDK was always freely available.
Culonavirus 3y ago
This is the part of HN I really like. People who can look at random C code working with memory and be like "yep this looks like modified X". Pretty amazing (to someone like me who mostly works in high level languages).
- msk-lywenn 3y ago
  With the phrasing of the article, I immediately thought it must have been something simple/dumb. I quickly thought MD5 so I looked at the magic numbers in the C code and looked them up on wikipedia. Then I noticed the four F G H I functions. Still I’m not sure it is (maybe it’s something common in hashes?) and I still don’t know what the change is
  - EMIRELADERO 3y ago
    From the commit message[1]: "dxil.dll is closed-source, so we cannot simply patch it in the same way. To fix this, we outright disable runtime loading of dxil.dll and silence warnings related to it not being present / the final binary not being code signed. Instead, once the compiler would emit a compiled shader blob, we perform our own code signing algorithm (Mach Siegbert Vogt DXCSA) which results in a bitwise identical compiled shader, and thus dxil.dll is no longer needed to perform code signing of shaders." [1] https://github.com/hexops/DirectXShaderCompiler/commit/7a0138d6eab5ce712e6dc70d3dc200eb2193574f https://github.com/hexops/DirectXShaderCompiler/commit/7a013...

Franscoben 3y ago

[dead]

nraynaud 3y ago

Somebody went to take a big one for the team Open Source, thank you.

sylware 3y ago

I guess the less troublesome would be something like HLSL/GLSL->SPIR-V<->DXIL, or shaders could be authored directly in SPIR-V. In vkd3d (from wine), I think you have a DXIL->SPIR-V translater (much more robust than any high level shading language converter since it is a much more simple intermediate language). That said, apart from this abomination of llvm, is there a HLSL->DXIL compiler written in plain and simple C99 (namely which require NOT gcc or clang to compile)?

noahwhygodwhy 3y ago
"Shaders could be authored directly in SPIRV" oh god please no lmao Also to answer your question, no. The hlsl to dxil translation is basically owned by microsoft. There's been little effort to move away from that
- NekkoDroid 3y ago
  Well... they are rewriting & upstreaming DXC to LLVM main (https://github.com/microsoft/DirectXShaderCompiler/wiki/Contributing-HLSL-to-LLVM-main https://github.com/microsoft/DirectXShaderCompiler/wiki/Cont...) and want to kinda deprecate DXC. IIRC currently only compute shaders are supported but I may be very wrong.
  - sylware 3y ago
    llvm (apple) is a mistake, but I am not expecting microsoft to do anything else than mistakes anyway.
- sylware 3y ago
  I am really serious about direct authoring of shaders using SPIR-V (with probably a little SSA checker on the side). We can reasonably think it will significantly increase the compatibility and bug avoidance of shaders among various drivers (and that is priceless when you think QA of big video games spaning many different drivers). I am perfectly aware this will require a bit more upfront work to write shaders, but due to their life cycle, it should be benign and we should get all the benefits (not even mentioning to free the SDK dependency from a massive and complex high level shading language compiler). I am going to give it a try. I need first a SPIR-V assembler, the one from the khronos spriv tools and the one from llvm are c++ diarrhea then a definitive nono, have to write my own. Let's think long run here: we don't have a _REAL_ standard very high level language yet (python? lua? javascript? perl5? ruby? so_many_others?), I'll go rv64 assembly then (I'll write a mini rv64 interpreter for x86_64).

detay 3y ago

Very promising and deserves support.

a1o 3y ago

SDL people are working in creating some shader language in the form of SDL_gpu (not the old one), to launch in SDL3, so it could have some cross platform way to work with 3D graphics there for games. I have been looking at it closely for a little while.

jsheard 3y ago
Is SDL_gpu meaningfully different from WebGPU? Looking at their goals they seem to be more or less the same, building an accessible lowest-common-denominator abstraction over DX12/Vulkan/Metal, except SDL_gpu is an early work in progress and WebGPU already has two good open implementations.
- stephc_int13 3y ago
  Not sure about WebGPU. Is it really good? Have you checked with some benchmarks? From my experience, it's been disappointing. Big, complex to setup, not convinced about many tradeoffs.
  - jsheard 3y ago
    WebGPU running on the web necessarily has overhead compared to native due to all the extra validation it has to do for security, but there's nothing stopping native implementations from offering unsafe escape hatches as extensions for those who want them.
    - kvark 3y ago
      And wgpu has been doing this for years. Things like descriptor indexing are not exposed to the web but used by Rust (mostly) engines on native. https://wgpu.rs/ https://wgpu.rs/

cactusplant7374 3y ago

Playing Halo CE on Mac M1 with wine is an abject failure. I am assuming it is because Direct3D support is implemented poorly?

Cu3PO42 3y ago
Maybe. If you haven't tried it, you could look into Apple's own Direct3D to Metal translation layer from Game Porting Toolkit. It generally gives better results than DXVK chained with MoltenVK.

steeve 3y ago

I highly advise people to look into the Mach ecosytem, especially mach-sysgpu, which is a complete reimplementation of WebGPU, written (mostly) by Ali Chraghi, who is 17

fifteen1506 3y ago

Microsoft <3 Open Source

delta_p_delta_x 3y ago

> However, it doesn’t require any additional .dlls to be shipped with the application. Many video games already do this with all the proprietary middleware they use (Bink, SpeedTree, PhysX, etc). Most launchers (Steam, GOG, Epic, etc) also require their respective .DLLs. Many games also use D3D11On12. Many shipping games in my list have dxil.dll amongst their installed files. Therefore, an honest question: what's the problem with shipping an additional DLL? The work done here to reverse-engineer and re-implement the code-signing is fantastic—especially the fact that it is bitwise equal to dxil.dll's output. But I am ridiculously lazy and prefer to take the easier way out, and would've just shipped the DLL.

msk-lywenn 3y ago
As you pointed out, it’s really not a problem. However, the article points out at the end the real interesting bit: cross building from any OS/arch.
chris37879 3y ago
This opensource implementation can be included in existing game engines without requiring the programmer to know or understand what dxil.dll even is. The Mach engine, for instance, can use this to create a compiler that compiles zig code into shaders across all its targeted platforms in a way that doesn't require any additional setup or dependencies for the end user, it's just built into Mach's core.
badsectoracula 3y ago
There are more games than AAA games and there are more graphics applications than games. Something that is fine for your average 100GB AAA game may not be for a 30MB game (of which 24MB are DXC) on itch.io or a 3D model viewer/converter/animator/whatever on someone's website. In addition, even for AAA games (but also for other games and software, this adds extra dependencies that may be broken outside your control. At a company i worked several years ago, we spent more effort upgrading Visual Studio because for some middle we only had DLLs (and their libraries) to link against and had to obtain new versions (part of the effort was that the company that wrote said middleware hadn't upgraded their own so we also became QA for their compatibility with the newer VS, though i don't remember if there were any issues in practice). Of course having the code doesn't mean updates will be frictionless, but there will certainly be way less friction and you wont need to wait for someone else. (btw i think Visual Studio has tried to be backwards compatible in recent versions with binary C++ libraries, but i don't think this is something you can rely on in the long term) Also all that are with the assumption that the requirement is that your code remain on the same platform and targets. But at some point you may want to work with another platform (be it as host or target) and not having source code can make that from incredibly difficult to impossible. This may not seem much of an issue for something as platform-specific as DXIL - but actually notice the article mentioning that the binary blobbiness of DXIL made it impossible to precompile shaders on platforms outside specific Windows and Linux architectures for which the DLL was provided by Microsoft.

nightowl_games 3y ago

Using zig itself as a shading language is awesome. Zig is the one true language. It's a build system! It's a shading language! Wow!

unnouinceput 3y ago

Quote: "All that was left was that pesky dxil.dll - what sort of magic might Microsoft be employing in that library to “sign shaders”? How can they prevent unsigned shaders from running on Windows machines that aren’t in developer mode? How are they able to distribute that binary on Linux, too? I won’t comment on any of those questions, but will say that you’ll find dxil.dll is NOT a dependency of mach-dxcompiler in any form. You can compile an HLSL shader on a macOS machine using mach-dxcompiler, without the proprietary dxil.dll blob - and end up with a DXIL bytecode file that is byte-for-byte equal to one which runs it on a standard Windows box. Enjoy!" That above is the real magic. Since he won't comment on the how, I guess he took a swing at poping the hood underneath, and did exactly what Wine developers did 2 decades ago. Any old timers here remembering that scandal? Smart kid to not comment on the how, this way Microsoft won't have any legal leg and since times have changed with all that "Microsoft loves Linux" shit they yell at all corners (not that I believe that for a single second), then it will be swept all under the rug and FOSS wins. For now.

rofrol 3y ago
> So the signing[1] DXIL.dll does is just a modified MD5? [1] https://github.com/hexops/DirectXShaderCompiler/blob/4190bb0c90d374c6b4d0b0f2c7b45b604eda24b6/tools/clang/tools/dxcompiler/MachSiegbertVogtDXCSA.cpp#L178 https://github.com/hexops/DirectXShaderCompiler/blob/4190bb0... https://news.ycombinator.com/item?id=39325654 https://news.ycombinator.com/item?id=39325654
charcircuit 3y ago
>Since he won't comment on the how, I guess he took a swing at poping the hood underneath The way signing is done is already public knowledge that can be found using google as other open source projects have implemented it.

deleted 3y ago

[deleted]

trevortheblack 3y ago

DXIL is pronounced "dixel", like "pixel" but replace the p with a d

chrsig 3y ago

> DXC’s fork of LLVM removed and/or damaged much of the code generation layer and infrastructure [of LLVM]. Given that, supporting DXBC generation in DXC would be a massive task to fix and restore broken LLVM functionality. Due to the large scale of this issue and resource constraints on our team we’re not going to address this issue in [the new] DXC [compiler] ever. > > We may support DXBC generation in Clang in the future (we mentioned that in the original proposal to LLVM). That work is unlikely to begin for a few years as our focus will be on supporting DXIL and SPIR-V generation first. I appreciate this quote[0] from the microsoft camp. Setting clear expectations that something will not be done is a nice bit of fresh air. [0] https://github.com/microsoft/DirectXShaderCompiler/issues/5773#issuecomment-1735794551 https://github.com/microsoft/DirectXShaderCompiler/issues/57...