5 ms·

I'm not sure why bytecode is supposed to be significantly better than storing the shaders as strings. Sure, you get rid of a complex parser and can more easily

by Athas 10y ago

I'm not sure why bytecode is supposed to be significantly better than storing the shaders as strings. Sure, you get rid of a complex parser and can more easily use it as a compiler target, but the core problem remains: the compiler can still not reason across the device boundary. The CPU compiler does not understand what happens on the GPU, and the GPU compiler has no idea what the CPU is going to do do it. Although I'm not a big fan of CUDA, it does have an advantage in that it combines both worlds in a single language. This does permit optimisations that cross devices boundaries, but I have no idea whether the CUDA compiler does any of this. Apologies for tooting my own horn, buI have been working on a language that can optimise on both sides of the CPU-GPU-boundary: http://futhark-lang.org/ http://futhark-lang.org/ (Although it is more high-level and less graphics-oriented than what I think the author is looking for.)

_pmf_ 10y ago
I think the article's author does not necessarily bemoan the lack of a comprehensive integration, but the complete lack of any non-textual interface definition.
corysama 10y ago
The issue is that every driver implementation ever ships with a slightly different implementation of that complex parser. That means (every mobile device)X(every OS update) and (every PC ad-in card)X(every driver update) all have the potential to misinterpret/reject your shaders in slightly different ways. Case in point: I recently learned that "#line 0" is a spec violation. A single OS revision of a single Android device refused to compile it. This is after shipping that line in every shader in multiple games for multiple years to millions of happy customers. Apparently, Unreal Engine's boot process involves compiling a series of test shaders to determine which common compiler bugs the game needs to work around for this particular run.
- sillysaurus3 10y ago
  The issue is that every driver implementation ever ships with a slightly different implementation of that complex parser. That means (every mobile device)X(every OS update) and (every PC ad-in card)X(every driver update) all have the potential to misinterpret/reject your shaders in slightly different ways. This doesn't have anything to do with shaders as strings. It's been true since the beginning of shader programming, whether bytecode shaders or otherwise.
  - ufo 10y ago
    His point is that shaders-as-strings have a larger surface area for misinterpretation than shaders-as-bytecode.
- Athas 10y ago
  Couldn't a bytecode format have the same problem? Probably not with respect to parsing, but the semantics could still differ.
  - corysama 10y ago
    In theory, yes. Neither system represents perfection. But, what's more important in practice is the likelihood of problems. Parsing bytecode is trivial. Thus problems with drivers inconsistently rejecting D3D shader bytecode has been relatively rare. Parsing GLSL is quite hard. Thus, problems with drivers inconsistently rejecting GLSL text has been relatively common. OpenGL's text requirement makes great sense in theory, but encouraged lots of problems in practice.
    - fixermark 10y ago
      The software engineers who write drivers for graphics hardware have, unfortunately, an ill-earned reputation for being unable to code their way out of wet paper bags. The fact that the GLSL standard forced every hardware vendor to vend a fully-functional compiler for a C-like language (even a simplified C-like language) is no small part of the source of that reputation.
      - corysama 10y ago
        You know what's great fun? Getting errors that only apply to desktop GLSL when trying to compile valid GLES GLSL. Can you guess how that particular vendor shortcutted implementing their ES compiler for that Android OS revision? Can you guess how likely it is that those Android customers will ever get a driver update? ;P
- overgard 10y ago
  > The issue is that every driver implementation ever ships with a slightly different implementation of that complex parser. That means (every mobile device)X(every OS update) and (every PC ad-in card)X(every driver update) all have the potential to misinterpret/reject your shaders in slightly different ways. I can confirm this happens a LOT. (At least with ESSL) In it's defense, it usually is a mistake I made, but some implementations are definitely more lenient than others.
- p0nce 10y ago
  Can empathize with that. The problem with OpenGL is not really the API but the impossibility to really have something correct. There is a sheer difficulty of finding combinations of GPU x drivers x OS to test OpenGL. It's quite common that the drivers lie or reject something valid.
- deleted 10y ago
  [deleted]
marmaduke 10y ago
Furthark looks really nice, but what does the future look like for it? (I'm currently working on a Python hpc codebase with pyopencl and would be interested in reducing some of the manual labor required for cpu-gpu coordination)
- Athas 10y ago
  Well, I still have a year left on my PhD, and I hope (with some reason) to get funding to continue after that. But I must emphasize that the language is still a little young and raw.
  - marmaduke 10y ago
    I just read the paper and it looks better than most of the alternatives for GPU programming.
dibanez 10y ago
I agree, CUDA took a step in the right direction by allowing annotated C++ source to be compiled for the device. As an HPC developer, I'm using the Kokkos C++ library https://github.com/kokkos/kokkos https://github.com/kokkos/kokkos to switch between CUDA, OpenMP, or Serial modes from a single source code.
scott_s 10y ago
I was poking around on your site, and I took a look at your publications list. I found this paper: http://futhark-lang.org/publications/icfp16.pdf http://futhark-lang.org/publications/icfp16.pdf But it has no authors! I assume it was a double-blind submission to ICFP, which required removing the author names. When posting it publicly, though, I think you should put the author names back on. You don't want your work floating around the internet without your names attached.
robbies 10y ago
Besides the good reasons brought up by other posters, game devs tend to be very secretive of their shader code, so they like the bytecode as an obfuscation layer.

by Athas 10y ago

_pmf_ 10y ago

I think the article's author does not necessarily bemoan the lack of a comprehensive integration, but the complete lack of any non-textual interface definition.

corysama 10y ago

The issue is that every driver implementation ever ships with a slightly different implementation of that complex parser. That means (every mobile device)X(every OS update) and (every PC ad-in card)X(every driver update) all have the potential to misinterpret/reject your shaders in slightly different ways. Case in point: I recently learned that "#line 0" is a spec violation. A single OS revision of a single Android device refused to compile it. This is after shipping that line in every shader in multiple games for multiple years to millions of happy customers. Apparently, Unreal Engine's boot process involves compiling a series of test shaders to determine which common compiler bugs the game needs to work around for this particular run.

sillysaurus3 10y ago
The issue is that every driver implementation ever ships with a slightly different implementation of that complex parser. That means (every mobile device)X(every OS update) and (every PC ad-in card)X(every driver update) all have the potential to misinterpret/reject your shaders in slightly different ways. This doesn't have anything to do with shaders as strings. It's been true since the beginning of shader programming, whether bytecode shaders or otherwise.
- ufo 10y ago
  His point is that shaders-as-strings have a larger surface area for misinterpretation than shaders-as-bytecode.
Athas 10y ago
Couldn't a bytecode format have the same problem? Probably not with respect to parsing, but the semantics could still differ.
- corysama 10y ago
  In theory, yes. Neither system represents perfection. But, what's more important in practice is the likelihood of problems. Parsing bytecode is trivial. Thus problems with drivers inconsistently rejecting D3D shader bytecode has been relatively rare. Parsing GLSL is quite hard. Thus, problems with drivers inconsistently rejecting GLSL text has been relatively common. OpenGL's text requirement makes great sense in theory, but encouraged lots of problems in practice.
  - fixermark 10y ago
    The software engineers who write drivers for graphics hardware have, unfortunately, an ill-earned reputation for being unable to code their way out of wet paper bags. The fact that the GLSL standard forced every hardware vendor to vend a fully-functional compiler for a C-like language (even a simplified C-like language) is no small part of the source of that reputation.
    - corysama 10y ago
      You know what's great fun? Getting errors that only apply to desktop GLSL when trying to compile valid GLES GLSL. Can you guess how that particular vendor shortcutted implementing their ES compiler for that Android OS revision? Can you guess how likely it is that those Android customers will ever get a driver update? ;P
overgard 10y ago
> The issue is that every driver implementation ever ships with a slightly different implementation of that complex parser. That means (every mobile device)X(every OS update) and (every PC ad-in card)X(every driver update) all have the potential to misinterpret/reject your shaders in slightly different ways. I can confirm this happens a LOT. (At least with ESSL) In it's defense, it usually is a mistake I made, but some implementations are definitely more lenient than others.
p0nce 10y ago
Can empathize with that. The problem with OpenGL is not really the API but the impossibility to really have something correct. There is a sheer difficulty of finding combinations of GPU x drivers x OS to test OpenGL. It's quite common that the drivers lie or reject something valid.
deleted 10y ago
[deleted]

marmaduke 10y ago

Furthark looks really nice, but what does the future look like for it? (I'm currently working on a Python hpc codebase with pyopencl and would be interested in reducing some of the manual labor required for cpu-gpu coordination)

Athas 10y ago
Well, I still have a year left on my PhD, and I hope (with some reason) to get funding to continue after that. But I must emphasize that the language is still a little young and raw.
- marmaduke 10y ago
  I just read the paper and it looks better than most of the alternatives for GPU programming.

dibanez 10y ago

I agree, CUDA took a step in the right direction by allowing annotated C++ source to be compiled for the device. As an HPC developer, I'm using the Kokkos C++ library https://github.com/kokkos/kokkos https://github.com/kokkos/kokkos to switch between CUDA, OpenMP, or Serial modes from a single source code.

scott_s 10y ago

I was poking around on your site, and I took a look at your publications list. I found this paper: http://futhark-lang.org/publications/icfp16.pdf http://futhark-lang.org/publications/icfp16.pdf But it has no authors! I assume it was a double-blind submission to ICFP, which required removing the author names. When posting it publicly, though, I think you should put the author names back on. You don't want your work floating around the internet without your names attached.

robbies 10y ago

Besides the good reasons brought up by other posters, game devs tend to be very secretive of their shader code, so they like the bytecode as an obfuscation layer.