5 ms·

The case of the curiously slow shader

77 points by raphlinus 5y ago

uyt 5y ago
Thanks for the great list of resources for debugging gpu performance! Android GPU Inspector and http://shader-playground.timjones.io/ http://shader-playground.timjones.io/ all look pretty cool. Do you have any other debugging tips or tooling recommendations for a complete graphics noob? I have only done one webgl project before but that experience was pretty painful. When writing my first glsl shader I basically committed every performance sin possible. Like in your case, the slowness is hard to notice on a beefy desktop, it was only on mobile devices when it becomes apparent. My debugging process was then to comment out lines, rerun, and squint at the fps. There must be a better workflow than this! (iirc, my issue was that I was using "discard" in my fragment shader which is apparently especially bad on powervr architectures https://stackoverflow.com/questions/8509051/is-discard-bad-for-program-performance-in-opengl https://stackoverflow.com/questions/8509051/is-discard-bad-f...)
- raphlinus 5y ago
  Commenting out lines, rerunning, and trying to get meaningful performance counters is pretty much my workflow, and, from what I've observed, not very uncommon. A tool like AGI should give at least somewhat better insight than just FPS though, as that is a crude measure that can conflate a lot of different things. There's a fair amount of academic work on GPU simulators, but a lot of the activity there seemed to be a few years ago. And in any case, I'm not sure how usable these are for tuning real workloads. If you have good hardware performance counters, those can give you almost as good information as you'd get out of a simulator. I think there's tremendous potential to do better tools in this space. I'm not quite sure what the business model is, though.
jjoonathan 5y ago
I love crazy shader stories! My crazy shader story: 8 bit floats. Back in 2007-ish, I was having a hell of a time porting a shader from my desktop to my laptop. For the most part it seemed to work but the output was all one color. That isn't particularly strange for a bug in the shader world, so I went about bisecting it and narrowed the problem down to a critical statement. The input parameters were correct, as verified by color-mapping them, but whenever I performed a division the output would fix to a single color. I could get division to work by feeding it different parameters, though, so what could be the problem? My big break happened when I noticed that a tiny zone around the edges was outputting a different color. The calculation was happening, it was just unexpectedly almost always producing a single value due to extreme rounding. With the problem identified, it was easy enough to work around by using a scaling factor. After paging through Intel documentation I finally found the cause: the integrated graphics card supported floating point all right, but with 8 bit floats. Implicit, mind you, and there was no compiler warning. I asked for a float and it gave me a float, what's the problem? ¯\_( ツ )_/¯ I'm guessing someone tasked the Intel Integrated graphics team with supporting floating point but didn't really give them the transistor or TDP budget for it, so some evil genius noted that they never specified the bit width and cheerfully squared that circle.
- valleyer 5y ago
  Great story. Which Intel integrated GPU, if you remember? And what shader language? It looks like GLSL expects `float` to be a normal float32. https://www.khronos.org/opengl/wiki/Data_Type_(GLSL)#Scalars https://www.khronos.org/opengl/wiki/Data_Type_(GLSL)#Scalars
  - kevingadd 5y ago
    "The specific sizes and formats for integers and floats in GLSL are only for GLSL 1.30 and above. Lower versions of GLSL may not use these exact specifications." Pre-1.3.0 was a truly dark era. Lots of really nasty driver and hardware issues lurking around.
  - jjoonathan 5y ago
    It was a Core 2 Duo white plastic macbook (or were they still iBooks?). I don't remember the exact GPU, but I do remember that it was the very early "wild west" days of GPGPU. Getting GPUs to do things other than draw textured polygons was a strange, crazy, cool idea. It would be years until transcoding and media editing tools managed to make it work. My PPC mac still absolutely smoked the new intel mac at those tasks, even with vector extensions, but of course the transition was underway and there would be no going back. The language was GLSL. I'm glad to see that sometime in the last decade and a half they decided to mandate float32. Pulling out the float8s to win a benchmark these days would be... something!
    - formerly_proven 5y ago
      Funny how this turned around - until very recently low-precision (<32 bit floats) arithmetic was intentionally limited on consumer cards - iirc FP16 rate was limited to 1/100th or something crazy like that. Intel being the true pioneers they are not only made this available at full performance to their customers, but even the default!
- jhgb 5y ago
  > and this is what they came up with Programmers and insufficient specifications, name a more iconic duo?
zozbot234 5y ago
Very impressive work. It's really sad how the lack of open documentation wrt. GPU internals is hindering and impacting this kind of innovative, performance-focused analysis and development. OP had to resort to a crude hack in order to get low-level GPU shader code from the proprietary driver, and then analyze it via a reverse engineered GPU-support stack.

77 points by raphlinus 5y ago

uyt 5y ago

Thanks for the great list of resources for debugging gpu performance! Android GPU Inspector and http://shader-playground.timjones.io/ http://shader-playground.timjones.io/ all look pretty cool. Do you have any other debugging tips or tooling recommendations for a complete graphics noob? I have only done one webgl project before but that experience was pretty painful. When writing my first glsl shader I basically committed every performance sin possible. Like in your case, the slowness is hard to notice on a beefy desktop, it was only on mobile devices when it becomes apparent. My debugging process was then to comment out lines, rerun, and squint at the fps. There must be a better workflow than this! (iirc, my issue was that I was using "discard" in my fragment shader which is apparently especially bad on powervr architectures https://stackoverflow.com/questions/8509051/is-discard-bad-for-program-performance-in-opengl https://stackoverflow.com/questions/8509051/is-discard-bad-f...)

raphlinus 5y ago
Commenting out lines, rerunning, and trying to get meaningful performance counters is pretty much my workflow, and, from what I've observed, not very uncommon. A tool like AGI should give at least somewhat better insight than just FPS though, as that is a crude measure that can conflate a lot of different things. There's a fair amount of academic work on GPU simulators, but a lot of the activity there seemed to be a few years ago. And in any case, I'm not sure how usable these are for tuning real workloads. If you have good hardware performance counters, those can give you almost as good information as you'd get out of a simulator. I think there's tremendous potential to do better tools in this space. I'm not quite sure what the business model is, though.

jjoonathan 5y ago

I love crazy shader stories! My crazy shader story: 8 bit floats. Back in 2007-ish, I was having a hell of a time porting a shader from my desktop to my laptop. For the most part it seemed to work but the output was all one color. That isn't particularly strange for a bug in the shader world, so I went about bisecting it and narrowed the problem down to a critical statement. The input parameters were correct, as verified by color-mapping them, but whenever I performed a division the output would fix to a single color. I could get division to work by feeding it different parameters, though, so what could be the problem? My big break happened when I noticed that a tiny zone around the edges was outputting a different color. The calculation was happening, it was just unexpectedly almost always producing a single value due to extreme rounding. With the problem identified, it was easy enough to work around by using a scaling factor. After paging through Intel documentation I finally found the cause: the integrated graphics card supported floating point all right, but with 8 bit floats. Implicit, mind you, and there was no compiler warning. I asked for a float and it gave me a float, what's the problem? ¯\_( ツ )_/¯ I'm guessing someone tasked the Intel Integrated graphics team with supporting floating point but didn't really give them the transistor or TDP budget for it, so some evil genius noted that they never specified the bit width and cheerfully squared that circle.

valleyer 5y ago
Great story. Which Intel integrated GPU, if you remember? And what shader language? It looks like GLSL expects `float` to be a normal float32. https://www.khronos.org/opengl/wiki/Data_Type_(GLSL)#Scalars https://www.khronos.org/opengl/wiki/Data_Type_(GLSL)#Scalars
- kevingadd 5y ago
  "The specific sizes and formats for integers and floats in GLSL are only for GLSL 1.30 and above. Lower versions of GLSL may not use these exact specifications." Pre-1.3.0 was a truly dark era. Lots of really nasty driver and hardware issues lurking around.
- jjoonathan 5y ago
  It was a Core 2 Duo white plastic macbook (or were they still iBooks?). I don't remember the exact GPU, but I do remember that it was the very early "wild west" days of GPGPU. Getting GPUs to do things other than draw textured polygons was a strange, crazy, cool idea. It would be years until transcoding and media editing tools managed to make it work. My PPC mac still absolutely smoked the new intel mac at those tasks, even with vector extensions, but of course the transition was underway and there would be no going back. The language was GLSL. I'm glad to see that sometime in the last decade and a half they decided to mandate float32. Pulling out the float8s to win a benchmark these days would be... something!
  - formerly_proven 5y ago
    Funny how this turned around - until very recently low-precision (<32 bit floats) arithmetic was intentionally limited on consumer cards - iirc FP16 rate was limited to 1/100th or something crazy like that. Intel being the true pioneers they are not only made this available at full performance to their customers, but even the default!
jhgb 5y ago
> and this is what they came up with Programmers and insufficient specifications, name a more iconic duo?

zozbot234 5y ago

Very impressive work. It's really sad how the lack of open documentation wrt. GPU internals is hindering and impacting this kind of innovative, performance-focused analysis and development. OP had to resort to a crude hack in order to get low-level GPU shader code from the proprietary driver, and then analyze it via a reverse engineered GPU-support stack.