7 ms·

Buggy shader compilers are no surprise to me but it's a serious issue that needs to be addressed by all GPU vendors. Vulkan and SPIR-V may help a little as the

by exDM69 10y ago

Buggy shader compilers are no surprise to me but it's a serious issue that needs to be addressed by all GPU vendors. Vulkan and SPIR-V may help a little as the GLSL compiler frontend is moved out of the "driver" component and the resulting SPIR-V blob can be more easily manipulated. There are some major issues at play: 1) Shipping an OpenGL-based product on multiple vendors' devices (especially on mobile) requires a huge amount of QA effort and per-device bug workarounds. Pretty much everyone makes their games on top of Unity/Unreal/etc because the engine vendors do most of the QA work. 2) Issue #1 is made worse by the fact that many consumer devices do not get GPU driver updates because the phone/tablet OEM end-of-lifes their products too soon and/or the mobile operators and other middlemen don't ship updates as they should. This means that if you ship a GPU-using product, you need to support devices with old, buggy drivers and this is expensive. 3) There's a huge untapped potential in GPUs, there are very few non-graphics apps taking advantage of the processing power but I can't see it changing for the better before GPUs become easier to target and verify correctness of operation. It would be too easy to blame GPU vendors' software engineering practices, but I (as a GPU driver programmer) see that the bigger problem is issue #2 (not that the GPU drivers are faultless). Even if the drivers do get fixed and updated, getting the updates to the hands of the customers is still going to be an issue. The middlemen need to be cut out of the equation, we can't be dependent on the business requirements of OEMs and operators when shipping mission critical software infrastructure. This is a bit of a chicken and egg problem, games are not important enough to OEMs and operators fixing their update delivery mechanisms, but no-one dares to use GPUs for anything more important before this issue gets sorted out.

Athas 10y ago
In my experience, the situation is a little better for AMD and NVIDIA GPUs - likely because they have already seen (relatively) heavy use for OpenCL and CUDA. The drivers are still pretty bad altogether, with some of intermittent faults and weird quirks, but compiler bugs are infrequent. It is however my general impression that the graphics parts of GPU drivers are generally more buggy than the compute parts. Likely because the graphics stack is older and more complicated. Is this a correct assessment?
- azernik 10y ago
  Also because the graphics parts have to implement more logic logic; generally computing users of GPUs want access to arithmetic primitives, not complicated graphics-relevant functions.
- wolfgke 10y ago
  > It is however my general impression that the graphics parts of GPU drivers are generally more buggy than the compute parts. I'm not a graphics driver developer, but I know that game engines for modern games often are actually quite buggy and often the graphics driver stack tries to work around engine bugs. It's also well-known that for AAA titles the graphics driver internally even often replaces the game's shaders completely by hand-optimized ones made by the GPU vendor. I have read from an insider that in the last years NVIdia's graphics drivers often did not validate arguments (for a little bit more performance), while the AMD's did. This was as I heard the reason why one of the last Tomb Raider game had initially problems on AMD GPUs. Also for OpenGL drivers you have to realize that in the past Intel drivers did not really support modern OpenGL versions. So at least for Indie developers the safest way was to target OpenGL 2.1 + extensions. Though OpenGL >= 3.1 should better be initialized using a Core context many developers still use(d) a Compatibility context - this adds a lot of legacy stuff that one has to stay compatible with and thus makes the whole OpenGL implementation of Compatibility contexts very error-prone (in the driver). Luckily we now have Vulkan (though writing your engine against Vulkan is at least at the beginning more complicated (time to first triangle) than targeting OpenGL or DirectX 11.x (not DirectX 12 - DX 12 is a lot more like Vulkan)).
  - ploxiln 10y ago
    In the open-source-everything world, it's a lot more common to push for and eventually make a fix where the bug is, and then tell people to update or patch. (Though it still doesn't always happen.) In the closed-source and hardware worlds, it's a lot more common to implement a work-around wherever you are. (Alternatively, the entity paying money pressures the entity being paid to put in a workaround on their side.) Soon there are a bunch of workarounds in common use which in turn need to be worked-around. This is the world of commercial games and graphics drivers. I used to work on a WiFi access point product, though I only touched the actual driver a few times (other team members focused there), and there was a lot of working around bugs, and working around work-arounds in other WiFi devices for other bugs in WiFi APs, in that world too. Some crazy stuff ...
daveguy 10y ago
With the #2 issue -- it would be nice if the drivers were distributed as a drivers only option rather than a mega-advertising option. It looks like AMD does this now with a 55MB version, but Nvidia still only has the ridiculous "experience" application in place of a simple driver package (300+MB).
- exDM69 10y ago
  I understand your opinion as a consumer but it's not a real problem (minor annoyance at most) compared to millions of consumer devices with outdated drivers (with possible security problems). The "experience" thing is optional afaik.
  - matt4077 10y ago
    Let's meet in the middle and do it like Apple? Also, as a consumer the "experience" seems to matter a lot more to me than ... whatever the other side of this argument is ... the hardship for programmers? And since you're bringing up security: if there are security holes in GPU driver installations, I'd bet eliminating all the crapware would do a lot more for security than any number of updates could ever do.
    - gsnedders 10y ago
      The worst place for security bugs in drivers is mobile, for the normal mobile reason (try getting security updates for any OS component!), and there's no crapware there.
    - exDM69 10y ago
      Yes, some kind of meet in the middle solution would be nice. For example, Microsoft's official updater keeping the drivers up to date. And by up to date I mean the latest from the GPU vendor and not something a bit older but certified by MS. Getting rid of 3rd party installers and updaters should be a priority of Microsoft's anyway. > if there are security holes in GPU driver installations, I'd bet eliminating all the crapware would do a lot more for security than any number of updates could ever do. The "crapware" is just some unprivileged user space processes. The driver has kernel mode components, and security issues there could cause crashes, snooping memory of other processes or even kernel space remote code execution. Not that I'm defending bundled (or optional) crapware, but I don't think that's a security problem.
viewer5 10y ago
What causes #1? Why's OpenGL so troublesome and weird for each device?
- pjmlp 10y ago
  Easy just think how browsers render HTML/CSS/JavaScript, C compilers specific interpretation of ANSI C, UNIX variations from what POSIX leaves to each implementation, and many other implementation vs standards variations. Just like them OpenGL is a text standard describing how a 3D API is supposed to behave. It happens that what those papers state and what each team of developers at every card manufacturer understand is not always the same. Then there is the set of card specific extensions, for every nice feature they want to sell on their graphics card, but yet to be adopted by OpenGL paper standard. There are of course certification tests available, but they are costly and don't cover 100% of the API usage anyway. So programming OpenGL, happens to be like trying to write web applications, while trying to make the code portable and bug free (with workarounds) across all graphic cards out there.
  - p0nce 10y ago
    > So programming OpenGL, happens to be like trying to write web applications, while trying to make the code portable and bug free (with workarounds) across all graphic cards out there. And it's much harder to test because you have to find the outdated graphic card, with the outdated driver, and have a machine where you can plug the offending card.
- glaze 10y ago
  One reason is that shaders are stored as text and require each driver vendor to write parsers for them and not everyone implements them in the same way.
  - exDM69 10y ago
    Further, the drivers have to be more permissive than necessary. It's not an option for a GPU vendor to ship a driver update that would break Doom/Quake/Dota/etc. Ideally games would ship only shaders that are strictly valid GLSL but that's not quite the case. If you're developing GL, do everyone a favor and start using the official GLSL validator in your build scripts.
discreditable 10y ago
Compounding issues #1 and #2 you mention is that when the HPs, Lenovos, et al do ship updates they're usually a few months behind current at best. I've seen this for Intel, AMD, and NVIDIA. What's worse is if you run the "generic" Intel GPU driver installer, if it detects an OEM's driver on the system it will refuse to install.
DannyBee 10y ago
You missed one of the most major issues: Most of the vendors know they have serious issues here, but hide them. Literally. You never notice because they don't use open source compilers (even for things like CUDA, let alone shader compilers), so it's not obviously more than "just a bug" until it happens to you continuously. Instead, they pretty much never have to fix the bug until someone notices, and then they hack it some more and move on, instead of fixing underlying issues in their structurization/etc passes. Most vendors i've talked to can't even tell me what control flow breaks their compiler (again, doesn't matter if we are talking shaders, cuda, you name it. it's all broke), they know plenty does, but are fairly ¯\_(ツ)_/¯ about doing more than working around whatever bug they get given. Meanwhile, over in open source clang/llvm world, we can basically fuzz test CFG's, etc for CUDA. The death of some of these compilers can't come fast enough.
- mrb 10y ago
  "You never notice because they don't use open source compilers (even for things like CUDA, let alone shader compilers)" Actually AMD recently completely open sourced their entire GPGPU compute stack. See https://github.com/RadeonOpenCompute https://github.com/RadeonOpenCompute Compiler, assembler (LLVM based), linker, driver, etc. Everything is open to my knowledge.
  - DannyBee 10y ago
    Actually, we wrote the CUDA support in clang :) and yes, AMD has done this because they have nothing to lose anymore :)
- deleted 10y ago
  [deleted]
- akiselev 10y ago
  Even worse some of the mobile GPU vendors, coming from the hardware/silicon industry, sometimes provide fixes to their bugs only when a customer asks for it, leaving the fix out of the master repository for fear of providing evidence to potential litigators and IP to competitors.
benjaminPKP 10y ago
Just took an applied parallel programming class at a top 5 engineering school. The untapped potential for GPUs really is crazy. If GPUs could be used for common tasks able to be parallelized, then less work for the CPU and way quicker computations. I mean we made a convolutional neural network for handwriting recognition which took ~30 minutes to run on a CPU, ran it parallel in 400 milliseconds on GPUs. Once CPUs start leveling out due to the transistor-heat issue, I think GPUs might start taking on a bigger role, they simply need more support. So many people are all about the buzzwords in the tech world that no one wants to turn the attention something like parallelization, but hopefully someday soon
- deleted 10y ago
  [deleted]
- p1esk 10y ago
  Going from 30 min on CPU to 0.4 sec on GPU can mean only one thing - you had a really crappy CPU code. A well written C code should run about 20-50 times slower than a well written CUDA code on the latest GPUs, and that's for a single CPU core. This was true with my hand-written implementations, as well as when using libraries (Numpy vs CuDNN).
  - brigade 10y ago
    Going further, power for power on the current generation, for SGEMM, i'd expect GPUs to beat CPUs by maybe 5x (6700k has a theoretical 512 gflops at 91W, while xeons can come a bit shy of 1.5 tflops at 145w) Which is still pretty major, but nowhere near the wild claims that used to be common in academic papers. It's outdated, but [1] is still pretty good reading. [1] http://sbel.wisc.edu/Courses/ME964/Literature/LeeDebunkGPU2010.pdf http://sbel.wisc.edu/Courses/ME964/Literature/LeeDebunkGPU20...
  - benjaminPKP 10y ago
    We implemented the LeNet-5 convolutional neural network, and yes when run sequentially it really does take that long. The sequential code that took that long was also written by professors who worked at NVIDIA, it was written as well as it could be sequentially. The project was a competition to see how fast we could have it go in parallel so we used a ton of parallelization techniques and tricks like streaming to make it go at .4 sec. Think again before you call someones code "crappy"

by exDM69 10y ago

Athas 10y ago

In my experience, the situation is a little better for AMD and NVIDIA GPUs - likely because they have already seen (relatively) heavy use for OpenCL and CUDA. The drivers are still pretty bad altogether, with some of intermittent faults and weird quirks, but compiler bugs are infrequent. It is however my general impression that the graphics parts of GPU drivers are generally more buggy than the compute parts. Likely because the graphics stack is older and more complicated. Is this a correct assessment?

azernik 10y ago
Also because the graphics parts have to implement more logic logic; generally computing users of GPUs want access to arithmetic primitives, not complicated graphics-relevant functions.
wolfgke 10y ago
> It is however my general impression that the graphics parts of GPU drivers are generally more buggy than the compute parts. I'm not a graphics driver developer, but I know that game engines for modern games often are actually quite buggy and often the graphics driver stack tries to work around engine bugs. It's also well-known that for AAA titles the graphics driver internally even often replaces the game's shaders completely by hand-optimized ones made by the GPU vendor. I have read from an insider that in the last years NVIdia's graphics drivers often did not validate arguments (for a little bit more performance), while the AMD's did. This was as I heard the reason why one of the last Tomb Raider game had initially problems on AMD GPUs. Also for OpenGL drivers you have to realize that in the past Intel drivers did not really support modern OpenGL versions. So at least for Indie developers the safest way was to target OpenGL 2.1 + extensions. Though OpenGL >= 3.1 should better be initialized using a Core context many developers still use(d) a Compatibility context - this adds a lot of legacy stuff that one has to stay compatible with and thus makes the whole OpenGL implementation of Compatibility contexts very error-prone (in the driver). Luckily we now have Vulkan (though writing your engine against Vulkan is at least at the beginning more complicated (time to first triangle) than targeting OpenGL or DirectX 11.x (not DirectX 12 - DX 12 is a lot more like Vulkan)).
- ploxiln 10y ago
  In the open-source-everything world, it's a lot more common to push for and eventually make a fix where the bug is, and then tell people to update or patch. (Though it still doesn't always happen.) In the closed-source and hardware worlds, it's a lot more common to implement a work-around wherever you are. (Alternatively, the entity paying money pressures the entity being paid to put in a workaround on their side.) Soon there are a bunch of workarounds in common use which in turn need to be worked-around. This is the world of commercial games and graphics drivers. I used to work on a WiFi access point product, though I only touched the actual driver a few times (other team members focused there), and there was a lot of working around bugs, and working around work-arounds in other WiFi devices for other bugs in WiFi APs, in that world too. Some crazy stuff ...

daveguy 10y ago

With the #2 issue -- it would be nice if the drivers were distributed as a drivers only option rather than a mega-advertising option. It looks like AMD does this now with a 55MB version, but Nvidia still only has the ridiculous "experience" application in place of a simple driver package (300+MB).

exDM69 10y ago
I understand your opinion as a consumer but it's not a real problem (minor annoyance at most) compared to millions of consumer devices with outdated drivers (with possible security problems). The "experience" thing is optional afaik.
- matt4077 10y ago
  Let's meet in the middle and do it like Apple? Also, as a consumer the "experience" seems to matter a lot more to me than ... whatever the other side of this argument is ... the hardship for programmers? And since you're bringing up security: if there are security holes in GPU driver installations, I'd bet eliminating all the crapware would do a lot more for security than any number of updates could ever do.
  - gsnedders 10y ago
    The worst place for security bugs in drivers is mobile, for the normal mobile reason (try getting security updates for any OS component!), and there's no crapware there.
  - exDM69 10y ago
    Yes, some kind of meet in the middle solution would be nice. For example, Microsoft's official updater keeping the drivers up to date. And by up to date I mean the latest from the GPU vendor and not something a bit older but certified by MS. Getting rid of 3rd party installers and updaters should be a priority of Microsoft's anyway. > if there are security holes in GPU driver installations, I'd bet eliminating all the crapware would do a lot more for security than any number of updates could ever do. The "crapware" is just some unprivileged user space processes. The driver has kernel mode components, and security issues there could cause crashes, snooping memory of other processes or even kernel space remote code execution. Not that I'm defending bundled (or optional) crapware, but I don't think that's a security problem.

viewer5 10y ago

What causes #1? Why's OpenGL so troublesome and weird for each device?

pjmlp 10y ago
Easy just think how browsers render HTML/CSS/JavaScript, C compilers specific interpretation of ANSI C, UNIX variations from what POSIX leaves to each implementation, and many other implementation vs standards variations. Just like them OpenGL is a text standard describing how a 3D API is supposed to behave. It happens that what those papers state and what each team of developers at every card manufacturer understand is not always the same. Then there is the set of card specific extensions, for every nice feature they want to sell on their graphics card, but yet to be adopted by OpenGL paper standard. There are of course certification tests available, but they are costly and don't cover 100% of the API usage anyway. So programming OpenGL, happens to be like trying to write web applications, while trying to make the code portable and bug free (with workarounds) across all graphic cards out there.
- p0nce 10y ago
  > So programming OpenGL, happens to be like trying to write web applications, while trying to make the code portable and bug free (with workarounds) across all graphic cards out there. And it's much harder to test because you have to find the outdated graphic card, with the outdated driver, and have a machine where you can plug the offending card.
glaze 10y ago
One reason is that shaders are stored as text and require each driver vendor to write parsers for them and not everyone implements them in the same way.
- exDM69 10y ago
  Further, the drivers have to be more permissive than necessary. It's not an option for a GPU vendor to ship a driver update that would break Doom/Quake/Dota/etc. Ideally games would ship only shaders that are strictly valid GLSL but that's not quite the case. If you're developing GL, do everyone a favor and start using the official GLSL validator in your build scripts.

discreditable 10y ago

Compounding issues #1 and #2 you mention is that when the HPs, Lenovos, et al do ship updates they're usually a few months behind current at best. I've seen this for Intel, AMD, and NVIDIA. What's worse is if you run the "generic" Intel GPU driver installer, if it detects an OEM's driver on the system it will refuse to install.

DannyBee 10y ago

You missed one of the most major issues: Most of the vendors know they have serious issues here, but hide them. Literally. You never notice because they don't use open source compilers (even for things like CUDA, let alone shader compilers), so it's not obviously more than "just a bug" until it happens to you continuously. Instead, they pretty much never have to fix the bug until someone notices, and then they hack it some more and move on, instead of fixing underlying issues in their structurization/etc passes. Most vendors i've talked to can't even tell me what control flow breaks their compiler (again, doesn't matter if we are talking shaders, cuda, you name it. it's all broke), they know plenty does, but are fairly ¯\_(ツ)_/¯ about doing more than working around whatever bug they get given. Meanwhile, over in open source clang/llvm world, we can basically fuzz test CFG's, etc for CUDA. The death of some of these compilers can't come fast enough.

mrb 10y ago
"You never notice because they don't use open source compilers (even for things like CUDA, let alone shader compilers)" Actually AMD recently completely open sourced their entire GPGPU compute stack. See https://github.com/RadeonOpenCompute https://github.com/RadeonOpenCompute Compiler, assembler (LLVM based), linker, driver, etc. Everything is open to my knowledge.
- DannyBee 10y ago
  Actually, we wrote the CUDA support in clang :) and yes, AMD has done this because they have nothing to lose anymore :)
deleted 10y ago
[deleted]
akiselev 10y ago
Even worse some of the mobile GPU vendors, coming from the hardware/silicon industry, sometimes provide fixes to their bugs only when a customer asks for it, leaving the fix out of the master repository for fear of providing evidence to potential litigators and IP to competitors.

benjaminPKP 10y ago

Just took an applied parallel programming class at a top 5 engineering school. The untapped potential for GPUs really is crazy. If GPUs could be used for common tasks able to be parallelized, then less work for the CPU and way quicker computations. I mean we made a convolutional neural network for handwriting recognition which took ~30 minutes to run on a CPU, ran it parallel in 400 milliseconds on GPUs. Once CPUs start leveling out due to the transistor-heat issue, I think GPUs might start taking on a bigger role, they simply need more support. So many people are all about the buzzwords in the tech world that no one wants to turn the attention something like parallelization, but hopefully someday soon

deleted 10y ago
[deleted]
p1esk 10y ago
Going from 30 min on CPU to 0.4 sec on GPU can mean only one thing - you had a really crappy CPU code. A well written C code should run about 20-50 times slower than a well written CUDA code on the latest GPUs, and that's for a single CPU core. This was true with my hand-written implementations, as well as when using libraries (Numpy vs CuDNN).
- brigade 10y ago
  Going further, power for power on the current generation, for SGEMM, i'd expect GPUs to beat CPUs by maybe 5x (6700k has a theoretical 512 gflops at 91W, while xeons can come a bit shy of 1.5 tflops at 145w) Which is still pretty major, but nowhere near the wild claims that used to be common in academic papers. It's outdated, but [1] is still pretty good reading. [1] http://sbel.wisc.edu/Courses/ME964/Literature/LeeDebunkGPU2010.pdf http://sbel.wisc.edu/Courses/ME964/Literature/LeeDebunkGPU20...
- benjaminPKP 10y ago
  We implemented the LeNet-5 convolutional neural network, and yes when run sequentially it really does take that long. The sequential code that took that long was also written by professors who worked at NVIDIA, it was written as well as it could be sequentially. The project was a competition to see how fast we could have it go in parallel so we used a ton of parallelization techniques and tricks like streaming to make it go at .4 sec. Think again before you call someones code "crappy"