3 ms·

Statistically, in my experience, plain old memory corruption bugs within your code or within the GPU itself are a more likely explanation than issues ECC would

Statistically, in my experience, plain old memory corruption bugs within your code or within the GPU itself are a more likely explanation than issues ECC would fix. Games have pretty terrible quality because games can remain playable in the face of bugs. GPUs have similar issues because numerical accuracy is an after thought but numerical accuracy issues can cause issues in code that assume stability of results. Things have gotten better particular on the GPU vendor side of things, but still, memory safety issues and UB issues in c/C++ code far dominate as the root cause any issue you’re likely to see vs memory bit flips (unless people are running very overclocked and unstable machines).

kimixa 3y ago
I'm very much aware of issues mistakes around memory can cause, and we use many tools (including different languages) to avoid them or verify code. My point is even taking that into account there's a long tail of unexplained issues. And UB shouldn't be a scary thing 99% of the time, as you won't be hitting it anyway (or its actually the result of another real bug, like not handling overflows etc.), though as at some level you start trying on platform specific behavior and start /defining/ them. And again, there are tools and options around that, or highlighting areas where you might still be relying on them. They just might be toolchain and/or platform specific. It's never been the scary monster some online programming language fanatics seem to make it sound like, but is something to be aware of and managed. And > unless people are running very overclocked and unstable machines Yes. Have you seen the Gamer Hardware Enthusiast community?
- worthless-trash 3y ago
  > And UB shouldn't be a scary thing 99% of the time, as you won't be hitting it anyway (or its actually the result of another real bug, This is exactly how exploits are made. Accessing UB usually is the result of "another bug" because the UB isnt a bug itself.
- Sweepi 3y ago
  >> unless people are running very overclocked and unstable machines > Yes. Have you seen the Gamer Hardware Enthusiast community? Having seen it for the last 2 decades: A lot of them are interested in OC, but nobody is doing it. Maybe 5%, most likely <2%. Also, most modern Motherboards do a "RAM-Training" session before boot. On unstable machines, this will result in a "test failed" once in a while, with the motherboard showing a "Boot failed, returned every clock setting to default" message, which most likely is not being read by the user, who is either not at the PC the moment it happens or in "why is this taking so long? skip, skip, skip"- mode. What does happen is that Motherboards have OC-like settings out-of-the-box/per default, which were stable most of the time and de facto-accepted by Intel, however is now hitting issues and diminishing returns: https://www.radgametools.com/oodleintel.htm https://www.radgametools.com/oodleintel.htm https://www.igorslab.de/en/intel-spielt-mit-dem-namen-und-den-daten-das-intel-baseline-profile-wird-zu-intel-default-settings/ https://www.igorslab.de/en/intel-spielt-mit-dem-namen-und-de...
  - gpderetta 3y ago
    With XMP many motherboards effectively overclock ram out of the box.
  - fourfour3 3y ago
    What I see a lot of too is poor quality hardware causing crashes - eg poor quality power supplies, actually faulty memory, memory that is marginal at the speeds that it's sold at (eg fine at the standard JEDEC speeds, but fails at the XMP speeds), but also poor cooling - eg not enough airflow in the chassis paired with a modern GPU. I had a PC that was having inexplicable game crashes, but only in stressful games - swapping the power supply from an existing ~10 year old one (which was still rated as enough capacity for my hardware) to a brand new ATX 3.0 unit resolved it.
- jorvi 3y ago
  > Yes. Have you seen the Gamer Hardware Enthusiast community? I'd venture 95% of people run stock, 4% do undervolting and 1% do overclocking. Which isn't that strange because GPU vendors have become relatively adept at pushing the silicon near its max. Its why you can run an aggressive undervolt with a few percentage point lower clocks and be rewarded with a 20-30% wattage drop with commensurate drop temperatures and fan noise, all whilst losing perhaps <5% performance. But the GPU vendors want to put big number on box so they'll have users suffer loud fans :-)
  - kimixa 3y ago
    > I'd venture 95% of people run stock, 4% do undervolting and 1% do overclocking. As others are pointing out, there's an ongoing issue where it seems some products push it beyond the point of stability by default. I think I'm more pointing out the trend that even "informed" purchasers just look at the benchmark charts. There's no other information required. Perhaps things like the mentioned instability making the news will change things going forward? But not holding my breath. If there's still advantages to pushing it beyond the limit, then manufacturers will do so.
- vlovich123 3y ago
  > And UB shouldn't be a scary thing 99% of the time, as you won't be hitting it anyway (or its actually the result of another real bug, like not handling overflows etc.), though as at some level you start trying on platform specific behavior and start /defining/ them I’m talking about language-level UB - that’s not anything you can rely on on any toolchain / platform. And UB will also manifest as random unexplainable crashes in random spots just like memory corruption will. As for tooling, unless you’re using something 100% memory safe with absolutely no call out to unsafe code (which definitely isn’t a game/GPU scenario), you’re going to have this risk & it almost doesn’t matter how much you test or use memory checkers because the long tail of issues are going to be the result of a really difficult to reproduce sequence of events. Additionally, if you have any multi-threaded code, all your testing goes out the window because I’ve seen many concurrency bugs that have hidden in plain sight in very public spots until someone figured out a repro (e.g. https://reviews.llvm.org/D114119 https://reviews.llvm.org/D114119, https://probablydance.com/2022/09/17/finding-the-second-bug-in-glibcs-condition-variable/ https://probablydance.com/2022/09/17/finding-the-second-bug-...). Race conditions are notoriously difficult to reproduce in controlled environments. Oh and when I said you’re safe if you’re in a 100% memory safe language? I lied. There are compiler bugs that could be misgenerating your code + you’re likely running unsafe constructs somewhere in your code that has access to your memory space (whether the OS kernel, whatever is making syscalls to the kernel on your behalf, something that eventually calls the C library, something that is needed for performance, etc etc etc) not to mention bugs in the compiler/JIT that either allow unsafe/unsound constructs or just straight up miscompile correct code. And finally, there are sources of HW issues unrelated to memory bitflips (your CPU is super complex and has bugs too you know as does your memory controller). > research has shown that the majority of one-off soft errors in DRAM chips occur as a result of background radiation, chiefly neutrons from cosmic ray secondaries, > Recent studies[5] show that single-event upsets due to cosmic radiation have been dropping dramatically with process geometry and previous concerns over increasing bit cell error rates are unfounded. So aside from the fact that DRAM susceptibility to cosmic rays has been decreasing (I’m not convinced with the Wikipedia explanation - an alternate explanation can be that the percentage of critically important data in DRAM has shrunk as a percentage as the overall capacity has increased), you’re argument would be that random cosmic radiation is going to randomly hit the DRAM cell containing your code / critical data. On the order of things that are likely, that’s the last thing. Oh and all this relies on you correctly grouping related crashes correctly and I’ve generally seen that to be a significant challenge on any project I’ve participated on (e.g. Oculus for the longest time would group unrelated crash reports and not group the same ones correctly although I helped the team try to make progress on that). Again, it’s not impossible that it’s a legit HW corruption issue. However, all the engineers I know frequently also blamed cosmic rays but at the end it’s all just shorthand for “not worth wasting time trying to track down because it’s in the tail of issues you’ll never get to“.
asveikau 3y ago
Also statistically and in my experience, a low probability event multiplied by a large number of units means millions of people hit hardware failures. It's not at all surprising that someone working on graphics drivers ends up seeing a lot of it.
- vlovich123 3y ago
  Yes, and the issues that ECC can correct for are a small fraction of that overall HW failure rate. And SW issues will always still dominate.
  - asveikau 3y ago
    There are certain niches where the hardware issues will actually dominate the queue of issues and support requests. One I can relate to is if you have something with low code churn and extremely logically simple that does many gigabytes of I/O per machine over a short period. The maintainer of that code is going to see tons of bad disks. That was me once.

kimixa 3y ago

I'm very much aware of issues mistakes around memory can cause, and we use many tools (including different languages) to avoid them or verify code. My point is even taking that into account there's a long tail of unexplained issues. And UB shouldn't be a scary thing 99% of the time, as you won't be hitting it anyway (or its actually the result of another real bug, like not handling overflows etc.), though as at some level you start trying on platform specific behavior and start /defining/ them. And again, there are tools and options around that, or highlighting areas where you might still be relying on them. They just might be toolchain and/or platform specific. It's never been the scary monster some online programming language fanatics seem to make it sound like, but is something to be aware of and managed. And > unless people are running very overclocked and unstable machines Yes. Have you seen the Gamer Hardware Enthusiast community?

worthless-trash 3y ago
> And UB shouldn't be a scary thing 99% of the time, as you won't be hitting it anyway (or its actually the result of another real bug, This is exactly how exploits are made. Accessing UB usually is the result of "another bug" because the UB isnt a bug itself.
Sweepi 3y ago
>> unless people are running very overclocked and unstable machines > Yes. Have you seen the Gamer Hardware Enthusiast community? Having seen it for the last 2 decades: A lot of them are interested in OC, but nobody is doing it. Maybe 5%, most likely <2%. Also, most modern Motherboards do a "RAM-Training" session before boot. On unstable machines, this will result in a "test failed" once in a while, with the motherboard showing a "Boot failed, returned every clock setting to default" message, which most likely is not being read by the user, who is either not at the PC the moment it happens or in "why is this taking so long? skip, skip, skip"- mode. What does happen is that Motherboards have OC-like settings out-of-the-box/per default, which were stable most of the time and de facto-accepted by Intel, however is now hitting issues and diminishing returns: https://www.radgametools.com/oodleintel.htm https://www.radgametools.com/oodleintel.htm https://www.igorslab.de/en/intel-spielt-mit-dem-namen-und-den-daten-das-intel-baseline-profile-wird-zu-intel-default-settings/ https://www.igorslab.de/en/intel-spielt-mit-dem-namen-und-de...
- gpderetta 3y ago
  With XMP many motherboards effectively overclock ram out of the box.
- fourfour3 3y ago
  What I see a lot of too is poor quality hardware causing crashes - eg poor quality power supplies, actually faulty memory, memory that is marginal at the speeds that it's sold at (eg fine at the standard JEDEC speeds, but fails at the XMP speeds), but also poor cooling - eg not enough airflow in the chassis paired with a modern GPU. I had a PC that was having inexplicable game crashes, but only in stressful games - swapping the power supply from an existing ~10 year old one (which was still rated as enough capacity for my hardware) to a brand new ATX 3.0 unit resolved it.
jorvi 3y ago
> Yes. Have you seen the Gamer Hardware Enthusiast community? I'd venture 95% of people run stock, 4% do undervolting and 1% do overclocking. Which isn't that strange because GPU vendors have become relatively adept at pushing the silicon near its max. Its why you can run an aggressive undervolt with a few percentage point lower clocks and be rewarded with a 20-30% wattage drop with commensurate drop temperatures and fan noise, all whilst losing perhaps <5% performance. But the GPU vendors want to put big number on box so they'll have users suffer loud fans :-)
- kimixa 3y ago
  > I'd venture 95% of people run stock, 4% do undervolting and 1% do overclocking. As others are pointing out, there's an ongoing issue where it seems some products push it beyond the point of stability by default. I think I'm more pointing out the trend that even "informed" purchasers just look at the benchmark charts. There's no other information required. Perhaps things like the mentioned instability making the news will change things going forward? But not holding my breath. If there's still advantages to pushing it beyond the limit, then manufacturers will do so.
vlovich123 3y ago
> And UB shouldn't be a scary thing 99% of the time, as you won't be hitting it anyway (or its actually the result of another real bug, like not handling overflows etc.), though as at some level you start trying on platform specific behavior and start /defining/ them I’m talking about language-level UB - that’s not anything you can rely on on any toolchain / platform. And UB will also manifest as random unexplainable crashes in random spots just like memory corruption will. As for tooling, unless you’re using something 100% memory safe with absolutely no call out to unsafe code (which definitely isn’t a game/GPU scenario), you’re going to have this risk & it almost doesn’t matter how much you test or use memory checkers because the long tail of issues are going to be the result of a really difficult to reproduce sequence of events. Additionally, if you have any multi-threaded code, all your testing goes out the window because I’ve seen many concurrency bugs that have hidden in plain sight in very public spots until someone figured out a repro (e.g. https://reviews.llvm.org/D114119 https://reviews.llvm.org/D114119, https://probablydance.com/2022/09/17/finding-the-second-bug-in-glibcs-condition-variable/ https://probablydance.com/2022/09/17/finding-the-second-bug-...). Race conditions are notoriously difficult to reproduce in controlled environments. Oh and when I said you’re safe if you’re in a 100% memory safe language? I lied. There are compiler bugs that could be misgenerating your code + you’re likely running unsafe constructs somewhere in your code that has access to your memory space (whether the OS kernel, whatever is making syscalls to the kernel on your behalf, something that eventually calls the C library, something that is needed for performance, etc etc etc) not to mention bugs in the compiler/JIT that either allow unsafe/unsound constructs or just straight up miscompile correct code. And finally, there are sources of HW issues unrelated to memory bitflips (your CPU is super complex and has bugs too you know as does your memory controller). > research has shown that the majority of one-off soft errors in DRAM chips occur as a result of background radiation, chiefly neutrons from cosmic ray secondaries, > Recent studies[5] show that single-event upsets due to cosmic radiation have been dropping dramatically with process geometry and previous concerns over increasing bit cell error rates are unfounded. So aside from the fact that DRAM susceptibility to cosmic rays has been decreasing (I’m not convinced with the Wikipedia explanation - an alternate explanation can be that the percentage of critically important data in DRAM has shrunk as a percentage as the overall capacity has increased), you’re argument would be that random cosmic radiation is going to randomly hit the DRAM cell containing your code / critical data. On the order of things that are likely, that’s the last thing. Oh and all this relies on you correctly grouping related crashes correctly and I’ve generally seen that to be a significant challenge on any project I’ve participated on (e.g. Oculus for the longest time would group unrelated crash reports and not group the same ones correctly although I helped the team try to make progress on that). Again, it’s not impossible that it’s a legit HW corruption issue. However, all the engineers I know frequently also blamed cosmic rays but at the end it’s all just shorthand for “not worth wasting time trying to track down because it’s in the tail of issues you’ll never get to“.

asveikau 3y ago

Also statistically and in my experience, a low probability event multiplied by a large number of units means millions of people hit hardware failures. It's not at all surprising that someone working on graphics drivers ends up seeing a lot of it.

vlovich123 3y ago
Yes, and the issues that ECC can correct for are a small fraction of that overall HW failure rate. And SW issues will always still dominate.
- asveikau 3y ago
  There are certain niches where the hardware issues will actually dominate the queue of issues and support requests. One I can relate to is if you have something with low code churn and extremely logically simple that does many gigabytes of I/O per machine over a short period. The maintainer of that code is going to see tons of bad disks. That was me once.