31 ms·

I wish there was a hard requirement for ECC, as a developer working on GPU drivers, there's a huge amount of reported issues that just... don't make sense? One

by kimixa 2y ago

I wish there was a hard requirement for ECC, as a developer working on GPU drivers, there's a huge amount of reported issues that just... don't make sense? One offs with slightly different symptoms, memory dumps of nonsense, just nowhere to start rooting out the cause for an issue. Even on "widely reported" issues that make it to reddit and similar. Probably not surprising, there's a naturally antagonistic relationship between Performance and Reliability here, and it's clear which way many of those "enthusiast" forums lean. I haven't got actual numbers, but I feel that most [0] of the issues I start looking at just can never be reproduced, or even make sense from the backtrace or similar. I can't say it's 100% hardware issues for this, as many games are a little... loose... with reliability if it works "well enough", and is heavily interacting with code and data we work on so might also be a source for "impossible" issues. But even on straightforward code paths, no weird OS interaction, no allocation, nothing async etc. "Impossible" states happen pretty regularly. I would love there to be enough ECC-using gamers out there to statistically see if it makes a difference. [0] Most in terms of number of different issues, not total reports of the same issue. That's dominated by one or two things, normally around the latest game or update doing something dumb :P

a_t48 2y ago
Back when I heavily used GCP's GPUs I'd actually get failed tests with ECC errors. It was kind of neat.
vlovich123 2y ago
Statistically, in my experience, plain old memory corruption bugs within your code or within the GPU itself are a more likely explanation than issues ECC would fix. Games have pretty terrible quality because games can remain playable in the face of bugs. GPUs have similar issues because numerical accuracy is an after thought but numerical accuracy issues can cause issues in code that assume stability of results. Things have gotten better particular on the GPU vendor side of things, but still, memory safety issues and UB issues in c/C++ code far dominate as the root cause any issue you’re likely to see vs memory bit flips (unless people are running very overclocked and unstable machines).
- kimixa 2y ago
  I'm very much aware of issues mistakes around memory can cause, and we use many tools (including different languages) to avoid them or verify code. My point is even taking that into account there's a long tail of unexplained issues. And UB shouldn't be a scary thing 99% of the time, as you won't be hitting it anyway (or its actually the result of another real bug, like not handling overflows etc.), though as at some level you start trying on platform specific behavior and start /defining/ them. And again, there are tools and options around that, or highlighting areas where you might still be relying on them. They just might be toolchain and/or platform specific. It's never been the scary monster some online programming language fanatics seem to make it sound like, but is something to be aware of and managed. And > unless people are running very overclocked and unstable machines Yes. Have you seen the Gamer Hardware Enthusiast community?
  - worthless-trash 2y ago
    > And UB shouldn't be a scary thing 99% of the time, as you won't be hitting it anyway (or its actually the result of another real bug, This is exactly how exploits are made. Accessing UB usually is the result of "another bug" because the UB isnt a bug itself.
  - Sweepi 2y ago
    >> unless people are running very overclocked and unstable machines > Yes. Have you seen the Gamer Hardware Enthusiast community? Having seen it for the last 2 decades: A lot of them are interested in OC, but nobody is doing it. Maybe 5%, most likely <2%. Also, most modern Motherboards do a "RAM-Training" session before boot. On unstable machines, this will result in a "test failed" once in a while, with the motherboard showing a "Boot failed, returned every clock setting to default" message, which most likely is not being read by the user, who is either not at the PC the moment it happens or in "why is this taking so long? skip, skip, skip"- mode. What does happen is that Motherboards have OC-like settings out-of-the-box/per default, which were stable most of the time and de facto-accepted by Intel, however is now hitting issues and diminishing returns: https://www.radgametools.com/oodleintel.htm https://www.radgametools.com/oodleintel.htm https://www.igorslab.de/en/intel-spielt-mit-dem-namen-und-den-daten-das-intel-baseline-profile-wird-zu-intel-default-settings/ https://www.igorslab.de/en/intel-spielt-mit-dem-namen-und-de...
Onavo 2y ago
And then you have car companies like Tesla that removed ECC memory in newer iterations of their car computers to save money.
- bayindirh 2y ago
  At worst, you suddenly veer off the road and roll a couple of times, no biggie. It's not a rocket or something dangerous, right? Also, newer iterations of the car will be even more cheaper due to cut corners, so you may save some money in the process, too.
  - 72deluxe 2y ago
    I followed a Tesla yesterday. It was clearly on autopilot along this single lane road and it was veering all over the place like a drunk person, sometimes into the verge, sometimes over the double white line (the line you should not cross in the UK) and obviously making micro adjustments to steering with periodic random braking. I don't see how someone could relax as a passenger in that. Shockingly bad.
    - pipe01 2y ago
      The latest fsd version definitely doesn't do that, you can look up videos on yt
      - 72deluxe 2y ago
        Perhaps it really was a drunk driver then!
    - giantg2 2y ago
      I wonder if the software quality is lower for self-driving in non-US roads. I assume there would be less focus on development and testing in other markets.
      - bayindirh 2y ago
NohatCoder 2y ago
I forgot the source, but I recall some game dev telling that they had taken to run a basic stability test every time the game launched. Half the crash reports came from systems that failed the test, so those were summarily ignored. There is an error correction layer in DDR5, could that be spotted statistically as a difference between Ryzen 5000 and 7000 processors?
- kevingadd 2y ago
  We did this in Guild Wars, yeah. I'm sure lots of other games are doing it by now. There were a significant number of Problem Customer PCs that would just crash all the time because of stuff that was obviously a CPU or RAM defect, though I don't know if it was half our crash reports.
  - immibis 2y ago
    What kind of stability test is quick enough to run every time the game starts and good enough to detect real problems?
    - kevingadd 2y ago
      I believe it was actually an ambient check of things like floating point arithmetic that ran on an ongoing basis, and if it ever failed we popped up an error message. We already shipped with debug assertions enabled in release builds, so it wouldn't have been unprecedented to do things like validate result bytes after doing big memcpy operations or things like that. Not sure if we did, though.
  - steve_rambo 2y ago
    What was the difference between CPU and video card vendors (if you can talk about that at all)?
fourfour3 2y ago
I’ve found crashing in games using NVIDIA’s DX12 driver is one of the most reliable indicators of CPU/RAM instability in gaming PCs, so doesn’t surprise me! Lots of people’s gaming PCs (self built and otherwise) are very crashy - there’s a surprising number of people who get regular BSODs who just accept it as “normal”.
- formerly_proven 2y ago
  Also see the recent "UE5 is GARBAGE it KEEPS CRASHING on my rock-solid 14900K" episode.
kevin_nisbet 2y ago
Early in my career I had to deal with similar issues, but with cellular networking equipment. I do think it's one of those things where if you have a way to detect the bit flips it happens way more than you think... but most people can't detect them and mostly get lucky enough that they never see an impact when it does happen. But because we had some equipment that would checksum itself we were able to see the bit flips happen, probably every 1-3 weeks or so I'd find a bit flip.
- immibis 2y ago
  I literally just built a new workstation with 256GB of DDR5 ECC memory and had two detected bitflips within the first hour. It's likely that something's off with the timings or voltages, then, but... if I didn't have ECC, didn't have an AMD processor that generates MCEs Linux can decode, or didn't have a kernel set to log MCEs, would I know about them? Definitely not.
  - deleted 2y ago
    [deleted]
kevingadd 2y ago
I got ECC on my previous system (at great cost) but had to settle for non-ECC for my current workstation since the cost of Threadripper builds has gone up so much since the 3xxx era. It's really frustrating, even if DDR5 (supposedly) has much lower error rates. Every time a game crashes I'm going to suspect the RAM.

by kimixa 2y ago

a_t48 2y ago

Back when I heavily used GCP's GPUs I'd actually get failed tests with ECC errors. It was kind of neat.

vlovich123 2y ago

Statistically, in my experience, plain old memory corruption bugs within your code or within the GPU itself are a more likely explanation than issues ECC would fix. Games have pretty terrible quality because games can remain playable in the face of bugs. GPUs have similar issues because numerical accuracy is an after thought but numerical accuracy issues can cause issues in code that assume stability of results. Things have gotten better particular on the GPU vendor side of things, but still, memory safety issues and UB issues in c/C++ code far dominate as the root cause any issue you’re likely to see vs memory bit flips (unless people are running very overclocked and unstable machines).

kimixa 2y ago
I'm very much aware of issues mistakes around memory can cause, and we use many tools (including different languages) to avoid them or verify code. My point is even taking that into account there's a long tail of unexplained issues. And UB shouldn't be a scary thing 99% of the time, as you won't be hitting it anyway (or its actually the result of another real bug, like not handling overflows etc.), though as at some level you start trying on platform specific behavior and start /defining/ them. And again, there are tools and options around that, or highlighting areas where you might still be relying on them. They just might be toolchain and/or platform specific. It's never been the scary monster some online programming language fanatics seem to make it sound like, but is something to be aware of and managed. And > unless people are running very overclocked and unstable machines Yes. Have you seen the Gamer Hardware Enthusiast community?
- worthless-trash 2y ago
  > And UB shouldn't be a scary thing 99% of the time, as you won't be hitting it anyway (or its actually the result of another real bug, This is exactly how exploits are made. Accessing UB usually is the result of "another bug" because the UB isnt a bug itself.
- Sweepi 2y ago
  >> unless people are running very overclocked and unstable machines > Yes. Have you seen the Gamer Hardware Enthusiast community? Having seen it for the last 2 decades: A lot of them are interested in OC, but nobody is doing it. Maybe 5%, most likely <2%. Also, most modern Motherboards do a "RAM-Training" session before boot. On unstable machines, this will result in a "test failed" once in a while, with the motherboard showing a "Boot failed, returned every clock setting to default" message, which most likely is not being read by the user, who is either not at the PC the moment it happens or in "why is this taking so long? skip, skip, skip"- mode. What does happen is that Motherboards have OC-like settings out-of-the-box/per default, which were stable most of the time and de facto-accepted by Intel, however is now hitting issues and diminishing returns: https://www.radgametools.com/oodleintel.htm https://www.radgametools.com/oodleintel.htm https://www.igorslab.de/en/intel-spielt-mit-dem-namen-und-den-daten-das-intel-baseline-profile-wird-zu-intel-default-settings/ https://www.igorslab.de/en/intel-spielt-mit-dem-namen-und-de...

Onavo 2y ago

And then you have car companies like Tesla that removed ECC memory in newer iterations of their car computers to save money.

bayindirh 2y ago
At worst, you suddenly veer off the road and roll a couple of times, no biggie. It's not a rocket or something dangerous, right? Also, newer iterations of the car will be even more cheaper due to cut corners, so you may save some money in the process, too.
- 72deluxe 2y ago
  I followed a Tesla yesterday. It was clearly on autopilot along this single lane road and it was veering all over the place like a drunk person, sometimes into the verge, sometimes over the double white line (the line you should not cross in the UK) and obviously making micro adjustments to steering with periodic random braking. I don't see how someone could relax as a passenger in that. Shockingly bad.
  - pipe01 2y ago
    The latest fsd version definitely doesn't do that, you can look up videos on yt
    - 72deluxe 2y ago
      Perhaps it really was a drunk driver then!
  - giantg2 2y ago
    I wonder if the software quality is lower for self-driving in non-US roads. I assume there would be less focus on development and testing in other markets.
    - bayindirh 2y ago

NohatCoder 2y ago

I forgot the source, but I recall some game dev telling that they had taken to run a basic stability test every time the game launched. Half the crash reports came from systems that failed the test, so those were summarily ignored. There is an error correction layer in DDR5, could that be spotted statistically as a difference between Ryzen 5000 and 7000 processors?

kevingadd 2y ago
We did this in Guild Wars, yeah. I'm sure lots of other games are doing it by now. There were a significant number of Problem Customer PCs that would just crash all the time because of stuff that was obviously a CPU or RAM defect, though I don't know if it was half our crash reports.
- immibis 2y ago
  What kind of stability test is quick enough to run every time the game starts and good enough to detect real problems?
  - kevingadd 2y ago
    I believe it was actually an ambient check of things like floating point arithmetic that ran on an ongoing basis, and if it ever failed we popped up an error message. We already shipped with debug assertions enabled in release builds, so it wouldn't have been unprecedented to do things like validate result bytes after doing big memcpy operations or things like that. Not sure if we did, though.
- steve_rambo 2y ago
  What was the difference between CPU and video card vendors (if you can talk about that at all)?

fourfour3 2y ago

I’ve found crashing in games using NVIDIA’s DX12 driver is one of the most reliable indicators of CPU/RAM instability in gaming PCs, so doesn’t surprise me! Lots of people’s gaming PCs (self built and otherwise) are very crashy - there’s a surprising number of people who get regular BSODs who just accept it as “normal”.

formerly_proven 2y ago
Also see the recent "UE5 is GARBAGE it KEEPS CRASHING on my rock-solid 14900K" episode.

kevin_nisbet 2y ago

Early in my career I had to deal with similar issues, but with cellular networking equipment. I do think it's one of those things where if you have a way to detect the bit flips it happens way more than you think... but most people can't detect them and mostly get lucky enough that they never see an impact when it does happen. But because we had some equipment that would checksum itself we were able to see the bit flips happen, probably every 1-3 weeks or so I'd find a bit flip.

immibis 2y ago
I literally just built a new workstation with 256GB of DDR5 ECC memory and had two detected bitflips within the first hour. It's likely that something's off with the timings or voltages, then, but... if I didn't have ECC, didn't have an AMD processor that generates MCEs Linux can decode, or didn't have a kernel set to log MCEs, would I know about them? Definitely not.
- deleted 2y ago
  [deleted]

kevingadd 2y ago

I got ECC on my previous system (at great cost) but had to settle for non-ECC for my current workstation since the cost of Threadripper builds has gone up so much since the 3xxx era. It's really frustrating, even if DDR5 (supposedly) has much lower error rates. Every time a game crashes I'm going to suspect the RAM.