3 ms·

The strangest part of the whole thing is how they claim that the CPUs will work fine initially, but then degrade over time in a deterministic way. How is nobod

The strangest part of the whole thing is how they claim that the CPUs will work fine initially, but then degrade over time in a deterministic way. How is nobody else seeing this? Are these guys executing code paths that are normally not used? It sounds a bit like their code is buggy in a strange way that doesn't cause problems on AMD but does on later generation Intel, but that wouldn't explain why code would work initially but fail later unless maybe the problem was brought about by a microcode update from Intel? I fully agree that demanding Intel recall all of the CPUs at this point is premature. They've done nowhere near enough research to pin the problem on the processor and not their code.

jchw 2y ago
> How is nobody else seeing this? Well, ... they are. This has been being discussed for a while. It came up a bit ago, many people were seeing it in games with a weird error about low VRAM which apparently wound up being linked to Intel's 13th and 14th gen processors. Exactly why is anyone's guess. https://www.tomshardware.com/pc-components/cpus/nvidia-blames-intel-for-gpu-vram-errors-tells-geforce-gamers-experiencing-13th-or-14th-gen-cpu-instability-to-contact-intel-support https://www.tomshardware.com/pc-components/cpus/nvidia-blame... Level1Techs is relatively well-trusted, and they seem to have some contacts that can also corroborate these issues. https://www.youtube.com/watch?v=QzHcrbT5D_Y https://www.youtube.com/watch?v=QzHcrbT5D_Y
- willcipriano 2y ago
  One idea: Get 500 watt power supply, it powers new chip fine with it's unlimited configuration. 500 watt power supply becomes less efficient over time, now it's not quite 500 watts, parts of the chip brown out. Undefined behavior.
  - jchw 2y ago
    Hmmm, I think this is unlikely. For one thing, most people's power supplies are quite wildly overspecced in gaming rigs since its genuinely pretty hard to find good small ATX PSUs these days. For another, even crummy low-quality PSUs tend to mostly work pretty well for a reasonably long time in most cases, so it'd be weird to see this so much in high-end gaming rigs with brand new CPUs. I also don't know exactly what failure mode you are suggesting with this (voltage rail out of spec?) but I think unless it starts feeding too much voltage into the CPU it is probably unlikely to be able to cause damage. (Instability, sure.) It also doesn't explain why Intel's 12th generation CPUs are still running strong, along with most of AMD's lineup, by comparison. Don't get me wrong, every SKU of CPU has its issues, but all of the major issues with other CPU lines we have explanations for; the issues with Intel Raptor Lake are thusfar unexplained. Intel has blamed motherboard manufacturers quite extensively, but so far the evidence of this is pretty uncompelling, because the failures can apparently also be seen in abnormally high numbers on motherboards that never had unreasonable configurations.
  - eqvinox 2y ago
    Having built a few power management circuits myself: this is not how PSUs degrade, at least not in the first 5 years, and generally not "ever" in most cases. The PSU (notably in this case the M/B VRM, not the ATX PSU) can't really trigger brownouts. 1.05V is 1.05V, and if asked to do so, the PSU will deliver 1.05V. It will just get hotter while doing so over time (and specifically: capacitors degrading.) The voltage references are quite stable over time, and so are the window comparators that ensure "1.05V" is between, say, 1.035V and 1.065V. (Voltage references have essentially continuous engineering history back to the 50s≈60s, it's a "solved" problem.)
    - sliken 2y ago
      Varies by brand, some use better parts. I've definitely seen file servers that won't boot after 3 years, then I pull a few drives to lower the power needed and it boots. After another year or two I have to pull more drives. Even with generously specced power supplies, server class hardware, redundant power supplies, and enough smarts to stage the drive spin ups to avoid a large peak inrush current when 16 drives try to spin up at once.
      - touisteur 2y ago
        But GP's explanation still holds. The PSU will give the reference voltage, while being less and less efficient. Unable to deliver proper voltage (because it becomes less efficient with time), will not deliver any, until you remove the drives and it can deliver it? Did I misunderstand something ? Now I'm thinking of these HPE over-over-over-specced PSUs I run and y'all got me worried.
        sliken 2y ago
        Sort of. It's not just efficiency that's lost, but peak power. But if peak power was causing the CPU problems it wouldn't get better with a replacement CPU. Sure power supplies provide accurate power ... till they can't. Unlike peak power that starts to degrade from new, the voltages will hold as long as the PS can manage it.
  - jandrese 2y ago
    The problem description says it happens gradually over time, but affects roughly 100% of the machines client and server alike. I have a hard time believing that everybody has underspecced and failing power supplies, especially on their servers.
    - paulmd 2y ago
      that's just alderon_matt's assertion. wendell and GN have triaged this to approximately 10-25% of units, seems pretty consistent acros a whole bunch of different environments. https://www.youtube.com/watch?v=gTeubeCIwRw https://www.youtube.com/watch?v=gTeubeCIwRw similarly, the oodle maintainers only found it to affect a small % of machines (but reliably, of course). https://news.ycombinator.com/item?id=39486930 https://news.ycombinator.com/item?id=39486930 The alderon_matt thing is actually carefully worded to say affected machines invariably degrade... but they don't say how many machines are affected. It's deceptively worded, intended to be read the way you read it, but actually not saying that 100% of machines are affected either. at the end of the day game developers are just people too, and they're equally frustrated with the process/not willing to cut intel slack. but like, this is shades of the Ryzenfall thing (which, remember, contrary to early takes did have several real severity 9.0 issues behind it, including a PSP jailbreak and a UEFI module signing bypass, as well as memory encryption bypass and some other goodies - AMD doesn't issue AGESA patches for "root password lets you do root things"). People have moved past the facts and into the "trying to generate noise to get attention" on levels that are not supported by the facts. https://www.youtube.com/watch?v=QuqefIZrRWc https://www.youtube.com/watch?v=QuqefIZrRWc the Alderon Games assertion, specifically, is very aggressively/deceptively worded and is not supported by the broader facts, nor does he have any real idea what can or can't be patched. as wendell notes - all kinds of "unfixable" hardware bugs are indeed fixed in microcode, routinely. that's why it's there, because of the pentium FOOF bug. this isn't to say there isn't a degradation problem, there clearly is something (or more than one something) going on. but the alderon games press release is very over-the-top compared to all the rest of the evidence. GN is pointing at 10-25% and this is very consistent across a number of different operators, boards, environments, etc. That's the number Dell is getting in their validation (depending on the department/how stringent they validate), that's the number Wendell is seeing in operation in datacenter error reports. And they're thinking possibly some kind of fab defect affecting those units, basically (oxidation of the vias). https://youtu.be/5KHCLBqRrnY?t=2280 https://youtu.be/5KHCLBqRrnY?t=2280
  - sliken 2y ago
    Sure, that might explain it, except when people get replacements things are improved, but then start getting worse. Some even are on their 3rd CPU. What's making news is that Intel is starting (in at least one case) to deny RMA replacements after the first, despite shipping defective CPUs as replacements.
deleted 2y ago
[deleted]
eqvinox 2y ago
> Are these guys executing code paths that are normally not used? Yes: games are notorious for having poor multithreading and hitting one core with very high load. And unlike servers where you're rarely the only workload, the rest of the CPU will be rather idle on a desktop/gaming system. This pushes the CPU to a much more "imbalanced" mode (putting all the power and heat into a very small area) than is common elsewhere.
- hdhshdhshdjd 2y ago
  I have these issues on a dedicated Postgres server running Linux and literally nothing else. It’s not just games, it’s just a lot of gamers buy these chips.
wmf 2y ago
Silicon aging is a thing, especially at high voltage.

jchw 2y ago

> How is nobody else seeing this? Well, ... they are. This has been being discussed for a while. It came up a bit ago, many people were seeing it in games with a weird error about low VRAM which apparently wound up being linked to Intel's 13th and 14th gen processors. Exactly why is anyone's guess. https://www.tomshardware.com/pc-components/cpus/nvidia-blames-intel-for-gpu-vram-errors-tells-geforce-gamers-experiencing-13th-or-14th-gen-cpu-instability-to-contact-intel-support https://www.tomshardware.com/pc-components/cpus/nvidia-blame... Level1Techs is relatively well-trusted, and they seem to have some contacts that can also corroborate these issues. https://www.youtube.com/watch?v=QzHcrbT5D_Y https://www.youtube.com/watch?v=QzHcrbT5D_Y

willcipriano 2y ago
One idea: Get 500 watt power supply, it powers new chip fine with it's unlimited configuration. 500 watt power supply becomes less efficient over time, now it's not quite 500 watts, parts of the chip brown out. Undefined behavior.
- jchw 2y ago
  Hmmm, I think this is unlikely. For one thing, most people's power supplies are quite wildly overspecced in gaming rigs since its genuinely pretty hard to find good small ATX PSUs these days. For another, even crummy low-quality PSUs tend to mostly work pretty well for a reasonably long time in most cases, so it'd be weird to see this so much in high-end gaming rigs with brand new CPUs. I also don't know exactly what failure mode you are suggesting with this (voltage rail out of spec?) but I think unless it starts feeding too much voltage into the CPU it is probably unlikely to be able to cause damage. (Instability, sure.) It also doesn't explain why Intel's 12th generation CPUs are still running strong, along with most of AMD's lineup, by comparison. Don't get me wrong, every SKU of CPU has its issues, but all of the major issues with other CPU lines we have explanations for; the issues with Intel Raptor Lake are thusfar unexplained. Intel has blamed motherboard manufacturers quite extensively, but so far the evidence of this is pretty uncompelling, because the failures can apparently also be seen in abnormally high numbers on motherboards that never had unreasonable configurations.
- eqvinox 2y ago
  Having built a few power management circuits myself: this is not how PSUs degrade, at least not in the first 5 years, and generally not "ever" in most cases. The PSU (notably in this case the M/B VRM, not the ATX PSU) can't really trigger brownouts. 1.05V is 1.05V, and if asked to do so, the PSU will deliver 1.05V. It will just get hotter while doing so over time (and specifically: capacitors degrading.) The voltage references are quite stable over time, and so are the window comparators that ensure "1.05V" is between, say, 1.035V and 1.065V. (Voltage references have essentially continuous engineering history back to the 50s≈60s, it's a "solved" problem.)
  - sliken 2y ago
    Varies by brand, some use better parts. I've definitely seen file servers that won't boot after 3 years, then I pull a few drives to lower the power needed and it boots. After another year or two I have to pull more drives. Even with generously specced power supplies, server class hardware, redundant power supplies, and enough smarts to stage the drive spin ups to avoid a large peak inrush current when 16 drives try to spin up at once.
    - touisteur 2y ago
      But GP's explanation still holds. The PSU will give the reference voltage, while being less and less efficient. Unable to deliver proper voltage (because it becomes less efficient with time), will not deliver any, until you remove the drives and it can deliver it? Did I misunderstand something ? Now I'm thinking of these HPE over-over-over-specced PSUs I run and y'all got me worried.
      - sliken 2y ago
        Sort of. It's not just efficiency that's lost, but peak power. But if peak power was causing the CPU problems it wouldn't get better with a replacement CPU. Sure power supplies provide accurate power ... till they can't. Unlike peak power that starts to degrade from new, the voltages will hold as long as the PS can manage it.
- jandrese 2y ago
  The problem description says it happens gradually over time, but affects roughly 100% of the machines client and server alike. I have a hard time believing that everybody has underspecced and failing power supplies, especially on their servers.
  - paulmd 2y ago
    that's just alderon_matt's assertion. wendell and GN have triaged this to approximately 10-25% of units, seems pretty consistent acros a whole bunch of different environments. https://www.youtube.com/watch?v=gTeubeCIwRw https://www.youtube.com/watch?v=gTeubeCIwRw similarly, the oodle maintainers only found it to affect a small % of machines (but reliably, of course). https://news.ycombinator.com/item?id=39486930 https://news.ycombinator.com/item?id=39486930 The alderon_matt thing is actually carefully worded to say affected machines invariably degrade... but they don't say how many machines are affected. It's deceptively worded, intended to be read the way you read it, but actually not saying that 100% of machines are affected either. at the end of the day game developers are just people too, and they're equally frustrated with the process/not willing to cut intel slack. but like, this is shades of the Ryzenfall thing (which, remember, contrary to early takes did have several real severity 9.0 issues behind it, including a PSP jailbreak and a UEFI module signing bypass, as well as memory encryption bypass and some other goodies - AMD doesn't issue AGESA patches for "root password lets you do root things"). People have moved past the facts and into the "trying to generate noise to get attention" on levels that are not supported by the facts. https://www.youtube.com/watch?v=QuqefIZrRWc https://www.youtube.com/watch?v=QuqefIZrRWc the Alderon Games assertion, specifically, is very aggressively/deceptively worded and is not supported by the broader facts, nor does he have any real idea what can or can't be patched. as wendell notes - all kinds of "unfixable" hardware bugs are indeed fixed in microcode, routinely. that's why it's there, because of the pentium FOOF bug. this isn't to say there isn't a degradation problem, there clearly is something (or more than one something) going on. but the alderon games press release is very over-the-top compared to all the rest of the evidence. GN is pointing at 10-25% and this is very consistent across a number of different operators, boards, environments, etc. That's the number Dell is getting in their validation (depending on the department/how stringent they validate), that's the number Wendell is seeing in operation in datacenter error reports. And they're thinking possibly some kind of fab defect affecting those units, basically (oxidation of the vias). https://youtu.be/5KHCLBqRrnY?t=2280 https://youtu.be/5KHCLBqRrnY?t=2280
- sliken 2y ago
  Sure, that might explain it, except when people get replacements things are improved, but then start getting worse. Some even are on their 3rd CPU. What's making news is that Intel is starting (in at least one case) to deny RMA replacements after the first, despite shipping defective CPUs as replacements.

deleted 2y ago

[deleted]

eqvinox 2y ago

> Are these guys executing code paths that are normally not used? Yes: games are notorious for having poor multithreading and hitting one core with very high load. And unlike servers where you're rarely the only workload, the rest of the CPU will be rather idle on a desktop/gaming system. This pushes the CPU to a much more "imbalanced" mode (putting all the power and heat into a very small area) than is common elsewhere.

hdhshdhshdjd 2y ago
I have these issues on a dedicated Postgres server running Linux and literally nothing else. It’s not just games, it’s just a lot of gamers buy these chips.

wmf 2y ago

Silicon aging is a thing, especially at high voltage.