6 ms·

This sounds like motherboard manufacturers pushing aggressive OOTB performance settings, likely in excess of the Intel spec.

by scrlk 3y ago

eqvinox 3y ago
That's… an assumption. At least 3 motherboard vendors are affected, and going by the Gigabyte/MSI workarounds at the end of the article, it looks like things need to be adjusted away from Intel defaults. …it'll need a statement from Intel for some clarity on this…
- scrlk 3y ago
  > "Intel's default maximum TDP for the 13900K is 253 watts, though it can easily consume 300 watts or more when given a higher power limit. In our testing, manually setting the power limit to 275–300 watts and the amperage limit to 350A, proved to be perfectly stable for our 13900K. That required going into the advanced CPU settings in the BIOS to change the PL1/PL2 limits — called short and long duration power limits in our particular case. The motherboard's default "Auto" power and current limits meanwhile created instability issues — which correspond to a power limit of 4,096 watts and 4,096 amps." [0] The motherboard manufacturers are setting default/auto power and current limits that are way outside of Intel's specs (253 W, 307 A) [1]. [0] https://www.tomshardware.com/pc-components/cpus/is-your-intel-core-i9-13900k-crashing-in-games-your-motherboard-bios-settings-may-be-to-blame-other-high-end-intel-cpus-also-affected https://www.tomshardware.com/pc-components/cpus/is-your-inte... [1] https://www.intel.com/content/www/us/en/content-details/743844/13th-generation-intel-core-and-intel-core-14th-generation-processors-datasheet-volume-1-of-2.html https://www.intel.com/content/www/us/en/content-details/7438... (see pg. 98 and 184, the 13900K/14900K is 8P + 16E 125 W)
  - eqvinox 3y ago
    Your [0] also says: > It's not exactly clear why the 13900K suffers from these instability problems, and how exactly downclocking, lowering the power/current limits, and undervolting prevent further crashes. Clearly, something is going wrong with some CPUs. Are they "defective" or merely not capable of running the out of spec settings used by many motherboards?
    - scrlk 3y ago
      > Are they "defective" or merely not capable of running the out of spec settings used by many motherboards? I'd wager good money on the latter. Why would Intel validate their CPUs against power and current limits that are outside of spec? The users reporting issues probably have CPUs that just made it in to the performance envelope to be binned as a 13900K, so running out of spec settings on these weaker chips results in instability. It's cases like this where I wish Intel didn't exit the motherboard space, they were known to be reliable but typically at the cost of having a more limited feature set.
      - RetroTechie 3y ago
        > I'd wager good money on the latter. Don't guess, measure! The proper action here would be to change BIOS settings from their default / "auto" settings to per-Intel-spec safe ones. Same for RAM, and on systems with known good power supplies, CPU cooling, software installs etc. Then one of the following will happen: a) BIOS ignores user settings & problem persists. b) BIOS applies user settings & problem goes away. c) BIOS applies user settings but problem persists. Cases a & b count as "faulty BIOS" (motherboard manufacturer caused). Case c counts as "faulty CPU", and replacement cpu may or may not fix that. No need to guess. Just do the legwork on systems where problem occurs & power supply, RAM, CPU cooling & OS install can be ruled out. Sadly, no doubt there's many systems out there where that last condition doesn't hold.
        vdaea 3y ago
        I have a 13900K. The default BIOS settings set a maximum wattage of 4096W (!!!) that makes Prime95 fail. If I change the settings back to 253W, what Intel says is the maximum wattage, Prime95 stops failing. Still, I don't know if I should RMA. I got the K version because I intended to overclock in the future. And all of this sounds like I won't be able to. I think increasing the voltage a little bit makes the system more stable. I have to play with it. (Really, if someone can say whether I should RMA or not, I would appreciate some input) Edit: decided to RMA. I have no patience for a CPU that cost me +600€
      - 3y ago
  - mschuster91 3y ago
    > amperage limit to 350A Jesus. By German electrical code, you need a 70 mm² cross-section of copper to transfer that kind of current without the cable heating up to a point that it endangers the insulation. How do mainboard manufacturers supply that kind of current without resistive loss from the traces frying everything?
    - bonzini 3y ago
      Those electrical code cross sections are for 350A at 230V, corresponding to about 80 kW (400V is the same as it's actually three 230V wires). Processors operate at about 1V. At 300W it's enough to use a much smaller cross section, which is split across many traces.
      - coryrc 3y ago
        That's not how I^2R losses work. Voltage is not relevant.
        planede 3y ago
        I agree. However voltage is relevant for insulation, which also affects how heat can dissipate for the wire, and might also be relevant for a failing wire, when higher and higher voltage can build up at the point of failure (not sure if it's a common engineering consideration outside of fuses, which are designed to fail).
        bonzini 3y ago
        At 300W there's only so much power that can become heat. With 80 kW flowing through the wire, if the insulation melts due to excessive wire resistance you call the fire brigade.
    - coryrc 3y ago
      That code is for round wire (minimal surface area per volume) that can be placed inside insulation in walls. This 350A is flat conductors (maximal surface area thus heat dissipation) and very short (not that much power to dissipate so the things it connects to have a significant effect on heat dissipation).
  - michaelt 3y ago
    > The motherboard manufacturers are setting default/auto power and current limits that are way outside of Intel's specs The CPU only draws as much power as it needs, though? I mean, if you plug a 20 watt phone into a 60 watt USB-C power supply, or a 60 watt laptop into a 100 watt USB-C power supply the device doesn't get overloaded with power. It draws no more current than it needs. The motherboard's power limits should state the amount of power the PCB traces and buck regulators are rated to provide to the socket - and if that's more than the processor needs that's good, as it avoids throttling.
    - rfoo 3y ago
      The problem is these processors are unstable if they are not properly throttled. Of course users, especially enthusiast motherboard consumers, hate throttling, hence the default.
    - scrlk 3y ago
      * User is running a demanding application (e.g. game). * CPU clock speed increases (turbo boost), as long as the CPU isn't hitting: 1) Tj_MAX (max temp before thermal throttling kicks in); 2) the power and current limits specified by the motherboard (in this case, effectively disabled by the out of spec settings). * Weaker chips will require more power to hit or maintain a given turbo clock speed: with the power and current limits disabled, the CPU will attempt to draw out of spec power & current, causing issues for the on die fully-integrated voltage regulator (noting that there's also performance/quality variance for the FIVR), resulting in the user experiencing instability.
  - wongarsu 3y ago
    Drawing peak power far in excess of the TDP is what all Intel processors have been designed to do for many years now. Some consider it cheating the benchmarks, but the justification is that TDP is the Thermal Design Power. It's about the cooling system you need, not the power delivery. If you make reasonable assumptions about the thermal inertia of the cooling system you can Turbo Boost at higher power and hope the workload is over before you are forced to throttle down again. Any mainboard that sets power limits to the TDP would be considered wrong by both the community and Intel. This looks like a solid indication that the issue is with Intel
- lupusreal 3y ago
  > At least 3 motherboard vendors are affected Boss says, "Do the thing." Engineer says, "The thing is out of spec!" Boss says, "Competitor is doing the thing already and it works." Engineer does the thing.
  - eqvinox 3y ago
    Yes, but also no. The motherboard market is "timing-competitive", your product needs to be ready when the CPU launches, especially for the kind of flagship CPU that this specific issue is about. You can't wait and see what the competitors are doing.
    - lupusreal 3y ago
      Fair point. Maybe "This sort of thing worked fine in the past."
  - crote 3y ago
    Or perhaps they're all copying from the same reference design?
- paulmd 3y ago
  all three motherboard vendors enabling some out-of-spec defaults wouldn't actually be surprising, though? people forget, blowing up AM5 cpus wasn't just as Asus thing... they were just the most ham-handed with the voltages. Everyone was operating out of spec, there were chips that blew up on MSI and Gigabyte boards, and it wasn't just X3D either. Intel is no different - nobody enforces the power limit out of the box, and XMP will happily punch voltages up to levels that result in eventual degradation/electromigration of processors (on the order of years). Every enthusiast knows that CPU failures are "rare" and yet either has had some, or knows someone who's had some in their immediate circles. Because XMP actually has caused non-trivial degradation even on most DDR4 platforms. In fact it's entirely possible that this is an electromigration issue right here too - notice how this affects some 13700Ks and 13900Ks too? Those chips have been run for a year or two now. And if the processors were marginal to begin with, and operated at out-of-spec voltages (intentionally or not)... they could be starting to wear out a little bit under the heaviest loads. Or the memory controllers could be starting to lose stability at the highest clocks (under the heaviest loads). That's a thing that's not uncommon on 7nm and 5nm tier nodes.
  - rygorous 3y ago
    This is blowing up now, but the first report of this kind of issue that reached me (I'm the current Oodle maintainer) was in spring of last year. We've been trying to track it down (and been in contact with Intel) since then. The page linked in the OP has been up since December. Epic Games Tools is B2B and we don't generally get bug reports from end users (although later last year, we did have 2 end users write to us directly because of this problem - first time this has happened for Oodle that I can think of, and I've been working on this project since 2015). Point being, we're normally at least one level removed from end user bug reports, so add at least a few weeks while our customers get bug reports from end users but haven't seen enough of them yet to get in touch with us (this is a rare failure that only affects a small fraction of machines). 13900Ks have been out since late Oct 2022. It's possible that this doesn't show up on parts right out of the box and takes a few months. It's equally plausible that it's been happening for some people for as long as they've had those CPUs, and the first such customers just bought their new machines late 2022, maybe reported a bug around the holidays/EOY that nobody looked at until January, and then it took another 2-3 months for 3-4 other similar crashes to show up that ultimately resulted in this case getting escalated to us.

This sounds like motherboard manufacturers pushing aggressive OOTB performance settings, likely in excess of the Intel spec.

by scrlk 3y ago

This sounds like motherboard manufacturers pushing aggressive OOTB performance settings, likely in excess of the Intel spec.

eqvinox 3y ago

That's… an assumption. At least 3 motherboard vendors are affected, and going by the Gigabyte/MSI workarounds at the end of the article, it looks like things need to be adjusted away from Intel defaults. …it'll need a statement from Intel for some clarity on this…

scrlk 3y ago
> "Intel's default maximum TDP for the 13900K is 253 watts, though it can easily consume 300 watts or more when given a higher power limit. In our testing, manually setting the power limit to 275–300 watts and the amperage limit to 350A, proved to be perfectly stable for our 13900K. That required going into the advanced CPU settings in the BIOS to change the PL1/PL2 limits — called short and long duration power limits in our particular case. The motherboard's default "Auto" power and current limits meanwhile created instability issues — which correspond to a power limit of 4,096 watts and 4,096 amps." [0] The motherboard manufacturers are setting default/auto power and current limits that are way outside of Intel's specs (253 W, 307 A) [1]. [0] https://www.tomshardware.com/pc-components/cpus/is-your-intel-core-i9-13900k-crashing-in-games-your-motherboard-bios-settings-may-be-to-blame-other-high-end-intel-cpus-also-affected https://www.tomshardware.com/pc-components/cpus/is-your-inte... [1] https://www.intel.com/content/www/us/en/content-details/743844/13th-generation-intel-core-and-intel-core-14th-generation-processors-datasheet-volume-1-of-2.html https://www.intel.com/content/www/us/en/content-details/7438... (see pg. 98 and 184, the 13900K/14900K is 8P + 16E 125 W)
- eqvinox 3y ago
  Your [0] also says: > It's not exactly clear why the 13900K suffers from these instability problems, and how exactly downclocking, lowering the power/current limits, and undervolting prevent further crashes. Clearly, something is going wrong with some CPUs. Are they "defective" or merely not capable of running the out of spec settings used by many motherboards?
  - scrlk 3y ago
    > Are they "defective" or merely not capable of running the out of spec settings used by many motherboards? I'd wager good money on the latter. Why would Intel validate their CPUs against power and current limits that are outside of spec? The users reporting issues probably have CPUs that just made it in to the performance envelope to be binned as a 13900K, so running out of spec settings on these weaker chips results in instability. It's cases like this where I wish Intel didn't exit the motherboard space, they were known to be reliable but typically at the cost of having a more limited feature set.
    - RetroTechie 3y ago
      > I'd wager good money on the latter. Don't guess, measure! The proper action here would be to change BIOS settings from their default / "auto" settings to per-Intel-spec safe ones. Same for RAM, and on systems with known good power supplies, CPU cooling, software installs etc. Then one of the following will happen: a) BIOS ignores user settings & problem persists. b) BIOS applies user settings & problem goes away. c) BIOS applies user settings but problem persists. Cases a & b count as "faulty BIOS" (motherboard manufacturer caused). Case c counts as "faulty CPU", and replacement cpu may or may not fix that. No need to guess. Just do the legwork on systems where problem occurs & power supply, RAM, CPU cooling & OS install can be ruled out. Sadly, no doubt there's many systems out there where that last condition doesn't hold.
      - vdaea 3y ago
        I have a 13900K. The default BIOS settings set a maximum wattage of 4096W (!!!) that makes Prime95 fail. If I change the settings back to 253W, what Intel says is the maximum wattage, Prime95 stops failing. Still, I don't know if I should RMA. I got the K version because I intended to overclock in the future. And all of this sounds like I won't be able to. I think increasing the voltage a little bit makes the system more stable. I have to play with it. (Really, if someone can say whether I should RMA or not, I would appreciate some input) Edit: decided to RMA. I have no patience for a CPU that cost me +600€
    - 3y ago
- mschuster91 3y ago
  > amperage limit to 350A Jesus. By German electrical code, you need a 70 mm² cross-section of copper to transfer that kind of current without the cable heating up to a point that it endangers the insulation. How do mainboard manufacturers supply that kind of current without resistive loss from the traces frying everything?
  - bonzini 3y ago
    Those electrical code cross sections are for 350A at 230V, corresponding to about 80 kW (400V is the same as it's actually three 230V wires). Processors operate at about 1V. At 300W it's enough to use a much smaller cross section, which is split across many traces.
    - coryrc 3y ago
      That's not how I^2R losses work. Voltage is not relevant.
      - planede 3y ago
        I agree. However voltage is relevant for insulation, which also affects how heat can dissipate for the wire, and might also be relevant for a failing wire, when higher and higher voltage can build up at the point of failure (not sure if it's a common engineering consideration outside of fuses, which are designed to fail).
      - bonzini 3y ago
        At 300W there's only so much power that can become heat. With 80 kW flowing through the wire, if the insulation melts due to excessive wire resistance you call the fire brigade.
  - coryrc 3y ago
    That code is for round wire (minimal surface area per volume) that can be placed inside insulation in walls. This 350A is flat conductors (maximal surface area thus heat dissipation) and very short (not that much power to dissipate so the things it connects to have a significant effect on heat dissipation).
- michaelt 3y ago
  > The motherboard manufacturers are setting default/auto power and current limits that are way outside of Intel's specs The CPU only draws as much power as it needs, though? I mean, if you plug a 20 watt phone into a 60 watt USB-C power supply, or a 60 watt laptop into a 100 watt USB-C power supply the device doesn't get overloaded with power. It draws no more current than it needs. The motherboard's power limits should state the amount of power the PCB traces and buck regulators are rated to provide to the socket - and if that's more than the processor needs that's good, as it avoids throttling.
  - rfoo 3y ago
    The problem is these processors are unstable if they are not properly throttled. Of course users, especially enthusiast motherboard consumers, hate throttling, hence the default.
  - scrlk 3y ago
    * User is running a demanding application (e.g. game). * CPU clock speed increases (turbo boost), as long as the CPU isn't hitting: 1) Tj_MAX (max temp before thermal throttling kicks in); 2) the power and current limits specified by the motherboard (in this case, effectively disabled by the out of spec settings). * Weaker chips will require more power to hit or maintain a given turbo clock speed: with the power and current limits disabled, the CPU will attempt to draw out of spec power & current, causing issues for the on die fully-integrated voltage regulator (noting that there's also performance/quality variance for the FIVR), resulting in the user experiencing instability.
- wongarsu 3y ago
  Drawing peak power far in excess of the TDP is what all Intel processors have been designed to do for many years now. Some consider it cheating the benchmarks, but the justification is that TDP is the Thermal Design Power. It's about the cooling system you need, not the power delivery. If you make reasonable assumptions about the thermal inertia of the cooling system you can Turbo Boost at higher power and hope the workload is over before you are forced to throttle down again. Any mainboard that sets power limits to the TDP would be considered wrong by both the community and Intel. This looks like a solid indication that the issue is with Intel
lupusreal 3y ago
> At least 3 motherboard vendors are affected Boss says, "Do the thing." Engineer says, "The thing is out of spec!" Boss says, "Competitor is doing the thing already and it works." Engineer does the thing.
- eqvinox 3y ago
  Yes, but also no. The motherboard market is "timing-competitive", your product needs to be ready when the CPU launches, especially for the kind of flagship CPU that this specific issue is about. You can't wait and see what the competitors are doing.
  - lupusreal 3y ago
    Fair point. Maybe "This sort of thing worked fine in the past."
- crote 3y ago
  Or perhaps they're all copying from the same reference design?
paulmd 3y ago
all three motherboard vendors enabling some out-of-spec defaults wouldn't actually be surprising, though? people forget, blowing up AM5 cpus wasn't just as Asus thing... they were just the most ham-handed with the voltages. Everyone was operating out of spec, there were chips that blew up on MSI and Gigabyte boards, and it wasn't just X3D either. Intel is no different - nobody enforces the power limit out of the box, and XMP will happily punch voltages up to levels that result in eventual degradation/electromigration of processors (on the order of years). Every enthusiast knows that CPU failures are "rare" and yet either has had some, or knows someone who's had some in their immediate circles. Because XMP actually has caused non-trivial degradation even on most DDR4 platforms. In fact it's entirely possible that this is an electromigration issue right here too - notice how this affects some 13700Ks and 13900Ks too? Those chips have been run for a year or two now. And if the processors were marginal to begin with, and operated at out-of-spec voltages (intentionally or not)... they could be starting to wear out a little bit under the heaviest loads. Or the memory controllers could be starting to lose stability at the highest clocks (under the heaviest loads). That's a thing that's not uncommon on 7nm and 5nm tier nodes.
- rygorous 3y ago
  This is blowing up now, but the first report of this kind of issue that reached me (I'm the current Oodle maintainer) was in spring of last year. We've been trying to track it down (and been in contact with Intel) since then. The page linked in the OP has been up since December. Epic Games Tools is B2B and we don't generally get bug reports from end users (although later last year, we did have 2 end users write to us directly because of this problem - first time this has happened for Oodle that I can think of, and I've been working on this project since 2015). Point being, we're normally at least one level removed from end user bug reports, so add at least a few weeks while our customers get bug reports from end users but haven't seen enough of them yet to get in touch with us (this is a rare failure that only affects a small fraction of machines). 13900Ks have been out since late Oct 2022. It's possible that this doesn't show up on parts right out of the box and takes a few months. It's equally plausible that it's been happening for some people for as long as they've had those CPUs, and the first such customers just bought their new machines late 2022, maybe reported a bug around the holidays/EOY that nobody looked at until January, and then it took another 2-3 months for 3-4 other similar crashes to show up that ultimately resulted in this case getting escalated to us.