5 ms·
Puget Systems' Perspective on Intel CPU Instability Issues
- layer8 2y agoThe subsection “Failure Rates in Context” struck me as the most interesting.
- stn8188 2y agoI read this article yesterday and thought it was interesting but would like a bit more data. Most importantly, the first plot shows similar failures per month of 11th and 14th Gen, but the final plot shows that failure rates of 11th Gen was far higher (about double). Does this mean there are about double the number of systems built by Puget with the 14th Gen than they had of 11th Gen? I'd also love to see the first two plots with AMD data.
- JBiserkov 2y agoThey got 1 pun in their comedic relief budget and they spent it brilliantly: > We have enough data to know that we don’t have an acute problem on the horizon with 13th Gen — it is more of a slow burn.
- deleted 2y ago[deleted]
- johnobrien1010 2y agoWhich part of that is a pun?
- michaelmrose 2y agoDo I read the graph correctly that 14th gen had a ~10% per month failure rate each each month of May–July 2024 for a cumulative failure rate of almost 30% even with much more conservative than industry average power settings/clocking? Have I misunderstood the graph or is it actually that awful?
- upon_drumhead 2y agoThe chart is raw counts, not a percentage > Even though failure rates (as a percentage) are the most consequential, I think showing the absolute number of failures illustrates our experience best
- michaelmrose 2y agoHow would absolute numbers mean anything at all without the percentages? Did they spend their time making a useless graph?
- upon_drumhead 2y agoThey cover that down the page under the "Failure Rates in Context" section > Everything I’ve shown you so far is our raw number of failures, but what matters most is failure rate percentages. Let’s look at total failure rates in the context of multiple generations and with comparison to AMD Ryzen CPUs.
- w10-1 2y agoSorry, unable to believe: 2-4% failure rate for CPU's? That's for detected/known failures: what about random, unable-to-reproduce, hardly noticed the data skip failures? Have I been living in a fantasy bubble where CPU's do exactly what you asked of them (and errors come from not holding it right)?
- gjsman-1000 2y ago4% seems very high to me; but CPU errors happen with relative frequency, and design mistakes are common. If you ever run “cat /proc/cpuinfo” on, say, Skylake - Linux will happily tell you it has 5-6 workarounds active for hardware mistakes. CPUs are still pretty darn reliable. Think about how many GHz your CPU runs at, multiplied many instructions per cycle there are, and then calculate the failure rate if there was just 1 mistake per minute. Nothing on earth would compare.
- upon_drumhead 2y agoMechanical hard drives are absolutely on the same, or higher, level of reliability. It's mind boggling what we can achieve when we really focus on quality outcomes.
- deleted 2y ago[deleted]
- veqq 2y agoWow, you weren't joking: > apic_c1e spectre_v1 spectre_v2 spec_store_bypass swapgs taa itlb_multihit srbds mmio_stale_data retbleed eibrs_pbrsb gds bhi
- asveikau 2y agoI mean, a bunch of those are timing issues in speculative execution, you could make an argument that it's working as designed but people didn't anticipate the existence of timing exploits. I'd call that different from computation errors.
- o11c 2y agoHm, once again this lacks numbers on variation within a generation. How much applies to K vs non-K processors? How much between i3, i5, i7, and i9? Does it affect low-end 13th-generation-branded Alder Lake?
- forbiddenlake 2y ago> Starting with 10th Gen, we have only sold the top 2 SKUs (XX700K and XX900K) in volume, which gives us a nice clean set of data.
- cyanydeez 2y agoShould Intel be telling you these things?
- shrubble 2y agoOne of the alarming things about Intel's reaction is how tone-deaf they seem to have been during this entire process in terms of reassuring customers, especially gamer/enthusiast buyers. The Costco desktops in stock at the local Costco are all 13th/14th gen systems priced reasonably enough with Nvidia 4060 or 4070 cards... but my perception is they are not selling well because of the concerns raised about these CPUs.
- Dalewyn 2y agoIf they aren't selling well, that's probably because 14th gen (Raptor Lake Refresh) and especially 13th gen (Raptor Lake) are actually last gen old stock products. This is made even more obvious because Costco usually has all those "Core Ultra 1" (15th gen, Meteor Lake) branded laptops nearby. We've been on 15th gen (Meteor Lake) for quite a while now, with 16th gen (Arrow and Lunar Lakes) presumably coming later this year. I sympathize with Costco wanting to move all that old stock out now rather than later.
- wmf 2y agoFor one thing, Meteor Lake is not available for desktops so Raptor Lake is still the current generation there.
- shrubble 2y agoMicrocenter.com (US retailer) doesn't have anything newer than 14th gen CPUs for desktops, on their site. Did Intel release a newer desktop chip already?
- layer8 2y agoNo, the 15th gen desktop will be Arrow Lake-S aka Core Ultra 200, and is on track for release in Q4 2024.
- Dalewyn 2y agoNo, Arrow Lake is 16th gen. 15th gen (Meteor Lake) was supposed to also have a desktop segment, but that got axed because Intel(tm). No, I am not going to entertain the desktop/mobile generation split bullshit that AMD also has. Screw that noise, disjointed market segmentation is anti-consumer. (No, I am not going to entertain Intel's new Core XX branding either.)
- Dalewyn 2y agoI've had the impression that while this problem is definitely real, it's also suffering from very bad media sensationalism (both mainstream and social) and some very emotional chest thumping a la Boeing. Puget's numbers kind of vindicate that by showing 11th gen was even worse and AMD clearly benefitted from their "underdog", "cult favorite" status. It would be nice if we could be rid of most of this noise so we could get down to what truly matters.
- cyanydeez 2y agoThe problem isn't just the propaganda fight, it's that it's basically a heisenbug. Intel, a few months ago, blamed these very failures on motherboard manufacturers and claimed they were overlooking and the failures at that time were their fault. Unless you do a very thorough timeline, you might be confused. But go dig into the last 6 months and you'll see that Intel has either no idea or are absolutely muddying the waters. Neither of these cognitive conclusions should result in Intel looking anything but half rate purveyors of silicon.
- deleted 2y ago[deleted]
- stqism 2y agoIn a sense, they were partially right, while being wrong. Based on Puget’s data, it’s apparent that motherboard vendors overly aggressive default settings helped contribute to the issue being so prominent, when reasonable settings would fail at a lower rate than comparable zen CPUs. Obviously Intel messed up badly, and those settings shouldn’t result in this behavior, but maybe this will convince system integrators to have more reasonable defaults in the future. In a top end system, we’re already sitting in territory where our GPU is our benchmark, do we really need to default to giving the cpu so much power?
- jtriangle 2y agoThey didn't pull those numbers out of thin air, those were intel's specs when those boards were designed. They are, obviously, dangerous specs to run a chip at, hindsight being 2020 and all. Intel trying to pass the buck is as much of a problem as the CPU's themselves really, because now you can't trust them.
- cyanydeez 2y agoLove Puget. Keep up the great work.
- ljoshua 2y agoI second that. I've only had a chance to buy one PC from them (I usually run Macs), but the level of attention to detail and customer service from Puget was bar none the best I've had from any tech outfit.
- CamperBob2 2y agoI bought my 13900K box from them after an HN story a couple of years ago in which they openly talked about Samsung's failure to live up to their expectations, and basically apologized for advocating the Samsung 990 Pro drives in the past ( https://www.pugetsystems.com/blog/2023/02/02/update-on-samsung-ssd-reliability/ https://www.pugetsystems.com/blog/2023/02/02/update-on-samsu... ). Not one vendor in 100 would stand up and say something like that in public. The system I bought from them has been great, and if/when that changes, I don't doubt that they'll have my back when it comes to haggling with Intel.
- frognumber 2y agoI've only bought from them rarely, a long long time ago, but I was impressed. If money is no object -- or developer time is expensive -- or failure is expensive -- it'd be my go to source. For what I'm doing now (education), pricing is quite a bit too high.
- iforgotpassword 2y agoHm, what's interesting here is that now the blame does somewhat lie on the mainboard manufacturers and their stock overclock, mostly. Wendell in his video pointed out that even cloud gaming providers running server boards have the same high failure rates, while applying much more conservative settings.
- J_Shelby_J 2y agoHere are some instructions I've been sharing that has led me to stability. Download OCCT. https://www.ocbase.com/ https://www.ocbase.com/ disable xmp stress test with OCCT if it crashes here, you have to downclock your cpu enable xmp stress test with OCCT if it crashes here you need to downclock your memory speed repeat 5 and 6 in the smallest interval of ram speeds in your bios until it's stable for ~5m Now, I'm sharing this here with because it's good to know but also to make the point: do you think supermicro is doing this for every server that leaves their factory? Not to say there isn't an issue with Intel, but based on what the article says about failure rates, and what I see from friends with AMD, there is an issue with system stability in general that extends beyond this specific issue. My guess BIOS settings being ran at redline didn't do well when we rolled out the new DDR5.
- Dalewyn 2y agoXMP (and AMD's EXPO) is an overclock, you should not and can not expect stability out of the box where overclocks are concerned. This is regardless whatever the RAM vendor might tell you; an overclock is an overclock, it is literally running hardware out of specification. Both Intel and AMD publish their memory controller specs and you should thoroughly understand them if you do want to overclock (read: use XMP/EXPO), anything that goes above the specs is not guaranteed to work. Incidentally, for all the flak the motherboard vendors rightfully got with their out of the box overclocking defaults, their default configuration for RAM is in fact to stick to Intel/AMD specifications like superglue.
- Mathnerd314 2y agoI don't think it is as cut and dried as "you should not and can not expect stability out of the box". If it doesn't overclock, you RMA it, if the new one doesn't overclock either, either you are bad at overclocking or you got really unlucky in the silicon lottery. It just doesn't fit the facts to say that you shouldn't expect 5400 Mhz ram to work at 5400 Mhz, even if Intel says the spec is 4800 or whatever. Now 7200 Mhz, that is pushing it and is a lot more like snake oil.
- scionescio 2y ago[flagged]
- deleted 2y ago[deleted]
- layer8 2y agoTheir statistics does include field failures, meaning failures reported by their customers, and they do note that the degrading problem may yet worsen the statistics in the coming months. They also note that they are applying (as they always have) more conservative settings than the often more aggressive motherboard defaults that other users tend to be using. They aren’t excusing Intels failures, they are presenting the data they have on hand, which is more than I have seen elsewhere.
- scionescio 2y agoits not about excusing intel failures. BUT imho it is misleading in the context of the current discussion, if you also include statistics data of initial failed cpu, youre kinda polluting the pool. Thats what im criticizing here. it may not be intentional, but its definitely not helping. it distracts from the degrading issue if you also add data from DOA cases. in opposite other (not yet public?) data, where inital tests have been passed but subsequent tests not. ps. and it should be crystal clear that Intel has a massive interest in anything that distracts from the degrading issue. be it letting some youtuber blaming (already debunked) motherboardsvendors or possibly (tinfoil hat mode) by 'innocently' throwing statistics to he public that include other failures and pull competitors also into the spotlight, you know how detailed public looks at statistics. they see big bar labled with amd and say hey, amd fails too - that its a total different case, most will ignore.
- flyinghamster 2y ago> With Intel Core CPUs in particular, we pay close attention to voltage levels and time durations at which those levels are sustained. This has been especially challenging when those guidelines are difficult to find and when motherboard makers brand features with their own unique naming. (emphasis mine) That last little bit is a thing that has infuriated me for decades, and it's all over the place in anything that tech touches, not just motherboards. Can't we call things by their proper names?
- 1oooqooq 2y agowhat a surprise! CPU and GPU learned from decades of RAM business. Sell garbage. People buy, get blue screen, blame software. ECC? pffft. Let's just see how long it will take for NVMEs to fall to the same level of reliability as usb pen drives and everyone will just buy 4 of them to RAID1 and assume that's just "normal".
- deleted 2y ago[deleted]