12 ms·

ECC RAM on AMD Ryzen 7000 Desktop CPUs

344 points by mmastrac 3y ago

tiffanyh 3y ago
> ”Unfortunately, when the AMD Ryzen 7000 ‘Raphael’ CPUs were launched along with the brand new Socket AM5, all mention of ECC support was gone.” Had no idea that 7000s series doesn’t official state support of ECC.
- sunshowers 3y ago
  After publishing this post, I was alerted to the fact that there are some AM5 motherboards that do officially support ECC. ASRock's Rack line supports it, for example: https://www.asrockrack.com/general/productdetail.asp?Model=1U4LW-B650/2L2T#Specifications https://www.asrockrack.com/general/productdetail.asp?Model=1... This ASUS motherboard also claims to support ECC: https://www.asus.com/us/motherboards-components/motherboards/proart/proart-x670e-creator-wifi/techspec/ https://www.asus.com/us/motherboards-components/motherboards... Neither of those were out at the time I originally got my AM5 motherboard (which was right as they came out -- the performance numbers were so good I couldn't help but spring on one early.)
  - sysGlenn 3y ago
    The Asrock Racks, 1U4LW-B650/2L2T, 10g ports appear to be supported by VMWare ESXI 8: https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=43270 https://www.vmware.com/resources/compatibility/detail.php?de... I would be interested in hearing someones experience with getting this board running as a type 1 ESXI hypervisor.
- scheme271 3y ago
  It was really iffy and not clearly mentioned until this year. Also, it's confused with on-chip ECC which all DDR5 uses. DDR5 needs on chip ECC to correct errors that'll occur in normal operation but that won't provide ECC for transfers across the memory bus to the CPU.
- theevilsharpie 3y ago
  > Had no idea that 7000s series doesn’t official state support of ECC. You've misinterpreted the author. All currently-available Ryzen 7000 series CPUs have official ECC support, but it requires motherboard support as well.[1] This conditional ECC support has been the case for all past AMD consumer CPUs going back to the original Athlon 64, but the Ryzen 7000 series is the first time I can recall that this support has been explicitly listed on AMD's marketing materials. What the author is saying is that mention of ECC support had disappeared from ASRock's motherboard documentation. This was a notable change for ASRock, as they had explicitly mentioned ECC support on their previous Ryzen motherboards. [1] Example from the Ryzen 5 7600 spec page: https://www.amd.com/en/product/12756#:~:text=ECC%20Support,Requires%20mobo%20support https://www.amd.com/en/product/12756#:~:text=ECC%20Support,R...)
jadbox 3y ago
I am just immensely impressed by this story. A+ hardware and software investigation work.
- sunshowers 3y ago
  Thank you for the kind words!
  - PeterStuer 3y ago
    Though I have no intention of switching to ECC on my Ryzen system, I found this writeup both very clear and a great compelling educational read. Thank you for creating it.
- tonoto 3y ago
  Yes, a great article! It was no bigger surprise to see that the article was written by an Oxide employee - it has been a joyful signature that they've managed to attract curious people with an engineering approach to investigate problems and look up how the things work. I'm probably biased, but it seem like a dream company to me.
Scramblejams 3y ago
I don’t quite have the courage to physically short pins, nor the patience to slowly overclock my RAM, waiting multiple minutes for DDR5 link training each time. So instead, I’m content with knowing that the memory controller is reporting that ECC is enabled. How about hitting the RAM with a warm stream of air from a hair dryer? I've seen that technique used in the past to generate errors.
- willis936 3y ago
  I feel like potentially reflowing BGAs on a high speed I/O PCBA you want to run continuously for a decade is riskier than simply shorting a pair of diode-protected pins. Link training can be disabled in the BIOS, which would make searching for borderline bandwidth settings a quick operation. The results would not be very repeatable, but that's not important.
  - mlyle 3y ago
    > I feel like potentially reflowing BGAs on a high speed I/O PCBA y If you're trying to get chips up to something less than 100C, but instead get stuff underneath it with significant thermal mass up to above 200C, you're really messing up. Hair dryers tend to emit like 65C-70C air on their "high heat" setting (as opposed to hot air guns used for heat-shrink and reflow soldering).
  - 15155 3y ago
    SAC305 solder won't reflow until >230C vapor phase. Will a hair dryer get a chip to these temps, ever?
- latchkey 3y ago
  Boot the OS off ram and then do something that uses a lot of ram. If the machine crashes randomly or you have a corrupted "disk", chances are it is an ECC error.
- Tuna-Fish 3y ago
  More sensibly, you can do some memory overclocking live on a running system, with no link training. It's not advisable for actual use, but you can use it to find the limits of your memory, or to generate errors.
renewiltord 3y ago
I think we have AsrockRack motherboards with Ryzen 7950x and ECC RAM. Works well.
mlsu 3y ago
I am in need of a processor upgrade, but am really interested in an ECC RAM setup. I read this thread of 2 people arguing on /r/AMD about whether or not the AMD processors and/or their motherboards actually support ECC or not. And had no clue if they were right. [1] So this is topical for me! [1] https://www.reddit.com/r/Amd/comments/lzxqod/list_of_am4_motherboards_that_support_ecc/gq6ry64/?context=10000 https://www.reddit.com/r/Amd/comments/lzxqod/list_of_am4_mot... So this definitively settles it, that the AMD+ASRock combo is truly ECC RAM?
- bick_nyers 3y ago
  I use ECC on my X570 and B550 ASRock boards, they have allowed unbuffered ECC for a while. Kind of a shame they don't have a mini-itx/matx X670E board (only ASUS does).
  - SteveNuts 3y ago
    Supermicro has an interesting AM5 motherboard which I'm planning to start building a workstation around https://www.supermicro.com/en/products/motherboard/h13sae-mf https://www.supermicro.com/en/products/motherboard/h13sae-mf
    - bick_nyers 3y ago
      Oooh, nice find. Now to find some 5200MHz (or more) Unbuffered ECC dimms...
      - dvasdekis 3y ago
        This is all I could find: https://www.mouser.com/ProductDetail/Kingston/KSM56E46BD8KM-32HA?qs=17ckDYBRdek6DFSrgOcN4w%3D%3D&_gl=1*1r7p1p0*_ga*MTg1NTA0MTYwNC4xNjk2OTE1MTc2*_ga_15W4STQT4T*MTY5NjkxNTE3Ni4xLjEuMTY5NjkxNTIzOC41OC4wLjA https://www.mouser.com/ProductDetail/Kingston/KSM56E46BD8KM-....
      - sunshowers 3y ago
        The v-color RAM I linked to in the post is available at up to 5600MHz (that's what I got): https://v-color.net/products/ddr5-ecc-udimm-servermemory?variant=43445581906087 https://v-color.net/products/ddr5-ecc-udimm-servermemory?var...
Modified3019 3y ago
See also: https://forum.level1techs.com/t/ecc-capable-verified-motherboards-for-ryzen-7000/ https://forum.level1techs.com/t/ecc-capable-verified-motherb... >"I have ECC working on ASUS ProArt X670E-Creator with AMD Ryzen 9 7950X. But, you have to explicitly turn ECC on in the bios. If left on ‘Auto’, it will be off. I use four sticks of Supermicro (Hynix) 32GB 288-Pin DDR5 4800 (PC5-38400) Server Memory (MEM-DR532MD-EU48)." https://www.reddit.com/r/truenas/comments/10lqofy/ecc_support_for_am5_motherboards/ https://www.reddit.com/r/truenas/comments/10lqofy/ecc_suppor... <Read yourself>
- deleted 3y ago
  [deleted]
snvzz 3y ago
If only if legislators could get their shit together and mandate ECC. It is unnerving most computing is done in fragile non-ecc systems. A very messed up form of artificial market segmentation.
- jjeaff 3y ago
  I hear this all the time, that non-ecc machines are just completely fragile, and the math seems to indicate that one should be getting bit flips all the time. and yet, my Intel system has 64gb of non-ecc ram and it runs half the day everyday and is hibernated at night. I run 3d cad software on it, Photoshop, vs code with tons of extensions and software running in wsl2 docker containers and I just never really get any errors. certainly no crashing or blue screen. what exactly should one expect from bitflip errors? I would think that even single bitflips as often as they should happen with 64gb of ram being used almost fully would show up somehow.
  - justinclift 3y ago
    > what exactly should one expect from bitflip errors? Probably silently wrong values happening in memory somewhere. If it's not in something where that value is not actively running code code, you probably won't have a crash. If it's in (say) Excel calculation formula it'll probably just screw up the calculation. Which may or may not be obvious. Similar thing for 3D cad, it could be completely non-obvious.
- declan_roberts 3y ago
  Fragile non-ecc systems? Something that works absolutely fine 99.9999% of the time is not fragile.
  - cwaffles 3y ago
    Probability of a bit error increases as memory increases. Opens a large can of worms if control flow breaks.
    - 3y ago
aportnoy 3y ago
> But the lack of ECC was a huge bummer at the time of purchasing my system. Why?..
- dharmab 3y ago
  Ever had to troubleshoot bit flips on a non-ECC system? One friend felt like he was going crazy as over the course of two months his system degraded from occasional random errors to random crashes, blue screens and finally to no POST. Another time, a coworker had to stare at raw bytestreams in Wireshark for hours to find a consistently flipped bit.
  - ceeam 3y ago
    > Ever had to troubleshoot but flips on a non-ECC system? No. > One friend felt like he was going crazy Tell him about memtest86.
    - aportnoy 3y ago
      Wow I came back to post this exact reply. I set my system to a slightly high frequency, ran memtest overnight with errors. Set it back down to a supported frequency, ran a full memtest suite again with no errors. Never had any issues since.
      - nirvdrum 3y ago
        You likely wouldn't know if you had random bit flips. It'd manifest as silent data corruption. You might be okay with that. Others aren't. It's not a matter of overclocking. Bit flips are a fact of life running with 32+ GB RAM. Leaving your machine on 24/7 (even if in sleep) stacks the odds against you.
        gambiting 3y ago
        Obviously this is just anecdote but I have a work laptop with 128GB of non-ECC ram , use all of it every day and never noticed any issues. I'm not saying there aren't any, but it just....works.
YetAnotherNick 3y ago
> ApplesOfEpicness did so by shorting a data and ground pin on their motherboard. This doesn't seem to be the proof that ECC is working. Error detection on things like this is different than bit flip correction.
- type0 3y ago
  It is the proof, on unregistered sticks that's the job of your software, so it does the bit flip correction.
mixmastamyk 3y ago
Recently decided not to buy the AMD laptop from framework because they didn't bother to implement ECC. Started me thinking that I don't really need a new laptop, have an old one... so why continue to suffer from compromised parts at my desk? Unfortunately AMD has been going the wrong way with this. Oddly enough, Intel has quietly added a few cheaper models with ECC support. Unlike the author I don't have time for building a new PC and a bunch of maybes. I found this machine that's pretty cheap: https://www.dell.com/en-us/shop/desktop-computers/precision-3660-tower/spd/precision-3660-workstation https://www.dell.com/en-us/shop/desktop-computers/precision-... This one has a CPU option with a TDP of 65 W. What do y'all think?
- nrp 3y ago
  It’s not a question of whether we want it or not. You’ll have to take it up with AMD on enablement of ECC support on non-PRO U-series mobile processors.
  - kingsleyopara 3y ago
    Thanks for clarifying. That said, if you did want it, couldn’t you just offer a PRO series option e.g. AMD Ryzen 7 PRO 7840U?
  - mixmastamyk 3y ago
    Do we not deserve pro parts? Just a bunch of amateurs over here at HN? ;-) I suspect the group that wants to physically maintain their laptop but doesn’t care about data integrity isn’t too large.
- speed_spread 3y ago
  I haven't looked but I suspect not many laptops support ECC, if any. That's not a good reason to skip AMD. The Dell may support ECC, but as any workstation it's pricier than consumer grade desktop with equivalent performance. If you need it, you need it but usually individuals don't pay for that, their employers do.
  - 3y ago
ChuckMcM 3y ago
An excellent read. I have ECC RAM on my Threadripper board. One of the things that ops at Blekko found was that you needed to tell Intel boards (NOTE vendor change!) to actually REPORT correctable errors. The default was machine-check if there was an unrecoverable error, otherwise just go with the flow. With ~ 1600 192GB systems we would see correctable errors about once a week as I recall. I don't recall a single unrecoverable error in 6 years so that was pretty good. Greg will surely correct me if I missed one :-).
- toast0 3y ago
  Your luck was better than mine, but at least our systems reported correctable errors without us having to ask. We had a roughly similar amount of machines, probably about similar ram on average (many with less, and some with much more) We definitely had unrecoverable errors from time to time; maybe one or two a year, enough to have a policy: give it a shot to see if it was a one off, if it doesn't fail again shortly after it's probably fine; if it does fail again shortly after, swap the ram. Nicer server boards even have a led to tell you which ram module to replace. For correctable errors, we wouldn't swap unless the error counts got pretty big; some systems went for a long time with one or two errors a day, which is fine. Others went zero errors for a long time, then a couple days at a small number, and then big numbers. There was one system that managed to get thousands per hour and the system was unusable because handling machine check exceptions was too expensive; unfortunately the reporting interval was one hour, so we didn't realize why the system was unusable until the next report.
  - ChuckMcM 3y ago
    Well to be fair if we started regularly getting UCE's on a stick of memory we did preemptively replace it. (where regularly was more frequently than one a week)
    - toast0 3y ago
      Yeah, I'd guess a good number of the systems where we got one UCE, we got another one within a day after reboot. If so, off to repair. If not, it was pretty rare for that machine to cause trouble again; seemed worth the risk to check if repair was needed. The correctables were trickier to judge, because system halt on UCE is easy to recover from and easy to diagnose (system is down, check console, see UCE message); system slow as heck because of constant machine check exceptions is hard to diagnose and a slow system can disrupt a distributed system a lot more than a dead system.
tripdout 3y ago
Looking to get an ASRock B650 Pro RS to go with a 7700X, hopefully ECC will be supported.
brucethemoose2 3y ago
Oh, I just happened to grab a B650E-ITX! ...But are there any ECC sticks with "known" good Hynix ICs? I got lucky with a pair of sticks that can do DDR5 6000 CL30 at 1.3V (and maybe better), and I wouldn't want to sacrifice a ton of RAM performance for ECC.
- sunshowers 3y ago
  The v-color RAM I linked to in the post is Hynix: https://v-color.net/products/ddr5-ecc-udimm-servermemory?variant=43445581906087 https://v-color.net/products/ddr5-ecc-udimm-servermemory?var... I haven't tried overclocking it.
  - brucethemoose2 3y ago
    That is A-Die! It should overclock like crazy. You should do it and post it. Ryzen 7000 likes OC'd RAM, and overclocked ECC ram is kind of a desktop hardware myth/legend because its so rare and oxymoronic. Not sure if you've done it before, but DDR5 overclocking is quite a rabbit hole, and there are some "easy" tutorials out there like this: https://www.youtube.com/watch?v=dlYxmRcdLVw https://www.youtube.com/watch?v=dlYxmRcdLVw Test the stock primary timings with the modded subtimings at 6000 first, then jump straight to cl34 for the primary timings and reduce the voltage/primary timings from there. I would grab these myself, but the kit is too rich for my blood at the moment.
    - sunshowers 3y ago
      Fine, you've convinced me! I'll try it next week. To be honest the only RAM overclocking I've done is by going into the XMP settings and turning on the shipped overclock configuration. For CPUs I tend to undervolt them a bit to reduce power consumption.
      - naagral 3y ago
        I'm also interested about what speeds and CL would that DIMM run. By the way I'm the one who started that ASRock forum topic, what a small world to randomly stumble here while scrolling HN. In the end I put PC upgrade on hold, partly because of that AMD bios ECC support mess and also cannot find fast DIMM-s. Just looking today it seems ASRock has still not re-added ECC support info on AM5 board specs pages so the more people test those boards and meomory combinations the better for community.
foresto 3y ago
Some (maybe all?) ASUS AM5 motherboards have official ECC support. It appears in both the board manual and the BIOS manual for the model I just checked. Note that one of the relevant BIOS settings is Auto by default, which (counterintuitively) leaves it disabled, so you'll want to change that. ECC reporting for these processors appeared in Linux 6.5, so Debian Stable users will have to either wait for it to appear in Backports or stray from the beaten path. https://www.phoronix.com/news/AMD-EDAC-Ryzen-7000-Series https://www.phoronix.com/news/AMD-EDAC-Ryzen-7000-Series
- bpye 3y ago
  > Note that one of the relevant BIOS settings is Auto by default, which (counterintuitively) leaves it disabled, so you'll want to change that. I really dislike Auto settings in the BIOS. It wouldn't be so bad if there was a way to see the effective value, but 9 times out of 10 - it's no obvious.
  - foresto 3y ago
    I don't mind auto settings themselves, but I agree about the opaque ones. In this case, the effective value is stated in the field's help message, but it's still easy to miss.
Arnavion 3y ago
I used ECC with a ThreadRipper 2920x on an Asrock motherboard, but for a new Ryzen 7950x build I went back to regular RAM. The problem with ECC with the TR was that only Unbuffered DIMMs work, not Registered DIMMS, and the 7950x is the same. Unbuffered ECC is slower or costlier (or both) than the equivalent non-ECC or Registered, enough that it just wasn't worth it in a new build. Ryzen has the additional disadvantage vs TR of only being able to run RAM fast if you limit yourself to two sticks. My TR build used 4x16GiB but for my 7950x build I had to get 2x32GiB to be able to run them at 6000MHz.
- nirvdrum 3y ago
  I currently run a 3700X with 64 GB ECC. I've been interested in a jump to an AM5 series, but have found the matter of ECC compatibility frustrating to research, particularly because DDR5 has limited ECC built in and that confuses the discussion. I'm curious what your use case is. I mostly use mine for software development (including VMs) and some family media processing. I like the peace of mind that ECC brings, but I have no way to really know if RAM speed is performance factor.
  - Arnavion 3y ago
    Personal Linux machine mostly used for watching video, code compiling, and some light gaming. The 7950x definitely compiles my stuff much faster than the TR 2920x did, but I don't know how much of that is because of the RAM speed (6000MHz vs 4800MHz, faster timings) and how much because the CPU clocks higher (~5.4GHz vs ~4.2GHz).
    - Arnavion 3y ago
      (Made a typo. The RAM speed in my 2920x build was 2666MHz, not 4800MHz, of course.)
- vetinari 3y ago
  > Ryzen has the additional disadvantage vs TR of only being able to run RAM fast if you limit yourself to two sticks. TR has it similar, just doubled; you are able to run 4 sticks faster than 8. On my TR2920X+X399 (probably the same config as yours, Asrock Taichi) I'm able to run any four of them at 3066 MHz and all of them only at 2933 MHz.
harshreality 3y ago
There's a rumor that older AGESA[1] versions had a bug that prevented the chipset from recognizing and utilizing ECC ram properly even though the chipsets should support it. Check any motherboard in question for a firmware update which includes at least AGESA 1.0.0.5 patch C. https://www.reddit.com/r/truenas/comments/10lqofy/ https://www.reddit.com/r/truenas/comments/10lqofy/ [1] Part of the firmware on AMD systems, that brings up core system components. https://en.wikipedia.org/wiki/AGESA https://en.wikipedia.org/wiki/AGESA
- sunshowers 3y ago
  Based on the ASRock forum thread, this sounds right. Before installing the ECC RAM I updated to AGESA 1.0.0.7b.
Roark66 3y ago
I'm currently on Ryzen 3700X and asus tuf gaming x570 MB. I'd love more single core performance and more nvme/disk speed. Also I have a dual GPU setup, 2 M2 nvme, and 6 sata occupied. I'm considering upgrading end of the year. I briefly considered thread ripper (for it's pcie lanes) until I found there is still no zen4 thread ripper and the price is likely yo be eye-watering. So the choice is, upgrade to Ryzen 5900x and keep the rest of the system, or spend a lot more and upgrade to a AM5 ryzen plus a new mobo(I considered Intel too, until I found it tops at 20 pcie lanes). I've always been buying mostly gigabyte and asus mobos, but I got burned more than once by them so I might go with another manufacturer this time. What do you think? Is it worth upgrading to AM5 for just single core and disk performance? What about Intel? Considering I'd love more pcie lanes for a 10gb adapter perhaps(so I can move some of these spinning disks out) Intel is probably not a good choice. I'm happy enough with the multi-core performance of ryzen 3700x. If I got a new MB I'd definitely want at least two nvmes in a mirror(for speed) - perhaps more and sata ports for my 6 spinning disks.
- Crabstove 3y ago
  I have 5900X and I wish I’ve waited for 5800X3D instead.
  - Havoc 3y ago
    What’s the rationale for that? Gaming?
    - beebeepka 3y ago
      Very likely. I jumped from 3700x and the difference is staggering. My 6800xt came alive and stutters are a thing of the past. I kind of want an OLED but they don't come cheap, vertical resolution is the same, and they are not amazing for coding. Maybe in time for a Zen 5 based 3d cache chip.
      - Havoc 3y ago
        >My 6800xt came alive and stutters are a thing of the past. Potentially because of resizable BAR. 3** CPUs didn't support it, 5** did.
_cenw 3y ago
The ASRock Rack B650D4U advertises ECC support on AM5. It's a bit pricy, but you also get IPMI.
demurgos 3y ago
I've been actually researching this a lot recently as I'm in the process of buying parts for a desktop build and want ECC support. > Unfortunately, when the AMD Ryzen 7000 “Raphael” CPUs were launched along with the brand new Socket AM5, all mention of ECC support was gone. 1. This was effectively a big point of concern for me. Previous ASRock Taichi motherboards officially advertised full ECC support, but not the latest X670E for AM5 model. Older version of the X670E Taichi page mentioned ECC support ("Supports DDR5 ECC/non-ECC, un-buffered memory up to 6600+(OC)") [0], and this Level1Tech review [1] had a segment on ECC support (they fully tested it). The segment was around the 10min mark, but it looks like the video was swapped to remove it during the last 30 days (the original video was 17min41sec long, the current one is only 16min32sec). So I assumed that ECC was supported, but it's a bit concerning that there's no official mention. The article reassures me. 2. Finding ECC memory for desktop computers is hard. For the X670E Taichi, you need DDR5 unregistered DIMM. The best solution I have to find ECC memory is to check QVL lists for similar server motherboards, for example this ASRock Rack [2] or this Gigabyte Motherboard [3]. I decided to go with two 32GiB Kingston sticks [4] for my own build. [0]: https://web.archive.org/web/20221002103925/https://www.asrock.com/mb/AMD/X670E%20Taichi/index.asp#Specification https://web.archive.org/web/20221002103925/https://www.asroc... [1]: https://www.youtube.com/watch?v=PhrqEV-VAjE https://www.youtube.com/watch?v=PhrqEV-VAjE [2]: https://www.asrockrack.com/general/productdetail.asp?Model=1U4LW-B650/2L2T#Memory https://www.asrockrack.com/general/productdetail.asp?Model=1... [3]: https://download.gigabyte.com/FileList/QVL/server_mb_qvl_MC13-LE0_LE1_v1.1.pdf?v=f850244d52af30a6bc90e9ec81ffaf7c https://download.gigabyte.com/FileList/QVL/server_mb_qvl_MC1... [4]: https://www.kingston.com/en/memory/server-premier/ddr5-4800mts-ecc-unbuffered-dimm https://www.kingston.com/en/memory/server-premier/ddr5-4800m...
c0l0 3y ago
Maybe slightly OT, since this concerns AMD older AM4 platform with a Zen3 APU core, but working ECC support looks like this and is definitely present on my system: $ sudo ras-mc-ctl --errors | tail -n5 14 2023-08-20 20:16:41 +0200 error: Corrected error, no action required., CPU 2, bank Unified Memory Controller (bank=17), mcg mcgstatus=0, mci CECC, memory_channel=0,csrow=0, mcgcap=0x0000011c, status=0x9c2040000000011b, addr=0x36e701dc0, misc=0xd01a000101000000, walltime=0x64e31c78, cpuid=0x00a50f00, bank=0x00000011 15 2023-08-23 17:17:49 +0200 error: Corrected error, no action required., CPU 2, bank Unified Memory Controller (bank=17), mcg mcgstatus=0, mci CECC, memory_channel=0,csrow=0, mcgcap=0x0000011c, status=0x9c2040000000011b, addr=0x36e701dc0, misc=0xd01a000101000000, walltime=0x64ea5188, cpuid=0x00a50f00, bank=0x00000011 16 2023-09-03 16:52:15 +0200 error: Corrected error, no action required., CPU 2, bank Unified Memory Controller (bank=17), mcg mcgstatus=0, mci CECC, memory_channel=0,csrow=0, mcgcap=0x0000011c, status=0x9c2040000000011b, addr=0x36e701dc0, misc=0xd01a000101000000, walltime=0x64f4d227, cpuid=0x00a50f00, bank=0x00000011 17 2023-09-15 21:37:59 +0200 error: Corrected error, no action required., CPU 2, bank Unified Memory Controller (bank=17), mcg mcgstatus=0, mci CECC, memory_channel=0,csrow=0, mcgcap=0x0000011c, status=0x9c2040000000011b, addr=0x36e701dc0, misc=0xd01a000101000000, walltime=0x65071ed7, cpuid=0x00a50f00, bank=0x00000011 This is with an ASRock B550M-ITX/ac and a AMD Ryzen 5 PRO 5650G. It used to work the same with a Ryzen 5 3600 (using a dedicated GPU for video output) before I upgraded the CPU. To detect and log ECC activity on modern GNU/Linux, you will want to have the "rasdaemon" service active. I will decode MCE (and other hardware-related) errors and persist them to the database that is shown being queried above.
- jacquesm 3y ago
  The frequency of these errors should be enough to cause you to distrust any kind of information output by a computer without ECC. Edit: thinking about this a bit longer: that frequency is actually so high that you may well have a broken module in there. Note how it is the same module and the same address every time.
  - code_biologist 3y ago
    It'd be cute to log what processes are using that memory at time of error. Fun to speculate about whether a kernel bit flip is better or worse than ones in a web browser, photo editor, spreadsheet, network storage client...
justinclift 3y ago
As a data point, Hetzner has offered Ryzen 7000 CPU's with ECC ram for months: https://www.hetzner.com/dedicated-rootserver/matrix-ax https://www.hetzner.com/dedicated-rootserver/matrix-ax The AX52 has optional ECC ram as an upgrade, while the AX102 comes with ECC by default. It'd be surprising if they'd been offering ECC that didn't really work.
- pingiun 3y ago
  I think they make their own motherboards (have them made). So they can fully make sure that ECC is supported from start to finish
TacticalCoder 3y ago
> The most foolproof way to test whether ECC is working is to introduce an error somehow. > > ApplesOfEpicness did so by shorting a data and ground pin on their motherboard. Epic! Some people, really... > I don’t quite have the courage to physically short pins, nor the patience to slowly overclock my RAM, waiting multiple minutes for DDR5 link training each time It's not clear to me: when is the DDR5 re-trained? I assembled my Ryzen 7000 desktop PC myself and I don't remember ever having to wait minutes for it to boot, not even the first time. It's a bit slower at boot I'd say than non-DDR5 PCs I had in the past but it's mere seconds, not minutes. And last but not least: DDR5 specs for ECC vs non-ECC, which price difference and speed difference are to be expected?
- wongarsu 3y ago
  > which price difference and speed difference are to be expected ECC RAM uses 1/8th more memory chips, so as a rule of thumb it costs 1/8th more. They can have very different dynamics on the used market though, because servers tend to be upgraded to larger modules over time, but there aren't many buyers for smaller modules. Not a big factor for DDR5 because it's too new, but for DDR4 you can get some great deals for 8GB and 16GB modules.
maccard 3y ago
I have a genuine question - in practice on a workstation/developer computer, what sort of protection does ECC ram give me? I've got two daily driver machines - a 3970x threadripper with 96GB ram, and an M1 Macbook Pro - neither of which have ECC ram. I've been using them both for over 2 years, and not once in that period (that I'm aware of) have I found myself with a problem due to faulty RAM, but I do regularly find myself wishing both were faster. What practical benefit would I get in exchange for the performance hit of ECC memory?
- frankreyes 3y ago
  It depends on your workload. Some industries require to be able to reproduce exact bit by bit data. Anything that requires absolute bit certainty, for example digital signatures, encryption, or financial information, will require ECC. If you don't care if your files eventually don't match a sha256 checksum, you'll be fine. for example source code, it doesn't matter because in the worst case, your code won't compile because variable names got flipped. And even then, if you're using Git for source code control, you're already using hash signatures to detect data corruption, as well as a very large redundancy and replication. Git is, in a philosophical way, ECC by software.
  - maccard 3y ago
    > It depends on your workload. > Some industries require to be able to reproduce exact bit by bit data. > Anything that requires absolute bit certainty, for example digital signatures, encryption, or financial information, will require ECC. Totally, and I get that. We've got people in this thread who are saying that non-ECC ram shouldn't be trusted [0], or that it should be standard [1]. I get the use cases, but why do I want that on my workstation. > your code won't compile because variable names got flipped. I have _never_ seen or even heard of this or anything vaguely resembling this happening. Are there any writeups of this anywhere at all? [0] https://news.ycombinator.com/item?id=37829796 https://news.ycombinator.com/item?id=37829796 [1] https://news.ycombinator.com/item?id=37828865 https://news.ycombinator.com/item?id=37828865
    - 3y ago
marginalia_nu 3y ago
I feel the need for ECC is a bit overstated, probably confused by listening too much to people in the server space. In servers, ECC is critical because servers simply have more RAM than PCs do. You can easily find systems with 32 sticks of RAM. If you roll 32 dice, odds of a rare freak error is a lot bigger than if you roll 2. It's the same reason why you probably don't need RAID on a computer with one or two disks. Stick 48 disks in a machine however, and you're a fool to go without it (or a plan at least). It's why you don't need an automatic fire suppression system in or homelab, because the odds of a fire in one or a few PSUs is fairly small, but stick a few thousands in a room and suddenly it's a very real problem. That, and most memory problems in PCs tend to be from dodgy overclocking or just bad sticks rather than cosmic rays. ECC won't really save you from that.
- Vogtinator 3y ago
  Counterpoint: Without ECC, you simply don't notice how many flips happened and that they were the cause of some strange behaviour.
  - marginalia_nu 3y ago
    Most of them simply don't matter though, and are inconsequential along other system instabilities. It's just neurotic to worry about issues when you couldn't tell they existed without being informed of them.
    - ilyt 3y ago
      Most issues with car don't matter for driving, till it catches on fire or crashes due to catastrophic failure. Why ever go to mechanic ?
      - marginalia_nu 3y ago
        Most car crashes just aren't due to mechanical failure.
breakwaterlabs 3y ago
This is both a little off-topic and a little out of my expertise, but shouldn't the file descriptors in the query function be closed?
benhoff 3y ago
This is a bit off topic, but I'm running Home Assistant on a system that doesn't have any ECC. I'll reboot it occasionally. Every once in awhile it will exhibit odd behavior that seems to resolve on reboot. Is it worth getting something that has ECC? I think I'm running like a Sandy Bridge i7 for reference.
- semi 3y ago
  it's not really possible to say if ecc would prevent the issue you're describing.. but that's why imo it is worth having ecc--so you don't have to worry as much about if the weird behavior you see is due to random memory errors. On the more immediately practical side.. if this is happening frequently enough then running memtest86 or GSAT google stressful application test and see if it can pick up any errors. You also might be able to improve things with a bios upgrade, or by lowering the clock speed of the ram.

344 points by mmastrac 3y ago

> ”Unfortunately, when the AMD Ryzen 7000 ‘Raphael’ CPUs were launched along with the brand new Socket AM5, all mention of ECC support was gone.” Had no idea that 7000s series doesn’t official state support of ECC.

sunshowers 3y ago
After publishing this post, I was alerted to the fact that there are some AM5 motherboards that do officially support ECC. ASRock's Rack line supports it, for example: https://www.asrockrack.com/general/productdetail.asp?Model=1U4LW-B650/2L2T#Specifications https://www.asrockrack.com/general/productdetail.asp?Model=1... This ASUS motherboard also claims to support ECC: https://www.asus.com/us/motherboards-components/motherboards/proart/proart-x670e-creator-wifi/techspec/ https://www.asus.com/us/motherboards-components/motherboards... Neither of those were out at the time I originally got my AM5 motherboard (which was right as they came out -- the performance numbers were so good I couldn't help but spring on one early.)
- sysGlenn 3y ago
  The Asrock Racks, 1U4LW-B650/2L2T, 10g ports appear to be supported by VMWare ESXI 8: https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=43270 https://www.vmware.com/resources/compatibility/detail.php?de... I would be interested in hearing someones experience with getting this board running as a type 1 ESXI hypervisor.
scheme271 3y ago
It was really iffy and not clearly mentioned until this year. Also, it's confused with on-chip ECC which all DDR5 uses. DDR5 needs on chip ECC to correct errors that'll occur in normal operation but that won't provide ECC for transfers across the memory bus to the CPU.
theevilsharpie 3y ago
> Had no idea that 7000s series doesn’t official state support of ECC. You've misinterpreted the author. All currently-available Ryzen 7000 series CPUs have official ECC support, but it requires motherboard support as well.[1] This conditional ECC support has been the case for all past AMD consumer CPUs going back to the original Athlon 64, but the Ryzen 7000 series is the first time I can recall that this support has been explicitly listed on AMD's marketing materials. What the author is saying is that mention of ECC support had disappeared from ASRock's motherboard documentation. This was a notable change for ASRock, as they had explicitly mentioned ECC support on their previous Ryzen motherboards. [1] Example from the Ryzen 5 7600 spec page: https://www.amd.com/en/product/12756#:~:text=ECC%20Support,Requires%20mobo%20support https://www.amd.com/en/product/12756#:~:text=ECC%20Support,R...)

jadbox 3y ago

I am just immensely impressed by this story. A+ hardware and software investigation work.

sunshowers 3y ago
Thank you for the kind words!
- PeterStuer 3y ago
  Though I have no intention of switching to ECC on my Ryzen system, I found this writeup both very clear and a great compelling educational read. Thank you for creating it.
tonoto 3y ago
Yes, a great article! It was no bigger surprise to see that the article was written by an Oxide employee - it has been a joyful signature that they've managed to attract curious people with an engineering approach to investigate problems and look up how the things work. I'm probably biased, but it seem like a dream company to me.

Scramblejams 3y ago

I don’t quite have the courage to physically short pins, nor the patience to slowly overclock my RAM, waiting multiple minutes for DDR5 link training each time. So instead, I’m content with knowing that the memory controller is reporting that ECC is enabled. How about hitting the RAM with a warm stream of air from a hair dryer? I've seen that technique used in the past to generate errors.

willis936 3y ago
I feel like potentially reflowing BGAs on a high speed I/O PCBA you want to run continuously for a decade is riskier than simply shorting a pair of diode-protected pins. Link training can be disabled in the BIOS, which would make searching for borderline bandwidth settings a quick operation. The results would not be very repeatable, but that's not important.
- mlyle 3y ago
  > I feel like potentially reflowing BGAs on a high speed I/O PCBA y If you're trying to get chips up to something less than 100C, but instead get stuff underneath it with significant thermal mass up to above 200C, you're really messing up. Hair dryers tend to emit like 65C-70C air on their "high heat" setting (as opposed to hot air guns used for heat-shrink and reflow soldering).
- 15155 3y ago
  SAC305 solder won't reflow until >230C vapor phase. Will a hair dryer get a chip to these temps, ever?
latchkey 3y ago
Boot the OS off ram and then do something that uses a lot of ram. If the machine crashes randomly or you have a corrupted "disk", chances are it is an ECC error.
Tuna-Fish 3y ago
More sensibly, you can do some memory overclocking live on a running system, with no link training. It's not advisable for actual use, but you can use it to find the limits of your memory, or to generate errors.

renewiltord 3y ago

I think we have AsrockRack motherboards with Ryzen 7950x and ECC RAM. Works well.

mlsu 3y ago

I am in need of a processor upgrade, but am really interested in an ECC RAM setup. I read this thread of 2 people arguing on /r/AMD about whether or not the AMD processors and/or their motherboards actually support ECC or not. And had no clue if they were right. [1] So this is topical for me! [1] https://www.reddit.com/r/Amd/comments/lzxqod/list_of_am4_motherboards_that_support_ecc/gq6ry64/?context=10000 https://www.reddit.com/r/Amd/comments/lzxqod/list_of_am4_mot... So this definitively settles it, that the AMD+ASRock combo is truly ECC RAM?

bick_nyers 3y ago
I use ECC on my X570 and B550 ASRock boards, they have allowed unbuffered ECC for a while. Kind of a shame they don't have a mini-itx/matx X670E board (only ASUS does).
- SteveNuts 3y ago
  Supermicro has an interesting AM5 motherboard which I'm planning to start building a workstation around https://www.supermicro.com/en/products/motherboard/h13sae-mf https://www.supermicro.com/en/products/motherboard/h13sae-mf
  - bick_nyers 3y ago
    Oooh, nice find. Now to find some 5200MHz (or more) Unbuffered ECC dimms...
    - dvasdekis 3y ago
      This is all I could find: https://www.mouser.com/ProductDetail/Kingston/KSM56E46BD8KM-32HA?qs=17ckDYBRdek6DFSrgOcN4w%3D%3D&_gl=1*1r7p1p0*_ga*MTg1NTA0MTYwNC4xNjk2OTE1MTc2*_ga_15W4STQT4T*MTY5NjkxNTE3Ni4xLjEuMTY5NjkxNTIzOC41OC4wLjA https://www.mouser.com/ProductDetail/Kingston/KSM56E46BD8KM-....
    - sunshowers 3y ago
      The v-color RAM I linked to in the post is available at up to 5600MHz (that's what I got): https://v-color.net/products/ddr5-ecc-udimm-servermemory?variant=43445581906087 https://v-color.net/products/ddr5-ecc-udimm-servermemory?var...

Modified3019 3y ago

See also: https://forum.level1techs.com/t/ecc-capable-verified-motherboards-for-ryzen-7000/ https://forum.level1techs.com/t/ecc-capable-verified-motherb... >"I have ECC working on ASUS ProArt X670E-Creator with AMD Ryzen 9 7950X. But, you have to explicitly turn ECC on in the bios. If left on ‘Auto’, it will be off. I use four sticks of Supermicro (Hynix) 32GB 288-Pin DDR5 4800 (PC5-38400) Server Memory (MEM-DR532MD-EU48)." https://www.reddit.com/r/truenas/comments/10lqofy/ecc_support_for_am5_motherboards/ https://www.reddit.com/r/truenas/comments/10lqofy/ecc_suppor... <Read yourself>

deleted 3y ago
[deleted]

snvzz 3y ago

If only if legislators could get their shit together and mandate ECC. It is unnerving most computing is done in fragile non-ecc systems. A very messed up form of artificial market segmentation.

jjeaff 3y ago
I hear this all the time, that non-ecc machines are just completely fragile, and the math seems to indicate that one should be getting bit flips all the time. and yet, my Intel system has 64gb of non-ecc ram and it runs half the day everyday and is hibernated at night. I run 3d cad software on it, Photoshop, vs code with tons of extensions and software running in wsl2 docker containers and I just never really get any errors. certainly no crashing or blue screen. what exactly should one expect from bitflip errors? I would think that even single bitflips as often as they should happen with 64gb of ram being used almost fully would show up somehow.
- justinclift 3y ago
  > what exactly should one expect from bitflip errors? Probably silently wrong values happening in memory somewhere. If it's not in something where that value is not actively running code code, you probably won't have a crash. If it's in (say) Excel calculation formula it'll probably just screw up the calculation. Which may or may not be obvious. Similar thing for 3D cad, it could be completely non-obvious.
declan_roberts 3y ago
Fragile non-ecc systems? Something that works absolutely fine 99.9999% of the time is not fragile.
- cwaffles 3y ago
  Probability of a bit error increases as memory increases. Opens a large can of worms if control flow breaks.
  - 3y ago

aportnoy 3y ago

> But the lack of ECC was a huge bummer at the time of purchasing my system. Why?..

dharmab 3y ago
Ever had to troubleshoot bit flips on a non-ECC system? One friend felt like he was going crazy as over the course of two months his system degraded from occasional random errors to random crashes, blue screens and finally to no POST. Another time, a coworker had to stare at raw bytestreams in Wireshark for hours to find a consistently flipped bit.
- ceeam 3y ago
  > Ever had to troubleshoot but flips on a non-ECC system? No. > One friend felt like he was going crazy Tell him about memtest86.
  - aportnoy 3y ago
    Wow I came back to post this exact reply. I set my system to a slightly high frequency, ran memtest overnight with errors. Set it back down to a supported frequency, ran a full memtest suite again with no errors. Never had any issues since.
    - nirvdrum 3y ago
      You likely wouldn't know if you had random bit flips. It'd manifest as silent data corruption. You might be okay with that. Others aren't. It's not a matter of overclocking. Bit flips are a fact of life running with 32+ GB RAM. Leaving your machine on 24/7 (even if in sleep) stacks the odds against you.
      - gambiting 3y ago
        Obviously this is just anecdote but I have a work laptop with 128GB of non-ECC ram , use all of it every day and never noticed any issues. I'm not saying there aren't any, but it just....works.

YetAnotherNick 3y ago

> ApplesOfEpicness did so by shorting a data and ground pin on their motherboard. This doesn't seem to be the proof that ECC is working. Error detection on things like this is different than bit flip correction.

type0 3y ago
It is the proof, on unregistered sticks that's the job of your software, so it does the bit flip correction.

mixmastamyk 3y ago

Recently decided not to buy the AMD laptop from framework because they didn't bother to implement ECC. Started me thinking that I don't really need a new laptop, have an old one... so why continue to suffer from compromised parts at my desk? Unfortunately AMD has been going the wrong way with this. Oddly enough, Intel has quietly added a few cheaper models with ECC support. Unlike the author I don't have time for building a new PC and a bunch of maybes. I found this machine that's pretty cheap: https://www.dell.com/en-us/shop/desktop-computers/precision-3660-tower/spd/precision-3660-workstation https://www.dell.com/en-us/shop/desktop-computers/precision-... This one has a CPU option with a TDP of 65 W. What do y'all think?

nrp 3y ago
It’s not a question of whether we want it or not. You’ll have to take it up with AMD on enablement of ECC support on non-PRO U-series mobile processors.
- kingsleyopara 3y ago
  Thanks for clarifying. That said, if you did want it, couldn’t you just offer a PRO series option e.g. AMD Ryzen 7 PRO 7840U?
- mixmastamyk 3y ago
  Do we not deserve pro parts? Just a bunch of amateurs over here at HN? ;-) I suspect the group that wants to physically maintain their laptop but doesn’t care about data integrity isn’t too large.
speed_spread 3y ago
I haven't looked but I suspect not many laptops support ECC, if any. That's not a good reason to skip AMD. The Dell may support ECC, but as any workstation it's pricier than consumer grade desktop with equivalent performance. If you need it, you need it but usually individuals don't pay for that, their employers do.
- 3y ago

ChuckMcM 3y ago

An excellent read. I have ECC RAM on my Threadripper board. One of the things that ops at Blekko found was that you needed to tell Intel boards (NOTE vendor change!) to actually REPORT correctable errors. The default was machine-check if there was an unrecoverable error, otherwise just go with the flow. With ~ 1600 192GB systems we would see correctable errors about once a week as I recall. I don't recall a single unrecoverable error in 6 years so that was pretty good. Greg will surely correct me if I missed one :-).

toast0 3y ago
Your luck was better than mine, but at least our systems reported correctable errors without us having to ask. We had a roughly similar amount of machines, probably about similar ram on average (many with less, and some with much more) We definitely had unrecoverable errors from time to time; maybe one or two a year, enough to have a policy: give it a shot to see if it was a one off, if it doesn't fail again shortly after it's probably fine; if it does fail again shortly after, swap the ram. Nicer server boards even have a led to tell you which ram module to replace. For correctable errors, we wouldn't swap unless the error counts got pretty big; some systems went for a long time with one or two errors a day, which is fine. Others went zero errors for a long time, then a couple days at a small number, and then big numbers. There was one system that managed to get thousands per hour and the system was unusable because handling machine check exceptions was too expensive; unfortunately the reporting interval was one hour, so we didn't realize why the system was unusable until the next report.
- ChuckMcM 3y ago
  Well to be fair if we started regularly getting UCE's on a stick of memory we did preemptively replace it. (where regularly was more frequently than one a week)
  - toast0 3y ago
    Yeah, I'd guess a good number of the systems where we got one UCE, we got another one within a day after reboot. If so, off to repair. If not, it was pretty rare for that machine to cause trouble again; seemed worth the risk to check if repair was needed. The correctables were trickier to judge, because system halt on UCE is easy to recover from and easy to diagnose (system is down, check console, see UCE message); system slow as heck because of constant machine check exceptions is hard to diagnose and a slow system can disrupt a distributed system a lot more than a dead system.

tripdout 3y ago

Looking to get an ASRock B650 Pro RS to go with a 7700X, hopefully ECC will be supported.

brucethemoose2 3y ago

Oh, I just happened to grab a B650E-ITX! ...But are there any ECC sticks with "known" good Hynix ICs? I got lucky with a pair of sticks that can do DDR5 6000 CL30 at 1.3V (and maybe better), and I wouldn't want to sacrifice a ton of RAM performance for ECC.

sunshowers 3y ago
The v-color RAM I linked to in the post is Hynix: https://v-color.net/products/ddr5-ecc-udimm-servermemory?variant=43445581906087 https://v-color.net/products/ddr5-ecc-udimm-servermemory?var... I haven't tried overclocking it.
- brucethemoose2 3y ago
  That is A-Die! It should overclock like crazy. You should do it and post it. Ryzen 7000 likes OC'd RAM, and overclocked ECC ram is kind of a desktop hardware myth/legend because its so rare and oxymoronic. Not sure if you've done it before, but DDR5 overclocking is quite a rabbit hole, and there are some "easy" tutorials out there like this: https://www.youtube.com/watch?v=dlYxmRcdLVw https://www.youtube.com/watch?v=dlYxmRcdLVw Test the stock primary timings with the modded subtimings at 6000 first, then jump straight to cl34 for the primary timings and reduce the voltage/primary timings from there. I would grab these myself, but the kit is too rich for my blood at the moment.
  - sunshowers 3y ago
    Fine, you've convinced me! I'll try it next week. To be honest the only RAM overclocking I've done is by going into the XMP settings and turning on the shipped overclock configuration. For CPUs I tend to undervolt them a bit to reduce power consumption.
    - naagral 3y ago
      I'm also interested about what speeds and CL would that DIMM run. By the way I'm the one who started that ASRock forum topic, what a small world to randomly stumble here while scrolling HN. In the end I put PC upgrade on hold, partly because of that AMD bios ECC support mess and also cannot find fast DIMM-s. Just looking today it seems ASRock has still not re-added ECC support info on AM5 board specs pages so the more people test those boards and meomory combinations the better for community.

foresto 3y ago

Some (maybe all?) ASUS AM5 motherboards have official ECC support. It appears in both the board manual and the BIOS manual for the model I just checked. Note that one of the relevant BIOS settings is Auto by default, which (counterintuitively) leaves it disabled, so you'll want to change that. ECC reporting for these processors appeared in Linux 6.5, so Debian Stable users will have to either wait for it to appear in Backports or stray from the beaten path. https://www.phoronix.com/news/AMD-EDAC-Ryzen-7000-Series https://www.phoronix.com/news/AMD-EDAC-Ryzen-7000-Series

bpye 3y ago
> Note that one of the relevant BIOS settings is Auto by default, which (counterintuitively) leaves it disabled, so you'll want to change that. I really dislike Auto settings in the BIOS. It wouldn't be so bad if there was a way to see the effective value, but 9 times out of 10 - it's no obvious.
- foresto 3y ago
  I don't mind auto settings themselves, but I agree about the opaque ones. In this case, the effective value is stated in the field's help message, but it's still easy to miss.

Arnavion 3y ago

I used ECC with a ThreadRipper 2920x on an Asrock motherboard, but for a new Ryzen 7950x build I went back to regular RAM. The problem with ECC with the TR was that only Unbuffered DIMMs work, not Registered DIMMS, and the 7950x is the same. Unbuffered ECC is slower or costlier (or both) than the equivalent non-ECC or Registered, enough that it just wasn't worth it in a new build. Ryzen has the additional disadvantage vs TR of only being able to run RAM fast if you limit yourself to two sticks. My TR build used 4x16GiB but for my 7950x build I had to get 2x32GiB to be able to run them at 6000MHz.

nirvdrum 3y ago
I currently run a 3700X with 64 GB ECC. I've been interested in a jump to an AM5 series, but have found the matter of ECC compatibility frustrating to research, particularly because DDR5 has limited ECC built in and that confuses the discussion. I'm curious what your use case is. I mostly use mine for software development (including VMs) and some family media processing. I like the peace of mind that ECC brings, but I have no way to really know if RAM speed is performance factor.
- Arnavion 3y ago
  Personal Linux machine mostly used for watching video, code compiling, and some light gaming. The 7950x definitely compiles my stuff much faster than the TR 2920x did, but I don't know how much of that is because of the RAM speed (6000MHz vs 4800MHz, faster timings) and how much because the CPU clocks higher (~5.4GHz vs ~4.2GHz).
  - Arnavion 3y ago
    (Made a typo. The RAM speed in my 2920x build was 2666MHz, not 4800MHz, of course.)
vetinari 3y ago
> Ryzen has the additional disadvantage vs TR of only being able to run RAM fast if you limit yourself to two sticks. TR has it similar, just doubled; you are able to run 4 sticks faster than 8. On my TR2920X+X399 (probably the same config as yours, Asrock Taichi) I'm able to run any four of them at 3066 MHz and all of them only at 2933 MHz.

harshreality 3y ago

There's a rumor that older AGESA[1] versions had a bug that prevented the chipset from recognizing and utilizing ECC ram properly even though the chipsets should support it. Check any motherboard in question for a firmware update which includes at least AGESA 1.0.0.5 patch C. https://www.reddit.com/r/truenas/comments/10lqofy/ https://www.reddit.com/r/truenas/comments/10lqofy/ [1] Part of the firmware on AMD systems, that brings up core system components. https://en.wikipedia.org/wiki/AGESA https://en.wikipedia.org/wiki/AGESA

sunshowers 3y ago
Based on the ASRock forum thread, this sounds right. Before installing the ECC RAM I updated to AGESA 1.0.0.7b.

Roark66 3y ago

I'm currently on Ryzen 3700X and asus tuf gaming x570 MB. I'd love more single core performance and more nvme/disk speed. Also I have a dual GPU setup, 2 M2 nvme, and 6 sata occupied. I'm considering upgrading end of the year. I briefly considered thread ripper (for it's pcie lanes) until I found there is still no zen4 thread ripper and the price is likely yo be eye-watering. So the choice is, upgrade to Ryzen 5900x and keep the rest of the system, or spend a lot more and upgrade to a AM5 ryzen plus a new mobo(I considered Intel too, until I found it tops at 20 pcie lanes). I've always been buying mostly gigabyte and asus mobos, but I got burned more than once by them so I might go with another manufacturer this time. What do you think? Is it worth upgrading to AM5 for just single core and disk performance? What about Intel? Considering I'd love more pcie lanes for a 10gb adapter perhaps(so I can move some of these spinning disks out) Intel is probably not a good choice. I'm happy enough with the multi-core performance of ryzen 3700x. If I got a new MB I'd definitely want at least two nvmes in a mirror(for speed) - perhaps more and sata ports for my 6 spinning disks.

Crabstove 3y ago
I have 5900X and I wish I’ve waited for 5800X3D instead.
- Havoc 3y ago
  What’s the rationale for that? Gaming?
  - beebeepka 3y ago
    Very likely. I jumped from 3700x and the difference is staggering. My 6800xt came alive and stutters are a thing of the past. I kind of want an OLED but they don't come cheap, vertical resolution is the same, and they are not amazing for coding. Maybe in time for a Zen 5 based 3d cache chip.
    - Havoc 3y ago
      >My 6800xt came alive and stutters are a thing of the past. Potentially because of resizable BAR. 3** CPUs didn't support it, 5** did.

_cenw 3y ago

The ASRock Rack B650D4U advertises ECC support on AM5. It's a bit pricy, but you also get IPMI.

demurgos 3y ago

I've been actually researching this a lot recently as I'm in the process of buying parts for a desktop build and want ECC support. > Unfortunately, when the AMD Ryzen 7000 “Raphael” CPUs were launched along with the brand new Socket AM5, all mention of ECC support was gone. 1. This was effectively a big point of concern for me. Previous ASRock Taichi motherboards officially advertised full ECC support, but not the latest X670E for AM5 model. Older version of the X670E Taichi page mentioned ECC support ("Supports DDR5 ECC/non-ECC, un-buffered memory up to 6600+(OC)") [0], and this Level1Tech review [1] had a segment on ECC support (they fully tested it). The segment was around the 10min mark, but it looks like the video was swapped to remove it during the last 30 days (the original video was 17min41sec long, the current one is only 16min32sec). So I assumed that ECC was supported, but it's a bit concerning that there's no official mention. The article reassures me. 2. Finding ECC memory for desktop computers is hard. For the X670E Taichi, you need DDR5 unregistered DIMM. The best solution I have to find ECC memory is to check QVL lists for similar server motherboards, for example this ASRock Rack [2] or this Gigabyte Motherboard [3]. I decided to go with two 32GiB Kingston sticks [4] for my own build. [0]: https://web.archive.org/web/20221002103925/https://www.asrock.com/mb/AMD/X670E%20Taichi/index.asp#Specification https://web.archive.org/web/20221002103925/https://www.asroc... [1]: https://www.youtube.com/watch?v=PhrqEV-VAjE https://www.youtube.com/watch?v=PhrqEV-VAjE [2]: https://www.asrockrack.com/general/productdetail.asp?Model=1U4LW-B650/2L2T#Memory https://www.asrockrack.com/general/productdetail.asp?Model=1... [3]: https://download.gigabyte.com/FileList/QVL/server_mb_qvl_MC13-LE0_LE1_v1.1.pdf?v=f850244d52af30a6bc90e9ec81ffaf7c https://download.gigabyte.com/FileList/QVL/server_mb_qvl_MC1... [4]: https://www.kingston.com/en/memory/server-premier/ddr5-4800mts-ecc-unbuffered-dimm https://www.kingston.com/en/memory/server-premier/ddr5-4800m...

c0l0 3y ago

Maybe slightly OT, since this concerns AMD older AM4 platform with a Zen3 APU core, but working ECC support looks like this and is definitely present on my system: $ sudo ras-mc-ctl --errors | tail -n5 14 2023-08-20 20:16:41 +0200 error: Corrected error, no action required., CPU 2, bank Unified Memory Controller (bank=17), mcg mcgstatus=0, mci CECC, memory_channel=0,csrow=0, mcgcap=0x0000011c, status=0x9c2040000000011b, addr=0x36e701dc0, misc=0xd01a000101000000, walltime=0x64e31c78, cpuid=0x00a50f00, bank=0x00000011 15 2023-08-23 17:17:49 +0200 error: Corrected error, no action required., CPU 2, bank Unified Memory Controller (bank=17), mcg mcgstatus=0, mci CECC, memory_channel=0,csrow=0, mcgcap=0x0000011c, status=0x9c2040000000011b, addr=0x36e701dc0, misc=0xd01a000101000000, walltime=0x64ea5188, cpuid=0x00a50f00, bank=0x00000011 16 2023-09-03 16:52:15 +0200 error: Corrected error, no action required., CPU 2, bank Unified Memory Controller (bank=17), mcg mcgstatus=0, mci CECC, memory_channel=0,csrow=0, mcgcap=0x0000011c, status=0x9c2040000000011b, addr=0x36e701dc0, misc=0xd01a000101000000, walltime=0x64f4d227, cpuid=0x00a50f00, bank=0x00000011 17 2023-09-15 21:37:59 +0200 error: Corrected error, no action required., CPU 2, bank Unified Memory Controller (bank=17), mcg mcgstatus=0, mci CECC, memory_channel=0,csrow=0, mcgcap=0x0000011c, status=0x9c2040000000011b, addr=0x36e701dc0, misc=0xd01a000101000000, walltime=0x65071ed7, cpuid=0x00a50f00, bank=0x00000011 This is with an ASRock B550M-ITX/ac and a AMD Ryzen 5 PRO 5650G. It used to work the same with a Ryzen 5 3600 (using a dedicated GPU for video output) before I upgraded the CPU. To detect and log ECC activity on modern GNU/Linux, you will want to have the "rasdaemon" service active. I will decode MCE (and other hardware-related) errors and persist them to the database that is shown being queried above.

jacquesm 3y ago
The frequency of these errors should be enough to cause you to distrust any kind of information output by a computer without ECC. Edit: thinking about this a bit longer: that frequency is actually so high that you may well have a broken module in there. Note how it is the same module and the same address every time.
- code_biologist 3y ago
  It'd be cute to log what processes are using that memory at time of error. Fun to speculate about whether a kernel bit flip is better or worse than ones in a web browser, photo editor, spreadsheet, network storage client...

justinclift 3y ago

As a data point, Hetzner has offered Ryzen 7000 CPU's with ECC ram for months: https://www.hetzner.com/dedicated-rootserver/matrix-ax https://www.hetzner.com/dedicated-rootserver/matrix-ax The AX52 has optional ECC ram as an upgrade, while the AX102 comes with ECC by default. It'd be surprising if they'd been offering ECC that didn't really work.

pingiun 3y ago
I think they make their own motherboards (have them made). So they can fully make sure that ECC is supported from start to finish

TacticalCoder 3y ago

> The most foolproof way to test whether ECC is working is to introduce an error somehow. > > ApplesOfEpicness did so by shorting a data and ground pin on their motherboard. Epic! Some people, really... > I don’t quite have the courage to physically short pins, nor the patience to slowly overclock my RAM, waiting multiple minutes for DDR5 link training each time It's not clear to me: when is the DDR5 re-trained? I assembled my Ryzen 7000 desktop PC myself and I don't remember ever having to wait minutes for it to boot, not even the first time. It's a bit slower at boot I'd say than non-DDR5 PCs I had in the past but it's mere seconds, not minutes. And last but not least: DDR5 specs for ECC vs non-ECC, which price difference and speed difference are to be expected?

wongarsu 3y ago
> which price difference and speed difference are to be expected ECC RAM uses 1/8th more memory chips, so as a rule of thumb it costs 1/8th more. They can have very different dynamics on the used market though, because servers tend to be upgraded to larger modules over time, but there aren't many buyers for smaller modules. Not a big factor for DDR5 because it's too new, but for DDR4 you can get some great deals for 8GB and 16GB modules.

maccard 3y ago

I have a genuine question - in practice on a workstation/developer computer, what sort of protection does ECC ram give me? I've got two daily driver machines - a 3970x threadripper with 96GB ram, and an M1 Macbook Pro - neither of which have ECC ram. I've been using them both for over 2 years, and not once in that period (that I'm aware of) have I found myself with a problem due to faulty RAM, but I do regularly find myself wishing both were faster. What practical benefit would I get in exchange for the performance hit of ECC memory?

frankreyes 3y ago
It depends on your workload. Some industries require to be able to reproduce exact bit by bit data. Anything that requires absolute bit certainty, for example digital signatures, encryption, or financial information, will require ECC. If you don't care if your files eventually don't match a sha256 checksum, you'll be fine. for example source code, it doesn't matter because in the worst case, your code won't compile because variable names got flipped. And even then, if you're using Git for source code control, you're already using hash signatures to detect data corruption, as well as a very large redundancy and replication. Git is, in a philosophical way, ECC by software.
- maccard 3y ago
  > It depends on your workload. > Some industries require to be able to reproduce exact bit by bit data. > Anything that requires absolute bit certainty, for example digital signatures, encryption, or financial information, will require ECC. Totally, and I get that. We've got people in this thread who are saying that non-ECC ram shouldn't be trusted [0], or that it should be standard [1]. I get the use cases, but why do I want that on my workstation. > your code won't compile because variable names got flipped. I have _never_ seen or even heard of this or anything vaguely resembling this happening. Are there any writeups of this anywhere at all? [0] https://news.ycombinator.com/item?id=37829796 https://news.ycombinator.com/item?id=37829796 [1] https://news.ycombinator.com/item?id=37828865 https://news.ycombinator.com/item?id=37828865
  - 3y ago

marginalia_nu 3y ago

I feel the need for ECC is a bit overstated, probably confused by listening too much to people in the server space. In servers, ECC is critical because servers simply have more RAM than PCs do. You can easily find systems with 32 sticks of RAM. If you roll 32 dice, odds of a rare freak error is a lot bigger than if you roll 2. It's the same reason why you probably don't need RAID on a computer with one or two disks. Stick 48 disks in a machine however, and you're a fool to go without it (or a plan at least). It's why you don't need an automatic fire suppression system in or homelab, because the odds of a fire in one or a few PSUs is fairly small, but stick a few thousands in a room and suddenly it's a very real problem. That, and most memory problems in PCs tend to be from dodgy overclocking or just bad sticks rather than cosmic rays. ECC won't really save you from that.

Vogtinator 3y ago
Counterpoint: Without ECC, you simply don't notice how many flips happened and that they were the cause of some strange behaviour.
- marginalia_nu 3y ago
  Most of them simply don't matter though, and are inconsequential along other system instabilities. It's just neurotic to worry about issues when you couldn't tell they existed without being informed of them.
  - ilyt 3y ago
    Most issues with car don't matter for driving, till it catches on fire or crashes due to catastrophic failure. Why ever go to mechanic ?
    - marginalia_nu 3y ago
      Most car crashes just aren't due to mechanical failure.

breakwaterlabs 3y ago

This is both a little off-topic and a little out of my expertise, but shouldn't the file descriptors in the query function be closed?

benhoff 3y ago

This is a bit off topic, but I'm running Home Assistant on a system that doesn't have any ECC. I'll reboot it occasionally. Every once in awhile it will exhibit odd behavior that seems to resolve on reboot. Is it worth getting something that has ECC? I think I'm running like a Sandy Bridge i7 for reference.

semi 3y ago
it's not really possible to say if ecc would prevent the issue you're describing.. but that's why imo it is worth having ecc--so you don't have to worry as much about if the weird behavior you see is due to random memory errors. On the more immediately practical side.. if this is happening frequently enough then running memtest86 or GSAT google stressful application test and see if it can pick up any errors. You also might be able to improve things with a bios upgrade, or by lowering the clock speed of the ram.