6 ms·

I have a genuine question - in practice on a workstation/developer computer, what sort of protection does ECC ram give me? I've got two daily driver machines -

I have a genuine question - in practice on a workstation/developer computer, what sort of protection does ECC ram give me? I've got two daily driver machines - a 3970x threadripper with 96GB ram, and an M1 Macbook Pro - neither of which have ECC ram. I've been using them both for over 2 years, and not once in that period (that I'm aware of) have I found myself with a problem due to faulty RAM, but I do regularly find myself wishing both were faster. What practical benefit would I get in exchange for the performance hit of ECC memory?

frankreyes 3y ago
It depends on your workload. Some industries require to be able to reproduce exact bit by bit data. Anything that requires absolute bit certainty, for example digital signatures, encryption, or financial information, will require ECC. If you don't care if your files eventually don't match a sha256 checksum, you'll be fine. for example source code, it doesn't matter because in the worst case, your code won't compile because variable names got flipped. And even then, if you're using Git for source code control, you're already using hash signatures to detect data corruption, as well as a very large redundancy and replication. Git is, in a philosophical way, ECC by software.
- maccard 3y ago
  > It depends on your workload. > Some industries require to be able to reproduce exact bit by bit data. > Anything that requires absolute bit certainty, for example digital signatures, encryption, or financial information, will require ECC. Totally, and I get that. We've got people in this thread who are saying that non-ECC ram shouldn't be trusted [0], or that it should be standard [1]. I get the use cases, but why do I want that on my workstation. > your code won't compile because variable names got flipped. I have _never_ seen or even heard of this or anything vaguely resembling this happening. Are there any writeups of this anywhere at all? [0] https://news.ycombinator.com/item?id=37829796 https://news.ycombinator.com/item?id=37829796 [1] https://news.ycombinator.com/item?id=37828865 https://news.ycombinator.com/item?id=37828865
  - frankreyes 3y ago
    In my opinion yes, ECC should be standard. ECC is expensive just because Intel had a monopoly over the server segment, and unilaterally forced desktop not to use ECC. Otherwise people would use desktop computers as servers, and that would reduce profits. At work, my workstation Desktop computer has ECC. The colleague's sitting next to me has on their screen sometimes kernel warnings of ECC errors. And cases of bit flips happen every day, that's why some problems are solved by just restarting your computer or the software. I'd argue that many times we blame software bugs on pure hardware bit flips. Here's a story of a variable name bit flip: https://alexbakker.me/post/did-cosmic-rays-break-my-linux-build.html https://alexbakker.me/post/did-cosmic-rays-break-my-linux-bu... Bit flips usually affect operating system stuff, here's another story: https://blogs.oracle.com/linux/post/attack-of-the-cosmic-rays https://blogs.oracle.com/linux/post/attack-of-the-cosmic-ray...
theevilsharpie 3y ago
> what sort of protection does ECC ram give me? The ability to detect (and possibly correct) physical memory errors. > I've been using them both for over 2 years, and not once in that period (that I'm aware of) have I found myself with a problem due to faulty RAM The "that I'm aware of" part is the key issue. ECC provides visibility into the health of the physical memory that you otherwise wouldn't have. > What practical benefit would I get in exchange for the performance hit of ECC memory? Side-band ECC (which is what you'd use on desktops/workstations/servers) doesn't have a performance hit, as the overhead of ECC is fully neutralized by the additional bandwidth and capacity of an ECC DIMM.
- maccard 3y ago
  > The ability to detect (and possibly correct) physical memory errors. This is a great example of theoretical benefits (note I'm not saying that those theoretical benefits are real, I'm asking how they practically benefit me). > The "that I'm aware of" part is the key issue. Ok so tell me what it looks like. _That's_ what I want to know. > into the health of the physical memory that you otherwise wouldn't have. And does what, on my windows workstation or my linux workstation? > Side-band ECC (which is what you'd use on desktops/workstations/servers) doesn't have a performance hit, as the overhead of ECC is fully neutralized by the additional bandwidth and capacity of an ECC DIMM. That statement doesn't hold water. If it takes extra bandwidth and capacity to provide ECC, I can use the extra bandwidth as memory instead of error correction, no? (Or I could if the non-ecc DIMM utilised that range). Either way, it's an extra x bits for ECC that _could_ be used for storage.
  - theevilsharpie 3y ago
    > Ok so tell me what it looks like. _That's_ what I want to know. Suppose an application running on your machine suffers some type of malfunction -- a segmentation fault or a seemingly-random kernel panic that you're unable to reproduce.[1] And suppose it happens often enough that you want to fix it, and understand the root cause. You research the issue, and are pointed to potentially faulty hardware. That then raises a question: how do you know your physical memory is working properly? You could run a diagnostic application like Memtest86. However, that has the following issues: - It only tests a particular point in time. If the test "passes", that doesn't say whether your memory encountered a fault in the past, nor does it guarantee that it won't fault in the future. - What does it even mean for a test to "pass?" Just a single run through a test suite? Running it for some period of time, like a few hours? - A diagnostic like Memtest86 is invasive. While it's running, you can't use your machine for other things. Troubleshooting these types of hardware errors without ECC is tricky, because there's not really a way to conclusively link a software fault to a hardware error.[2][3] However, on a machine equipped with ECC, this type of troubleshooting is a lot more straight-forward. If bits are flipped in memory, the memory controller can probably detect that (assuming only only one or two bits are flipped), and can raise a machine exception that the OS can catch and do something with (e.g., logging the error, terminating the process impacted if the error is correctable, etc). That saves a lot of time and headache that you'd otherwise be spending on guesswork. You may still be asking, "how big of an issue is this, really?" I suppose whether or not you care about having hardware that can detect memory faults is up to you.[4] However, during the portion of my career where I was managing large machine fleets, memory failures were the second most common type of hardware failure, behind only mechanical disk drives. > That statement doesn't hold water. If it takes extra bandwidth and capacity to provide ECC, I can use the extra bandwidth as memory instead of error correction, no? No. The memory controller is unable to use the additional capacity for anything other than error checking (hence why it's called "side-band"). What you're describing is "in-band ECC", which is something that you sometimes see on GPUs or small form factor systems aimed at professional markets. --- [1] An application crash is actually one of the better outcomes of a memory error. The worst-case scenario is silent corruption of data stored in memory that your application assumes to be valid. [2] Errors as a result of faulty memory have been particularly frustrating to OS developers, as it can result in nonsense error reports. Linus Torvalds has bemoaned the lack of ECC options on consumer hardware, claiming that it was the industry cheaping out. During the lead-up to the launch of Windows Vista, Microsoft officially encouraged the use of ECC. I don't follow development of the BSDs, but it wouldn't surprise me if they similarly wished their users would use ECC across the board. [3] When Google first started building out the infrastructure for their server farms, they opted not to use ECC memory for their servers as a cost-savings measure. They subsequently had a memory fault on one their machines that resulted on corruption of their search index. While they worked around the problem by adding logic to verify the contents of the index, all subsequent generations of machines that Google has deployed have ECC support. For specifics, see the second footnote in https://danluu.com/why-ecc/ https://danluu.com/why-ecc/ [4] Note that many of your other components that store data in some way (e.g., caches, persistent storage, etc.) are likely to have ECC capability. The only notable exceptions I can think of are GPU memory, the main system memory on consumer hardware, and the processor's registers.
orangepurple 3y ago
When your RAM starts to go bad it can severely screw up writes to your filesystem without you noticing for a long time.
neolefty 3y ago
Anecdotally, on a desktop machine with something like 64GB RAM, when I switched from non-ECC to ECC (and 128GB RAM), the number of unexplained crashes seemed to drop from one every couple of weeks to one every 6 months or so. Could have been from a software upgrade though, probably Ubuntu 18 to Ubuntu 20. Now I'm on an M1 MBP with 32GB RAM, and I get unexplained crashes roughly every 3-4 months, but I've been chalking it up to MacOS. Behavior also seems to degrade after a month or two of uptime. Hmm, maybe it's time for Apple to go ECC ...
- maccard 3y ago
  > Anecdotally, on a desktop machine with something like 64GB RAM, when I switched from non-ECC to ECC (and 128GB RAM), the number of unexplained crashes seemed to drop from one every couple of weeks to one every 6 months or so. Could have been from a software upgrade though, probably Ubuntu 18 to Ubuntu 20. Thanks. This is a great anecdote/example that is much more what I'm interested in. I don't see the same level of system instability on my machine, but it's a good example. Appreciate it. > Behavior also seems to degrade after a month or two of uptime. Ah, here's another thing. I do tend to reboot daily/weekly which may help
tpurves 3y ago
Bit flips in memory can affect instruction code, not just data. This will manifest as random bluescreen crashes or visual artifacts in the case of video memory. Similar to errors/crashes from overclocking a little too high, but occurring occasionally on a normally stable system.

frankreyes 3y ago

It depends on your workload. Some industries require to be able to reproduce exact bit by bit data. Anything that requires absolute bit certainty, for example digital signatures, encryption, or financial information, will require ECC. If you don't care if your files eventually don't match a sha256 checksum, you'll be fine. for example source code, it doesn't matter because in the worst case, your code won't compile because variable names got flipped. And even then, if you're using Git for source code control, you're already using hash signatures to detect data corruption, as well as a very large redundancy and replication. Git is, in a philosophical way, ECC by software.

maccard 3y ago
> It depends on your workload. > Some industries require to be able to reproduce exact bit by bit data. > Anything that requires absolute bit certainty, for example digital signatures, encryption, or financial information, will require ECC. Totally, and I get that. We've got people in this thread who are saying that non-ECC ram shouldn't be trusted [0], or that it should be standard [1]. I get the use cases, but why do I want that on my workstation. > your code won't compile because variable names got flipped. I have _never_ seen or even heard of this or anything vaguely resembling this happening. Are there any writeups of this anywhere at all? [0] https://news.ycombinator.com/item?id=37829796 https://news.ycombinator.com/item?id=37829796 [1] https://news.ycombinator.com/item?id=37828865 https://news.ycombinator.com/item?id=37828865
- frankreyes 3y ago
  In my opinion yes, ECC should be standard. ECC is expensive just because Intel had a monopoly over the server segment, and unilaterally forced desktop not to use ECC. Otherwise people would use desktop computers as servers, and that would reduce profits. At work, my workstation Desktop computer has ECC. The colleague's sitting next to me has on their screen sometimes kernel warnings of ECC errors. And cases of bit flips happen every day, that's why some problems are solved by just restarting your computer or the software. I'd argue that many times we blame software bugs on pure hardware bit flips. Here's a story of a variable name bit flip: https://alexbakker.me/post/did-cosmic-rays-break-my-linux-build.html https://alexbakker.me/post/did-cosmic-rays-break-my-linux-bu... Bit flips usually affect operating system stuff, here's another story: https://blogs.oracle.com/linux/post/attack-of-the-cosmic-rays https://blogs.oracle.com/linux/post/attack-of-the-cosmic-ray...

theevilsharpie 3y ago

> what sort of protection does ECC ram give me? The ability to detect (and possibly correct) physical memory errors. > I've been using them both for over 2 years, and not once in that period (that I'm aware of) have I found myself with a problem due to faulty RAM The "that I'm aware of" part is the key issue. ECC provides visibility into the health of the physical memory that you otherwise wouldn't have. > What practical benefit would I get in exchange for the performance hit of ECC memory? Side-band ECC (which is what you'd use on desktops/workstations/servers) doesn't have a performance hit, as the overhead of ECC is fully neutralized by the additional bandwidth and capacity of an ECC DIMM.

maccard 3y ago
> The ability to detect (and possibly correct) physical memory errors. This is a great example of theoretical benefits (note I'm not saying that those theoretical benefits are real, I'm asking how they practically benefit me). > The "that I'm aware of" part is the key issue. Ok so tell me what it looks like. _That's_ what I want to know. > into the health of the physical memory that you otherwise wouldn't have. And does what, on my windows workstation or my linux workstation? > Side-band ECC (which is what you'd use on desktops/workstations/servers) doesn't have a performance hit, as the overhead of ECC is fully neutralized by the additional bandwidth and capacity of an ECC DIMM. That statement doesn't hold water. If it takes extra bandwidth and capacity to provide ECC, I can use the extra bandwidth as memory instead of error correction, no? (Or I could if the non-ecc DIMM utilised that range). Either way, it's an extra x bits for ECC that _could_ be used for storage.
- theevilsharpie 3y ago
  > Ok so tell me what it looks like. _That's_ what I want to know. Suppose an application running on your machine suffers some type of malfunction -- a segmentation fault or a seemingly-random kernel panic that you're unable to reproduce.[1] And suppose it happens often enough that you want to fix it, and understand the root cause. You research the issue, and are pointed to potentially faulty hardware. That then raises a question: how do you know your physical memory is working properly? You could run a diagnostic application like Memtest86. However, that has the following issues: - It only tests a particular point in time. If the test "passes", that doesn't say whether your memory encountered a fault in the past, nor does it guarantee that it won't fault in the future. - What does it even mean for a test to "pass?" Just a single run through a test suite? Running it for some period of time, like a few hours? - A diagnostic like Memtest86 is invasive. While it's running, you can't use your machine for other things. Troubleshooting these types of hardware errors without ECC is tricky, because there's not really a way to conclusively link a software fault to a hardware error.[2][3] However, on a machine equipped with ECC, this type of troubleshooting is a lot more straight-forward. If bits are flipped in memory, the memory controller can probably detect that (assuming only only one or two bits are flipped), and can raise a machine exception that the OS can catch and do something with (e.g., logging the error, terminating the process impacted if the error is correctable, etc). That saves a lot of time and headache that you'd otherwise be spending on guesswork. You may still be asking, "how big of an issue is this, really?" I suppose whether or not you care about having hardware that can detect memory faults is up to you.[4] However, during the portion of my career where I was managing large machine fleets, memory failures were the second most common type of hardware failure, behind only mechanical disk drives. > That statement doesn't hold water. If it takes extra bandwidth and capacity to provide ECC, I can use the extra bandwidth as memory instead of error correction, no? No. The memory controller is unable to use the additional capacity for anything other than error checking (hence why it's called "side-band"). What you're describing is "in-band ECC", which is something that you sometimes see on GPUs or small form factor systems aimed at professional markets. --- [1] An application crash is actually one of the better outcomes of a memory error. The worst-case scenario is silent corruption of data stored in memory that your application assumes to be valid. [2] Errors as a result of faulty memory have been particularly frustrating to OS developers, as it can result in nonsense error reports. Linus Torvalds has bemoaned the lack of ECC options on consumer hardware, claiming that it was the industry cheaping out. During the lead-up to the launch of Windows Vista, Microsoft officially encouraged the use of ECC. I don't follow development of the BSDs, but it wouldn't surprise me if they similarly wished their users would use ECC across the board. [3] When Google first started building out the infrastructure for their server farms, they opted not to use ECC memory for their servers as a cost-savings measure. They subsequently had a memory fault on one their machines that resulted on corruption of their search index. While they worked around the problem by adding logic to verify the contents of the index, all subsequent generations of machines that Google has deployed have ECC support. For specifics, see the second footnote in https://danluu.com/why-ecc/ https://danluu.com/why-ecc/ [4] Note that many of your other components that store data in some way (e.g., caches, persistent storage, etc.) are likely to have ECC capability. The only notable exceptions I can think of are GPU memory, the main system memory on consumer hardware, and the processor's registers.

orangepurple 3y ago

When your RAM starts to go bad it can severely screw up writes to your filesystem without you noticing for a long time.

neolefty 3y ago

Anecdotally, on a desktop machine with something like 64GB RAM, when I switched from non-ECC to ECC (and 128GB RAM), the number of unexplained crashes seemed to drop from one every couple of weeks to one every 6 months or so. Could have been from a software upgrade though, probably Ubuntu 18 to Ubuntu 20. Now I'm on an M1 MBP with 32GB RAM, and I get unexplained crashes roughly every 3-4 months, but I've been chalking it up to MacOS. Behavior also seems to degrade after a month or two of uptime. Hmm, maybe it's time for Apple to go ECC ...

maccard 3y ago
> Anecdotally, on a desktop machine with something like 64GB RAM, when I switched from non-ECC to ECC (and 128GB RAM), the number of unexplained crashes seemed to drop from one every couple of weeks to one every 6 months or so. Could have been from a software upgrade though, probably Ubuntu 18 to Ubuntu 20. Thanks. This is a great anecdote/example that is much more what I'm interested in. I don't see the same level of system instability on my machine, but it's a good example. Appreciate it. > Behavior also seems to degrade after a month or two of uptime. Ah, here's another thing. I do tend to reboot daily/weekly which may help

tpurves 3y ago

Bit flips in memory can affect instruction code, not just data. This will manifest as random bluescreen crashes or visual artifacts in the case of video memory. Similar to errors/crashes from overclocking a little too high, but occurring occasionally on a normally stable system.