10 ms·

I wish regulators would step in to enforce ECC in consumer devices. Otherwise, organizations such as Microsoft (windows readyness logo) or RISC-V (application

by snvzz 3y ago

I wish regulators would step in to enforce ECC in consumer devices. Otherwise, organizations such as Microsoft (windows readyness logo) or RISC-V (application profiles such as RVA23) could make it a requirement. It is clear that we can't just rely on the hardware vendors to do it, and practically nobody should be running computers w/o ECC.

School-Cotton 3y ago
> practically nobody should be running computers w/o ECC Why? I’ve done so my whole life with zero problems.
- frognumber 3y ago
  I envy you. A life with zero problems.... I think you're probably the only person in the universe so fortunate. Here's the basic set of problems: 1) ECC is almost free if done at scale (e.g. every computer). The only reason it costs extra is that it is only done for high-paying customers and in limited volume. 2) Computer crashes cost lives. If e.g. your doctor's device has an issue mid-surgery, you potentially pay. 3) It's good for the economy. In either case, the cost savings of not having a call center rep say "Wait a minute. My computer just crashed." far outweighs the costs. 4) It's critical for resilience. Disasters (solar flares, EMP, etc.) are rare, but if every computer destroys data, we have a systemic problem. 5) The issue primarily is one of transparency. Most people don't realize it's an issue, and don't shop on it. 6) That's not to mention dual-market uses. E.g. if we need to ramp up military production, medical production, etc. next crisis, our infrastructure should be ready for it. Classical economic theory states government interventions do well for disaster resilience, for externalized costs, and for lack of transparency. Those are all places where markets fail. That means this should be regulated. We should have error correction in all storage and in all computation. Best intervention, though, isn't a ban but a tax. Sales tax should be on a sliding scale from 1-10%, with: - 1% for parts / support / service manuals being available for some extended period - 1% for all software being open-source - 1% for all hardware being open - 1% for use of standard, interchangeable parts (e.g. batteries, power connectors, etc.) - 1% for environmental factors (CO2, toxic materials, etc.) - 1% for reusable infrastructure (e.g. being able to repurpose manufacturing equipment for crisis use) - 1% for workplace factors (e.g reasonable benefits, audits/compliance, non-discrimination for target populations like ex-convicts, etc.) ... and so on Those sorts of gentle incentives accomplish the same thing via market mechanisms without limiting freedom. For a commodity product, a 10% edge far outweighs costs. If Apple wants to make a proprietary magnetic screen connector and save 1mm of space, a 1% extra tax won't even slow them down.
  - eptcyka 3y ago
    ECC is more costly in terms of the BoM and higher power consumption.
    - adrian_b 3y ago
      The power consumption is higher by an amount that is not easily detectable. The power consumed by the relatively few gates that generate and verify the ECC code is dwarfed by the power consumption needed to transport bits over the PCB traces. The extra memory cell arrays for ECC bits add some power consumption for refreshing their content and the extra PCB traces for the ECC bits add some power consumption, but this extra power is in the milliwatt range, so it is negligible in comparison with the CPU and GPU power even in a small laptop, not only in a desktop. Most gaming desktops use overclocked memory modules that have an extra power consumption many times greater than the DDR5-4800 or DDR5-5600 ECC UDIMMs. The ECC UDIMMs never need heatsinks, while many of the commonly used overclocked DIMMs will overheat without heatsinks. So the claim that using ECC will increase the power consumption is completely bogus. Something like using a power-save profile instead of a high-performance profile, for clock management, or using JEDEC timings for memory, instead of high-performance timings, would diminish the power consumption by orders of magnitude more than not using ECC memory.
      - frognumber 3y ago
        It's actually a bit more bogus than that. If I can tolerate occasional bit errors -- which ECC permits -- I can reduce my power consumption by far more than the overhead of ECC. Right now, I power rails set to where billions of bits to never flip. If I can tolerate a one-in-a-billion bit flip, my tolerances go up a lot, and power consumption goes down by a similar factor. Properly implemented, ECC would likely cut power while increasing reliability. "Properly implemented" only happens in volume. That sort of optimization makes no sense for the very limited volumes ECC has today. Neither do a lot of other optimizations.
        thfuran 3y ago
        ECC is ubiquitous in the server space and I wouldn’t really classify that as limited volumes.
- woadwarrior01 3y ago
  It's all fun and games until your computer starts acting all weird, like my 3 year old box did, last week. After being befuddled for a couple of hours, I decided to run MemTest86 off of an Ubuntu installer and found memory errors[1]. The manufacturer has RMA'd the whole set of memory sticks, but ECC would've detected this earlier and I'd have saved a couple of hours of my time. [1]: https://imgur.com/a/aRHWKxw https://imgur.com/a/aRHWKxw
  - eptcyka 3y ago
    If you've used the computer for rendering things or running heavy duty analysis, could you trust your results now, if you know that the memory was faulty for some time? ECC isn't about making life easy when the memory goes bad, it's about allowing one to have more confidence in their system operating correctly.
  - School-Cotton 3y ago
    So you think everyone should be forced to accept higher prices and lower battery life permanently because of something that cost you a few hours, once, that most people never encounter?
    - loeg 3y ago
      The manufacturing costs and power use are not meaningfully different.
      - kevin_thibedeau 3y ago
        The purpose of on chip ECC in DDR5 is to lower costs by hiding failures in marginal chips. They can't sell that memory as normal ECC.
      - Dylan16807 3y ago
        12.5% more chips with previous generations wasn't too bad, but 25% more chips with DDR5 gets to be pretty meaningful.
- throw0101b 3y ago
  > Why? I’ve done so my whole life with zero problems. And folks experienced zero file corruption… until they started using ZFS checksums and regular scrubbing.
  - echoangle 3y ago
    Ok, but that’s not a reason to mandate it by law. Or are you suggesting we should mandate file checksums too?
    - almostnormal 3y ago
      Devices sold should work to a certain expected level. Regulators could define (or even just judges decide in a specific case that can be applied more generally) what that level is. Losing bits randomly isn't within the range of what is expected. The more memory there is, the more likely it is going to happen. And at some point devices without ECC could simply be returned as faulty for a full refund. ECC would be required, without any explicit rule.
      - School-Cotton 3y ago
        That is already how it works now… you can already return defective ram, ECC or not. I am happy with the point in the price/reliability/power usage tradeoff that non-ECC ram represents. Why force me to have fewer choices and use something that is worse for my needs in order to solve a theoretical problem that I don’t care about?
        rvnx 3y ago
        To me, it sounds like "if it doesn't work 100.00% of the time, we cannot trust it and should forbid it", but the reality is that neither the world (and even internet) nor electricity nor healthcare, are "100.00% available/reliable". In general don't worry that for very important calculations, they are run a couple of times, at least just to verify.
        3y ago
- pyb 3y ago
  I would assume that the worldwide yearly cost of data corruption is orders of magnitude larger than the cost of mandating ECC everywhere.
  - brookst 3y ago
    Big assumption! I’d love to see a model showing this.
- thfuran 3y ago
  Are you sure about that?
- mrb 3y ago
  Zero problems? Very unlikely. The thing about memory corruption is that it tends to create problems that most people don't recognize as memory corruption issues. A random BSOD or freeze. A corrupted file. A video glitch. A website/JS code fragment misbehaving. Etc.
- qwertox 3y ago
  There's also the chance that you think you had zero problems because you never got notified. Some years ago I built a small N3150 based home server with 16GB RAM. All was fine. But there was this one process that would always act weird after running for a couple of days. It was a Python server with long-running WebSocket connections which I had configured to use `select()` instead of `epoll()`, and for some reason it would eat up file handles until it crashed or froze, can't remember what it was, very odd stuff. I think it was running inside a Docker container and that container then behaved very strangely. Turns out that when I then made a MemTest86 it had faulty RAM. All the data which passed through that area of the RAM and got written to disk was assumed to be corrupted. After replacing the RAM (RMA) and doing a MemTest on it for a day to ensure that it was OK, that problem never appeared again. But keep in mind, this was just one specific process which showed an issue after running for a couple of days, among other long-running processes.
  - NikolaNovak 3y ago
    1 >> I’ve done so my whole life with zero problems. 2 >> There's also the chance that you think you had zero problems because you never got notified. Both are likely true. I.e. Over my 40 years of dealing with computers ,it is likely there were errors which did not become problems. I have about 20tb of unique files in my personal storage at this point (used to run photography business, plus personal videos and photos,plus movies and tv shows, plus actually important documents which are less than 3% by size).I have about 4-8 computers running around my home doing stuff (plus dozens of computing devices - phones, googles, whatnots). I understand the overwhelming chance there were many errors over time in running computers or raid transfers or usb backups over decades. They just haven't manifested themselves as problems.
    - snvzz 3y ago
      The fact that you're dealing with silent corruption does not make it any better, but in fact, much worse. And yes, the corruption is there. DRAM is alarmingly unreliable. Because no ECC, we just do not notice it most of the time, or misattribute the problems it causes.
- 1letterunixname 3y ago
  Anecdotal non-evidence. You've never noticed or had specific evidence of silent corruption that probabilistically risked silent corruption to quiesced to permanent storage or caused unexplained storage. Perhaps you've never heard of domain bitsquatting but it was rampant before https:// https:// became widespread. All network gear, servers, and client devices need ECC or parity on all buses and at higher level storage, compute, and networking protocols to ensure integrity. Retail consumer electronics manufacturers throw out features customers lack the expertise to understand the consequences of. ECC RAM would need to be enforced by government mandate because that's the only way things will ever change. It's a massive, still present security and data integrity hole that could be closed today by spending a trivial amount of money if it were implemented at large scale. See also: PDF https://media.defcon.org/DEF%20CON%2021/DEF%20CON%2021%20presentations/DEF%20CON%2021%20-%20Schultz-Examining-the-Bitsquatting-Attack-Surface-WP.pdf https://media.defcon.org/DEF%20CON%2021/DEF%20CON%2021%20pre... https://cr.yp.to/hardware/ecc.html https://cr.yp.to/hardware/ecc.html
ddtaylor 3y ago
I'm very happy this should never happen.
brookst 3y ago
Not a fan of using regulatory hammer to forbid people from having different priorities than I do. Adding across the board cost to computers has consequences.
- snvzz 3y ago
  People do shopping, taxes and banking on their computers now. They use personal certificates to sign legally binding documents. Them not having ECC is insanity. There are of course niche cases where memory reliability does not matter. Exceptions can be made for those few. But think about the general case first. Got to protect the non-technical from non-ECC memory.
  - ranger_danger 3y ago
    > Them not having ECC is insanity. I guess the computing world has not survived up to this point then?
    - ThePowerOfFuet 3y ago
      As an information security professional, I can attest that, some days, you're not far off.
- ranger_danger 3y ago
  This is one of the big differences between US and EU culture. freedom TO vs freedom FROM
gosub100 3y ago
I've had way more problems with defective security and bugs from Microsoft than I have with ECC bit-flipping errors. I would rather have "regulators step in" and nullify the disclaimer about "no fitness for any particular purpose" clauses in software license agreements. If I buy a windows computer and it gets hacked, I should be able to take MSFT to court (small claims if need be) and sue for damages (if I can show them) due to their negligence. They market PCs as these great tools for running everything in your life, but first they want release for any responsibility on their part. Those terms do not agree.
- adrian_b 3y ago
  The frequency of software bugs is what has allowed the hardware manufacturers to omit ECC from most computer products. Every time when a program crashes or remains stuck or data corruption is discovered, the user automatically assumes that this is a software bug. While it is indeed true that software bugs are really much more frequent than memory errors before the memory modules happen to become defective, in a computer without ECC the user can never know whether it has been a software bug or it was a hardware memory problem. So many memory errors remain hidden by the abundant software bugs, which in most commercial programs, especially in those from Microsoft, almost never generate error messages from which the cause of the error can be determined.

by snvzz 3y ago

School-Cotton 3y ago

> practically nobody should be running computers w/o ECC Why? I’ve done so my whole life with zero problems.

frognumber 3y ago
I envy you. A life with zero problems.... I think you're probably the only person in the universe so fortunate. Here's the basic set of problems: 1) ECC is almost free if done at scale (e.g. every computer). The only reason it costs extra is that it is only done for high-paying customers and in limited volume. 2) Computer crashes cost lives. If e.g. your doctor's device has an issue mid-surgery, you potentially pay. 3) It's good for the economy. In either case, the cost savings of not having a call center rep say "Wait a minute. My computer just crashed." far outweighs the costs. 4) It's critical for resilience. Disasters (solar flares, EMP, etc.) are rare, but if every computer destroys data, we have a systemic problem. 5) The issue primarily is one of transparency. Most people don't realize it's an issue, and don't shop on it. 6) That's not to mention dual-market uses. E.g. if we need to ramp up military production, medical production, etc. next crisis, our infrastructure should be ready for it. Classical economic theory states government interventions do well for disaster resilience, for externalized costs, and for lack of transparency. Those are all places where markets fail. That means this should be regulated. We should have error correction in all storage and in all computation. Best intervention, though, isn't a ban but a tax. Sales tax should be on a sliding scale from 1-10%, with: - 1% for parts / support / service manuals being available for some extended period - 1% for all software being open-source - 1% for all hardware being open - 1% for use of standard, interchangeable parts (e.g. batteries, power connectors, etc.) - 1% for environmental factors (CO2, toxic materials, etc.) - 1% for reusable infrastructure (e.g. being able to repurpose manufacturing equipment for crisis use) - 1% for workplace factors (e.g reasonable benefits, audits/compliance, non-discrimination for target populations like ex-convicts, etc.) ... and so on Those sorts of gentle incentives accomplish the same thing via market mechanisms without limiting freedom. For a commodity product, a 10% edge far outweighs costs. If Apple wants to make a proprietary magnetic screen connector and save 1mm of space, a 1% extra tax won't even slow them down.
- eptcyka 3y ago
  ECC is more costly in terms of the BoM and higher power consumption.
  - adrian_b 3y ago
    The power consumption is higher by an amount that is not easily detectable. The power consumed by the relatively few gates that generate and verify the ECC code is dwarfed by the power consumption needed to transport bits over the PCB traces. The extra memory cell arrays for ECC bits add some power consumption for refreshing their content and the extra PCB traces for the ECC bits add some power consumption, but this extra power is in the milliwatt range, so it is negligible in comparison with the CPU and GPU power even in a small laptop, not only in a desktop. Most gaming desktops use overclocked memory modules that have an extra power consumption many times greater than the DDR5-4800 or DDR5-5600 ECC UDIMMs. The ECC UDIMMs never need heatsinks, while many of the commonly used overclocked DIMMs will overheat without heatsinks. So the claim that using ECC will increase the power consumption is completely bogus. Something like using a power-save profile instead of a high-performance profile, for clock management, or using JEDEC timings for memory, instead of high-performance timings, would diminish the power consumption by orders of magnitude more than not using ECC memory.
    - frognumber 3y ago
      It's actually a bit more bogus than that. If I can tolerate occasional bit errors -- which ECC permits -- I can reduce my power consumption by far more than the overhead of ECC. Right now, I power rails set to where billions of bits to never flip. If I can tolerate a one-in-a-billion bit flip, my tolerances go up a lot, and power consumption goes down by a similar factor. Properly implemented, ECC would likely cut power while increasing reliability. "Properly implemented" only happens in volume. That sort of optimization makes no sense for the very limited volumes ECC has today. Neither do a lot of other optimizations.
      - thfuran 3y ago
        ECC is ubiquitous in the server space and I wouldn’t really classify that as limited volumes.
woadwarrior01 3y ago
It's all fun and games until your computer starts acting all weird, like my 3 year old box did, last week. After being befuddled for a couple of hours, I decided to run MemTest86 off of an Ubuntu installer and found memory errors[1]. The manufacturer has RMA'd the whole set of memory sticks, but ECC would've detected this earlier and I'd have saved a couple of hours of my time. [1]: https://imgur.com/a/aRHWKxw https://imgur.com/a/aRHWKxw
- eptcyka 3y ago
  If you've used the computer for rendering things or running heavy duty analysis, could you trust your results now, if you know that the memory was faulty for some time? ECC isn't about making life easy when the memory goes bad, it's about allowing one to have more confidence in their system operating correctly.
- School-Cotton 3y ago
  So you think everyone should be forced to accept higher prices and lower battery life permanently because of something that cost you a few hours, once, that most people never encounter?
  - loeg 3y ago
    The manufacturing costs and power use are not meaningfully different.
    - kevin_thibedeau 3y ago
      The purpose of on chip ECC in DDR5 is to lower costs by hiding failures in marginal chips. They can't sell that memory as normal ECC.
    - Dylan16807 3y ago
      12.5% more chips with previous generations wasn't too bad, but 25% more chips with DDR5 gets to be pretty meaningful.
throw0101b 3y ago
> Why? I’ve done so my whole life with zero problems. And folks experienced zero file corruption… until they started using ZFS checksums and regular scrubbing.
- echoangle 3y ago
  Ok, but that’s not a reason to mandate it by law. Or are you suggesting we should mandate file checksums too?
  - almostnormal 3y ago
    Devices sold should work to a certain expected level. Regulators could define (or even just judges decide in a specific case that can be applied more generally) what that level is. Losing bits randomly isn't within the range of what is expected. The more memory there is, the more likely it is going to happen. And at some point devices without ECC could simply be returned as faulty for a full refund. ECC would be required, without any explicit rule.
    - School-Cotton 3y ago
      That is already how it works now… you can already return defective ram, ECC or not. I am happy with the point in the price/reliability/power usage tradeoff that non-ECC ram represents. Why force me to have fewer choices and use something that is worse for my needs in order to solve a theoretical problem that I don’t care about?
      - rvnx 3y ago
        To me, it sounds like "if it doesn't work 100.00% of the time, we cannot trust it and should forbid it", but the reality is that neither the world (and even internet) nor electricity nor healthcare, are "100.00% available/reliable". In general don't worry that for very important calculations, they are run a couple of times, at least just to verify.
        3y ago
pyb 3y ago
I would assume that the worldwide yearly cost of data corruption is orders of magnitude larger than the cost of mandating ECC everywhere.
- brookst 3y ago
  Big assumption! I’d love to see a model showing this.
thfuran 3y ago
Are you sure about that?
mrb 3y ago
Zero problems? Very unlikely. The thing about memory corruption is that it tends to create problems that most people don't recognize as memory corruption issues. A random BSOD or freeze. A corrupted file. A video glitch. A website/JS code fragment misbehaving. Etc.
qwertox 3y ago
There's also the chance that you think you had zero problems because you never got notified. Some years ago I built a small N3150 based home server with 16GB RAM. All was fine. But there was this one process that would always act weird after running for a couple of days. It was a Python server with long-running WebSocket connections which I had configured to use `select()` instead of `epoll()`, and for some reason it would eat up file handles until it crashed or froze, can't remember what it was, very odd stuff. I think it was running inside a Docker container and that container then behaved very strangely. Turns out that when I then made a MemTest86 it had faulty RAM. All the data which passed through that area of the RAM and got written to disk was assumed to be corrupted. After replacing the RAM (RMA) and doing a MemTest on it for a day to ensure that it was OK, that problem never appeared again. But keep in mind, this was just one specific process which showed an issue after running for a couple of days, among other long-running processes.
- NikolaNovak 3y ago
  1 >> I’ve done so my whole life with zero problems. 2 >> There's also the chance that you think you had zero problems because you never got notified. Both are likely true. I.e. Over my 40 years of dealing with computers ,it is likely there were errors which did not become problems. I have about 20tb of unique files in my personal storage at this point (used to run photography business, plus personal videos and photos,plus movies and tv shows, plus actually important documents which are less than 3% by size).I have about 4-8 computers running around my home doing stuff (plus dozens of computing devices - phones, googles, whatnots). I understand the overwhelming chance there were many errors over time in running computers or raid transfers or usb backups over decades. They just haven't manifested themselves as problems.
  - snvzz 3y ago
    The fact that you're dealing with silent corruption does not make it any better, but in fact, much worse. And yes, the corruption is there. DRAM is alarmingly unreliable. Because no ECC, we just do not notice it most of the time, or misattribute the problems it causes.
1letterunixname 3y ago
Anecdotal non-evidence. You've never noticed or had specific evidence of silent corruption that probabilistically risked silent corruption to quiesced to permanent storage or caused unexplained storage. Perhaps you've never heard of domain bitsquatting but it was rampant before https:// https:// became widespread. All network gear, servers, and client devices need ECC or parity on all buses and at higher level storage, compute, and networking protocols to ensure integrity. Retail consumer electronics manufacturers throw out features customers lack the expertise to understand the consequences of. ECC RAM would need to be enforced by government mandate because that's the only way things will ever change. It's a massive, still present security and data integrity hole that could be closed today by spending a trivial amount of money if it were implemented at large scale. See also: PDF https://media.defcon.org/DEF%20CON%2021/DEF%20CON%2021%20presentations/DEF%20CON%2021%20-%20Schultz-Examining-the-Bitsquatting-Attack-Surface-WP.pdf https://media.defcon.org/DEF%20CON%2021/DEF%20CON%2021%20pre... https://cr.yp.to/hardware/ecc.html https://cr.yp.to/hardware/ecc.html

ddtaylor 3y ago

I'm very happy this should never happen.

brookst 3y ago

Not a fan of using regulatory hammer to forbid people from having different priorities than I do. Adding across the board cost to computers has consequences.

snvzz 3y ago
People do shopping, taxes and banking on their computers now. They use personal certificates to sign legally binding documents. Them not having ECC is insanity. There are of course niche cases where memory reliability does not matter. Exceptions can be made for those few. But think about the general case first. Got to protect the non-technical from non-ECC memory.
- ranger_danger 3y ago
  > Them not having ECC is insanity. I guess the computing world has not survived up to this point then?
  - ThePowerOfFuet 3y ago
    As an information security professional, I can attest that, some days, you're not far off.
ranger_danger 3y ago
This is one of the big differences between US and EU culture. freedom TO vs freedom FROM

gosub100 3y ago

I've had way more problems with defective security and bugs from Microsoft than I have with ECC bit-flipping errors. I would rather have "regulators step in" and nullify the disclaimer about "no fitness for any particular purpose" clauses in software license agreements. If I buy a windows computer and it gets hacked, I should be able to take MSFT to court (small claims if need be) and sue for damages (if I can show them) due to their negligence. They market PCs as these great tools for running everything in your life, but first they want release for any responsibility on their part. Those terms do not agree.

adrian_b 3y ago
The frequency of software bugs is what has allowed the hardware manufacturers to omit ECC from most computer products. Every time when a program crashes or remains stuck or data corruption is discovered, the user automatically assumes that this is a software bug. While it is indeed true that software bugs are really much more frequent than memory errors before the memory modules happen to become defective, in a computer without ECC the user can never know whether it has been a software bug or it was a hardware memory problem. So many memory errors remain hidden by the abundant software bugs, which in most commercial programs, especially in those from Microsoft, almost never generate error messages from which the cause of the error can be determined.