8 ms·

ECC memory can't eliminate the chances of these failures entirely. They can still happen. Making software resilient against bitflips in memory seems very diffic

ECC memory can't eliminate the chances of these failures entirely. They can still happen. Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. So in theory the behavior of software under random bit flips is well... Random. You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. I could imagine that doing so would still be cheaper than using ECC ram, at least around 2000. Generally this goes against software engineering principles. You don't try to eliminate the chances of failure and hope for the best. You need to create these failures constantly (within reasonable bounds) and make sure your software is able to handle them. Using ECC ram is the opposite. You just make it so unlikely to happen, that you will generally not encounter these errors at scale anymore, but nontheless they can still happen and now you will be completely unprepared to deal with them, since you chose to ignore this class of errors and move it under the rug. Another intersting side effect of quorum is that it also makes certain attacks more difficult to pull off, since now you have to make sure that a quorum of machines gives the same "wrong" answer for an attack to work.

sobriquet9 6y ago
If you use multiple computers doing the same calculation and then take the answer from the quorum, how do you ensure the computer that does the comparison is not affected by memory failures? Remember that all queries have to through it, so it has to be comparable in scale and power.
- rovr138 6y ago
  > how do you ensure the computer that does the comparison is not affected by memory failures? You do the comparison on multiple nodes too. Get the calculations. Pass them to multiple nodes, validate again and if it all matches, you use it.
  - sobriquet9 6y ago
    > validate again Recursion, see recursion.
    - wtallis 6y ago
      See also Byzantine fault tolerance: https://scholar.harvard.edu/files/mickens/files/thesaddestmoment.pdf https://scholar.harvard.edu/files/mickens/files/thesaddestmo...
    - Guvante 6y ago
      I mean raft and similar algorithms run multiple verification machines because a single point of failure is a single point of failure.
      - sobriquet9 6y ago
        Raft, Paxos, and other consensus algorithms add even more overhead. Imagine running every Google query through Raft and think how long it will take and how much extra hardware would be needed. ECC memory is just as fast as non-ECC memory, and only cost a little more.
        Guvante 6y ago
        Your comment sounded like "your recursive definition is impossible". I am totally for ECC and was flabbergasted when it went away. But the article makes sense since I remember Intel pushing hard to keep it out of the consumer space. The freaking QX6800 didn't support ECC and it retailed for over a grand.
tomxor 6y ago
> Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. There is an OS that pretty much fits the bill here. There was a show where Andrew Tanenbaum had a laptop running Minix 3 hooked up to a button that injected random changes into module code while it was running to demonstrate it's resilience to random bugs. Quite fitting that this discussion was initiated by Linus! Although it was intended to protect against bad software I don't see why it wouldn't also go a long way in protecting the OS against bitflips. Minix 3 uses a microkernel with a "reincarnation server" which means it can automatically reload any misbehaving code not part of the core kernel on the fly (which for Minix is almost everything). This even includes disk drivers. In the case of misbehaving code there is some kind of triple redundancy mechanism much like the "quorum" you suggest, but that is where my crude understanding ends. AFAIR Userland software could in theory also benefit provided it was written in such a way to be able to continue gracefully on reloading.
- Jedd 6y ago
  At some point, whatever's watching the watchers is going to be vulnerable to bitflip and similar problems. Even with a triple-redundant quorum mechanism, slightly further up that stack you're going to have some bit of code running that processes the three returned results - if the memory that's sitting on gets corrupted, you're back where you started.
  - axiolite 6y ago
    > At some point, whatever's watching the watchers is going to be vulnerable to bitflip One advantage of microkernels is that the "watcher" is so small that it could be run directly from ROM, instead of loaded into RAM. QNX has advocated that route for robotics and such in the past. Minix may not be the best example of the type. While it is a microkernel, it's real world reliability has been poor in the past. More mature microkernel operating systems like QNX and OpenVMS are better examples.
    - KMag 6y ago
      Anyone have any idea why there haven't been any open-source QNX clones, at least not any widely known ones? Even before their Photon MicroGUI patents expired, the clones could have used X11. I used to occasionally boot into QNX on my desktop in college. It was a very responsive and stable system. Hypervisors are, to a first approximation, microkernels with a hardware-like interface. All of this kernel bypass work being done by RDBMSes, ScyllaDB, HFTs, etc. is, to a first approximation, making a monolithic kernel act a bit like a microkernel.
saagarjha 6y ago
There was an interesting challenge at DEF CON CTF a while back that tested this, actually. It turns out that it is possible to write x86 code that is 1-bit-flip tolerant–that is, a bit flip anywhere in its code can be detected and recovered from with the same output. Of course, finding the sequence took (or so I hear) something like 3600 cores running for a day to discover it ;)
- rfoo 6y ago
  Nit: not for a day, more like 8 hours, and that's because we were lazy and somebody said he "just happened" to have a cluster with unbalanced resources (mainly used for deep learning, but all GPUs occupied with quite a lot CPUs / RAMs left), so we decided to brute force the last 16 bits :) Also, the challenge host left useful state (which bit was flipped) in registers before running teams' code, without this I'm not sure if it is even possible.
  - saagarjha 6y ago
    Sure, all's fair in a CTF. That story came to me through the mouths of at least a handful of people, who might have a bit of an incentive to exaggerate given that they hadn't quite been able to get to zero and might be a just a little sour :P The state was quite helpful, yes–for x86 it seems like a "clean slate" shellcode would be quite difficult, if impossible, to achieve as we saw. However, I am left wondering how other ISAs would fare…perhaps worse, since x86 is notoriously dense. But maybe not? The fixed-width ones would probably be easy to try out, at least.
    - rfoo 6y ago
      Maybe being notoriously dense is not a bad thing? While those ModRM bytes popping up everywhere is annoying as f* (too easy to flip an instruction into a form with almost-guaranteed-to-be-invalid memory access), at least due to the density there won't be reserved bits. For example, in AArch64 if bit 28 and bit 27 is both zero the instruction will almost certainly be an invalid one (hitting unallocated area), and with a single bit flip all branch instructions will have [28:27] = b'00... [1] https://developer.arm.com/docs/ddi0596/h/top-level-encodings-for-a64 https://developer.arm.com/docs/ddi0596/h/top-level-encodings...
DSingularity 6y ago
You need two alpha particles hitting the same rank of memory for failure to happen. Although super rare, even then it is still correctable. You need three before it is silent data corruption. Silent corruption is what you get with non ECC with even a single flip.
- klodolph 6y ago
  Where are you getting this from? My understanding is that these errors are predominantly caused by secondary particles from cosmic rays hitting individual memory cells, and I've never heard something so precise as "you need two alpha particles". Aren't the capacitances in modern DRAM chips extremely small?
  - DSingularity 6y ago
    The structure of the ECC is at the rank level. This allows for correcting single bit flips in ranks and detecting double bit flip in ranks. So when you grab a cache line each 64bit is corrected and verified.
  - deleted 6y ago
    [deleted]
slumdev 6y ago
Error-correcting code (the "ECC" in ECC) is just a quorum at the bit level.
- sobriquet9 6y ago
  Modern error correction codes can do much better than that.
- eevilspock 6y ago
  I'm surprised that the other replies don't grasp this. This is the proper level to do the quorum. Doing quorum at the computer level would require synchronizing parallel computers, and unless that synchronization were to happen for each low level instruction, then it would have to be written into the software to take a vote at critical points. This is going to be greatly detrimental both to throughput and software complexity. I guess you could implement the quorum at the CPU level... e.g. have redundant cores each with their own memory. But unless there was a need to protect against CPU cores themselves being unreliable, I don't see this making sense either. At the end of the day, at some level, it will always come down to probabilities. "Software engineering principles" will never eliminate that.
  - slumdev 6y ago
    I would highly recommend a graduate-level course in computer architecture for anyone who thinks ECC is a 1980s solution to a modern problem. There are a lot of seemingly high-level problems that are solved (ingeniously) in hardware with very simple, very low-level solutions.
    - bollu 6y ago
      Could you please link me to such a course that displays the hardware level solutions? I'm super interested!
      - slumdev 6y ago
        https://www.udacity.com/course/high-performance-computer-architecture--ud007 https://www.udacity.com/course/high-performance-computer-arc...
hn3333 6y ago
Bit flips can happen, but regardless if they can get repaired by ECC code or not, the OS is notified, iirc. It will signal a corruption to the process that is mapped to the faulty address. I suppose that if the memory contains code, the process is killed (if ECC correction failed).
- wtallis 6y ago
  > I suppose that if the memory contains code, the process is killed (if ECC correction failed). Generally, it would make the most sense to kill the process if the corrupted page is data, but if it's code, then maybe re-load that page from the executable file on non-volatile storage. (You might also be able to rescue some data pages from swap space this way.)
  - gizmo686 6y ago
    If you go that route, you should be able to avoid the code/data distinction entirely; as data pages can also be completly backed by files. I believe the kernel already keeps track of what pages are a clean copy of data from the filesystem, so I would think it would be a simple matter of essentially pageing out the corrupted data. What would be interesting is if userspace could mark a region of memory as recomputable. If the kernel is notified of memory corruption there, it triggers a handler in the userspace process to rebuild the data. Granted, given the current state of hardware; I can't imagine that is anywhere near worth the effort to implement.
    - wtallis 6y ago
      > What would be interesting is if userspace could mark a region of memory as recomputable. I believe there's already some support for things like this, but intended as a mechanism to gracefully handle memory pressure rather than corruption. Apple has a Purgeable Memory mechanism, but handled through higher-level interfaces rather than something like madvise().
AaronFriel 6y ago
It can't eliminate it but: 1. Single bitflip correction along with Google's metrics could help them identify algorithms they've got, customer's VMs that are causing bitflips via rowhammer and machines which have errors regardless of workload 2. Double bitflip detection lets Google decide if they say, want to panic at that point and take the machine out of service, and they can report on what software was running or why. Their SREs are world-class and may be able to deduce if this was a fluke (orders of magnitude less likely than a single bit flip), if a workload caused it, or if hardware caused it. The advantage the 3 major cloud providers have is scale. If a Fortune 500 were running their own datacenters, how likely would it be that they have the same level of visibility into their workloads, the quality of SREs to diagnose, and the sheer statistical power of scale? I sincerely hope Google is not simply silencing bitflip corrections and detections. That would be a profound waste.
- tjoff 6y ago
  ECC seems like a trivial thing to log and keep track of. Surely any Fortune 500 could do it and would have enough scale to get meaningful data out of it?
  - watersb 6y ago
    IBM does. They will probably sell you the information if you rent the machines from them.
  - AaronFriel 6y ago
    It's not just tracking ECC errors, which as you point out is not hard, but correlating it with the other metrics needed to determine the cause and having the scale to reliably root cause bitflips to software (workloads that inadvertently rowhammer) or hardware or even malicious users (GCP customers that may intentionally run a rowhammer.)
colejohnson66 6y ago
> You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. The Apollo missions (or was it the Space Shuttle?) did this. They had redundant computers that would work with each other to determine the “true” answer.
- EvanAnderson 6y ago
  The Space Shuttle had redundant computers. The Apollo Guidance Computer was not redundant (though there were two AGCs onboard-- one in the CM and one in the LEM). The aerospace industry has a history of using redundant dissimilar computers (different CPU architectures, multiple implementations of the control software developed by separate teams in different languages, etc) in voting-based architectures to hedge against various failure modes.
- buildbuildbuild 6y ago
  This remains common in aerospace, each voting computer is referred to as a "string". https://space.stackexchange.com/questions/45076/what-is-a-flight-string https://space.stackexchange.com/questions/45076/what-is-a-fl...
  - sroussey 6y ago
    In aerospace where this is common, you often had multiple implementations, as you wanted to avoid software bugs made by humans. Problem was, different teams often created the same error at the same place, so it wasn’t as effective as it would have seemed.
- patates 6y ago
  Forgive my ignorance, but wouldn't the computer actually reacting to the calculation (and sending a command or displaying the data) still be very vulnerable to bit-flips? Or were they displaying the results from multiple machines to humans?
- haolez 6y ago
  Sounds similar to smart contracts running on a blockchain :)
giantrobot 6y ago
I don't think ECC is going to give anyone a false sense of security. The issue at Google's scale is they had to spend thousands of person-hours implementing in software what they would have gotten for "free" with ECC RAM. Lacking ECC (and generally using consumer-level hardware) compounded scale and reliability problems or at least made them more expensive than they might otherwise had been. Using consumer hardware and making up reliability with redundancy and software was not a bad idea for early Google but it did end up with an unforeseen cost. Just a thousand machines in a cosmic ray proof bunker will end up with memory errors ECC will correct for free. It's just reducing the surface area of "potential problems".
- Animats 6y ago
  consumer hardware... That's Intel's PR. Only "enterprise hardware", with a bigger markup, supports ECC memory. Adding ECC today should add only 12% to memory cost. AMD decided to break Intel's pricing model. Good for them. Now if we can get ECC at the retail level... The original IBM PC AT had parity in memory.
  - jfim 6y ago
    > Now if we can get ECC at the retail level You can actually, most AMD consumer chips (except the ones with integrated graphics) have ECC support, even though it's not officially supported. See this Reddit thread for more details: https://www.reddit.com/r/Amd/comments/ggmyyg/an_overview_of_ecc_memory_on_ryzen/ https://www.reddit.com/r/Amd/comments/ggmyyg/an_overview_of_...
    - wolrah 6y ago
      I think they may have been referring to the actual mainstream retail availability of ECC RAM. I can buy non-ECC RAM at almost any retailer that sells computers. If I need non-ECC RAM right now I can have it in my hands in 30 minutes. ECC on the other hand I pretty much have to buy online. Microcenter stocks a single 4GB stick of PC4-21300, and I can't think of a single use case where I'd want ECC but not more than 4GB.

sobriquet9 6y ago

If you use multiple computers doing the same calculation and then take the answer from the quorum, how do you ensure the computer that does the comparison is not affected by memory failures? Remember that all queries have to through it, so it has to be comparable in scale and power.

rovr138 6y ago
> how do you ensure the computer that does the comparison is not affected by memory failures? You do the comparison on multiple nodes too. Get the calculations. Pass them to multiple nodes, validate again and if it all matches, you use it.
- sobriquet9 6y ago
  > validate again Recursion, see recursion.
  - wtallis 6y ago
    See also Byzantine fault tolerance: https://scholar.harvard.edu/files/mickens/files/thesaddestmoment.pdf https://scholar.harvard.edu/files/mickens/files/thesaddestmo...
  - Guvante 6y ago
    I mean raft and similar algorithms run multiple verification machines because a single point of failure is a single point of failure.
    - sobriquet9 6y ago
      Raft, Paxos, and other consensus algorithms add even more overhead. Imagine running every Google query through Raft and think how long it will take and how much extra hardware would be needed. ECC memory is just as fast as non-ECC memory, and only cost a little more.
      - Guvante 6y ago
        Your comment sounded like "your recursive definition is impossible". I am totally for ECC and was flabbergasted when it went away. But the article makes sense since I remember Intel pushing hard to keep it out of the consumer space. The freaking QX6800 didn't support ECC and it retailed for over a grand.

tomxor 6y ago

> Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. There is an OS that pretty much fits the bill here. There was a show where Andrew Tanenbaum had a laptop running Minix 3 hooked up to a button that injected random changes into module code while it was running to demonstrate it's resilience to random bugs. Quite fitting that this discussion was initiated by Linus! Although it was intended to protect against bad software I don't see why it wouldn't also go a long way in protecting the OS against bitflips. Minix 3 uses a microkernel with a "reincarnation server" which means it can automatically reload any misbehaving code not part of the core kernel on the fly (which for Minix is almost everything). This even includes disk drivers. In the case of misbehaving code there is some kind of triple redundancy mechanism much like the "quorum" you suggest, but that is where my crude understanding ends. AFAIR Userland software could in theory also benefit provided it was written in such a way to be able to continue gracefully on reloading.

Jedd 6y ago
At some point, whatever's watching the watchers is going to be vulnerable to bitflip and similar problems. Even with a triple-redundant quorum mechanism, slightly further up that stack you're going to have some bit of code running that processes the three returned results - if the memory that's sitting on gets corrupted, you're back where you started.
- axiolite 6y ago
  > At some point, whatever's watching the watchers is going to be vulnerable to bitflip One advantage of microkernels is that the "watcher" is so small that it could be run directly from ROM, instead of loaded into RAM. QNX has advocated that route for robotics and such in the past. Minix may not be the best example of the type. While it is a microkernel, it's real world reliability has been poor in the past. More mature microkernel operating systems like QNX and OpenVMS are better examples.
  - KMag 6y ago
    Anyone have any idea why there haven't been any open-source QNX clones, at least not any widely known ones? Even before their Photon MicroGUI patents expired, the clones could have used X11. I used to occasionally boot into QNX on my desktop in college. It was a very responsive and stable system. Hypervisors are, to a first approximation, microkernels with a hardware-like interface. All of this kernel bypass work being done by RDBMSes, ScyllaDB, HFTs, etc. is, to a first approximation, making a monolithic kernel act a bit like a microkernel.

saagarjha 6y ago

There was an interesting challenge at DEF CON CTF a while back that tested this, actually. It turns out that it is possible to write x86 code that is 1-bit-flip tolerant–that is, a bit flip anywhere in its code can be detected and recovered from with the same output. Of course, finding the sequence took (or so I hear) something like 3600 cores running for a day to discover it ;)

rfoo 6y ago
Nit: not for a day, more like 8 hours, and that's because we were lazy and somebody said he "just happened" to have a cluster with unbalanced resources (mainly used for deep learning, but all GPUs occupied with quite a lot CPUs / RAMs left), so we decided to brute force the last 16 bits :) Also, the challenge host left useful state (which bit was flipped) in registers before running teams' code, without this I'm not sure if it is even possible.
- saagarjha 6y ago
  Sure, all's fair in a CTF. That story came to me through the mouths of at least a handful of people, who might have a bit of an incentive to exaggerate given that they hadn't quite been able to get to zero and might be a just a little sour :P The state was quite helpful, yes–for x86 it seems like a "clean slate" shellcode would be quite difficult, if impossible, to achieve as we saw. However, I am left wondering how other ISAs would fare…perhaps worse, since x86 is notoriously dense. But maybe not? The fixed-width ones would probably be easy to try out, at least.
  - rfoo 6y ago
    Maybe being notoriously dense is not a bad thing? While those ModRM bytes popping up everywhere is annoying as f* (too easy to flip an instruction into a form with almost-guaranteed-to-be-invalid memory access), at least due to the density there won't be reserved bits. For example, in AArch64 if bit 28 and bit 27 is both zero the instruction will almost certainly be an invalid one (hitting unallocated area), and with a single bit flip all branch instructions will have [28:27] = b'00... [1] https://developer.arm.com/docs/ddi0596/h/top-level-encodings-for-a64 https://developer.arm.com/docs/ddi0596/h/top-level-encodings...

DSingularity 6y ago

You need two alpha particles hitting the same rank of memory for failure to happen. Although super rare, even then it is still correctable. You need three before it is silent data corruption. Silent corruption is what you get with non ECC with even a single flip.

klodolph 6y ago
Where are you getting this from? My understanding is that these errors are predominantly caused by secondary particles from cosmic rays hitting individual memory cells, and I've never heard something so precise as "you need two alpha particles". Aren't the capacitances in modern DRAM chips extremely small?
- DSingularity 6y ago
  The structure of the ECC is at the rank level. This allows for correcting single bit flips in ranks and detecting double bit flip in ranks. So when you grab a cache line each 64bit is corrected and verified.
- deleted 6y ago
  [deleted]

slumdev 6y ago

Error-correcting code (the "ECC" in ECC) is just a quorum at the bit level.

sobriquet9 6y ago
Modern error correction codes can do much better than that.
eevilspock 6y ago
I'm surprised that the other replies don't grasp this. This is the proper level to do the quorum. Doing quorum at the computer level would require synchronizing parallel computers, and unless that synchronization were to happen for each low level instruction, then it would have to be written into the software to take a vote at critical points. This is going to be greatly detrimental both to throughput and software complexity. I guess you could implement the quorum at the CPU level... e.g. have redundant cores each with their own memory. But unless there was a need to protect against CPU cores themselves being unreliable, I don't see this making sense either. At the end of the day, at some level, it will always come down to probabilities. "Software engineering principles" will never eliminate that.
- slumdev 6y ago
  I would highly recommend a graduate-level course in computer architecture for anyone who thinks ECC is a 1980s solution to a modern problem. There are a lot of seemingly high-level problems that are solved (ingeniously) in hardware with very simple, very low-level solutions.
  - bollu 6y ago
    Could you please link me to such a course that displays the hardware level solutions? I'm super interested!
    - slumdev 6y ago
      https://www.udacity.com/course/high-performance-computer-architecture--ud007 https://www.udacity.com/course/high-performance-computer-arc...

hn3333 6y ago

Bit flips can happen, but regardless if they can get repaired by ECC code or not, the OS is notified, iirc. It will signal a corruption to the process that is mapped to the faulty address. I suppose that if the memory contains code, the process is killed (if ECC correction failed).

wtallis 6y ago
> I suppose that if the memory contains code, the process is killed (if ECC correction failed). Generally, it would make the most sense to kill the process if the corrupted page is data, but if it's code, then maybe re-load that page from the executable file on non-volatile storage. (You might also be able to rescue some data pages from swap space this way.)
- gizmo686 6y ago
  If you go that route, you should be able to avoid the code/data distinction entirely; as data pages can also be completly backed by files. I believe the kernel already keeps track of what pages are a clean copy of data from the filesystem, so I would think it would be a simple matter of essentially pageing out the corrupted data. What would be interesting is if userspace could mark a region of memory as recomputable. If the kernel is notified of memory corruption there, it triggers a handler in the userspace process to rebuild the data. Granted, given the current state of hardware; I can't imagine that is anywhere near worth the effort to implement.
  - wtallis 6y ago
    > What would be interesting is if userspace could mark a region of memory as recomputable. I believe there's already some support for things like this, but intended as a mechanism to gracefully handle memory pressure rather than corruption. Apple has a Purgeable Memory mechanism, but handled through higher-level interfaces rather than something like madvise().

AaronFriel 6y ago

It can't eliminate it but: 1. Single bitflip correction along with Google's metrics could help them identify algorithms they've got, customer's VMs that are causing bitflips via rowhammer and machines which have errors regardless of workload 2. Double bitflip detection lets Google decide if they say, want to panic at that point and take the machine out of service, and they can report on what software was running or why. Their SREs are world-class and may be able to deduce if this was a fluke (orders of magnitude less likely than a single bit flip), if a workload caused it, or if hardware caused it. The advantage the 3 major cloud providers have is scale. If a Fortune 500 were running their own datacenters, how likely would it be that they have the same level of visibility into their workloads, the quality of SREs to diagnose, and the sheer statistical power of scale? I sincerely hope Google is not simply silencing bitflip corrections and detections. That would be a profound waste.

tjoff 6y ago
ECC seems like a trivial thing to log and keep track of. Surely any Fortune 500 could do it and would have enough scale to get meaningful data out of it?
- watersb 6y ago
  IBM does. They will probably sell you the information if you rent the machines from them.
- AaronFriel 6y ago
  It's not just tracking ECC errors, which as you point out is not hard, but correlating it with the other metrics needed to determine the cause and having the scale to reliably root cause bitflips to software (workloads that inadvertently rowhammer) or hardware or even malicious users (GCP customers that may intentionally run a rowhammer.)

colejohnson66 6y ago

> You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. The Apollo missions (or was it the Space Shuttle?) did this. They had redundant computers that would work with each other to determine the “true” answer.

EvanAnderson 6y ago
The Space Shuttle had redundant computers. The Apollo Guidance Computer was not redundant (though there were two AGCs onboard-- one in the CM and one in the LEM). The aerospace industry has a history of using redundant dissimilar computers (different CPU architectures, multiple implementations of the control software developed by separate teams in different languages, etc) in voting-based architectures to hedge against various failure modes.
buildbuildbuild 6y ago
This remains common in aerospace, each voting computer is referred to as a "string". https://space.stackexchange.com/questions/45076/what-is-a-flight-string https://space.stackexchange.com/questions/45076/what-is-a-fl...
- sroussey 6y ago
  In aerospace where this is common, you often had multiple implementations, as you wanted to avoid software bugs made by humans. Problem was, different teams often created the same error at the same place, so it wasn’t as effective as it would have seemed.
patates 6y ago
Forgive my ignorance, but wouldn't the computer actually reacting to the calculation (and sending a command or displaying the data) still be very vulnerable to bit-flips? Or were they displaying the results from multiple machines to humans?
haolez 6y ago
Sounds similar to smart contracts running on a blockchain :)

giantrobot 6y ago

I don't think ECC is going to give anyone a false sense of security. The issue at Google's scale is they had to spend thousands of person-hours implementing in software what they would have gotten for "free" with ECC RAM. Lacking ECC (and generally using consumer-level hardware) compounded scale and reliability problems or at least made them more expensive than they might otherwise had been. Using consumer hardware and making up reliability with redundancy and software was not a bad idea for early Google but it did end up with an unforeseen cost. Just a thousand machines in a cosmic ray proof bunker will end up with memory errors ECC will correct for free. It's just reducing the surface area of "potential problems".

Animats 6y ago
consumer hardware... That's Intel's PR. Only "enterprise hardware", with a bigger markup, supports ECC memory. Adding ECC today should add only 12% to memory cost. AMD decided to break Intel's pricing model. Good for them. Now if we can get ECC at the retail level... The original IBM PC AT had parity in memory.
- jfim 6y ago
  > Now if we can get ECC at the retail level You can actually, most AMD consumer chips (except the ones with integrated graphics) have ECC support, even though it's not officially supported. See this Reddit thread for more details: https://www.reddit.com/r/Amd/comments/ggmyyg/an_overview_of_ecc_memory_on_ryzen/ https://www.reddit.com/r/Amd/comments/ggmyyg/an_overview_of_...
  - wolrah 6y ago
    I think they may have been referring to the actual mainstream retail availability of ECC RAM. I can buy non-ECC RAM at almost any retailer that sells computers. If I need non-ECC RAM right now I can have it in my hands in 30 minutes. ECC on the other hand I pretty much have to buy online. Microcenter stocks a single 4GB stick of PC4-21300, and I can't think of a single use case where I'd want ECC but not more than 4GB.