6 ms·
Isn't ECC memory supposed to mitigate these kind of bit-flips, specifically it should correct all single bit flips? As this is a single bit-flip, why wasn't it
by dannyw 5y ago
Isn't ECC memory supposed to mitigate these kind of bit-flips, specifically it should correct all single bit flips?
As this is a single bit-flip, why wasn't it corrected? Did ECC memory fail? Or was this bit-flip induced in the CPU pipeline, registers, or cache?
Do we need "RAID for ECC memory", where we halve user-accessible RAM and store each memory segment twice and check for parity?
- dijit 5y agoThese are all very fair statements but there’s no guarantee that ECC memory was even used. Computers typically fail open when ECC is potentially present but not available. People also cite early stage google and intentionally do not buy ECC components, running more consumer hardware for production workloads. Even if google later recanted that theology.
- caspper69 5y agoIt’s always humorous to me when people use the term theology in situations such as this; it makes me wonder, as human mental bandwidth becomes more strained and we increase our specializations to the n-th degree, what will constitute theology in the future? Food for thought.
- loa_in_ 5y agoThe benevolent and malevolent rogue AI, eluding capture or control by claiming critical infrastructure. Some generations of humans will pass and the deities in the outlets will become.
- caspper69 5y agoI like it!
- diegoperini 5y agoFuture is already here and we call it consensus. Trusting your peers, believing they are honest and proficient in their respective fields is a natural human response to the unknown phenomena.
- ZiiS 5y agoPublic CAs are not that type of people; I would be disapointed if that were not running two seperate systems checking each other for consistancy; having top of the range ECC running well inside its specification must be table stakes.
- dijit 5y ago> Public CAs are not that type of people I think you hold public CAs to a higher standard than many hold themselves to. There are hundreds of CAs and many (if not most) are shockingly awful. Which is why we have had a huge push back against the PKI cartels.
- tialaramex 5y agoNot hundreds. There are currently 52 root CA operators trusted by Mozilla (and thus Firefox, but also most Linux systems and lots of other stuff) a few more are trusted only by Microsoft or Apple, but not hundreds. But also, in this context we aren't talking about the CAs anyway, but the Log operators, and so for them reliability is about staying qualified, as otherwise their service is pointless. There are far fewer of those, about half-a-dozen total. Cloudflare, Google, Digicert, Sectigo, ISRG (Let's Encrypt), and Trust Asia. [Edited, I counted a column header, 53 rows minus 1 header = 52]
- fulafel 5y agoThese things exist, using trade names like chipkill or lockstep memory. Though they don't need to sacrifice half of the memory chips to get good error recovery properties. Note that this is still not end-to-end protection of data integrity. Bit flips happen in networking, storage, buses between everything, caches, CPUs, etc. See eg [1] [1] https://arxiv.org/abs/2102.11245 https://arxiv.org/abs/2102.11245 Silent Data Corruptions at Scale (based on empirical data at Facebook)
- formerly_proven 5y agoAccording to the Intel developer's manual L1 has parity and all caches up from that have ECC. This would seem to imply that the ring / mesh also has at least parity (to retry on error). Parity instead of ECC on L1(D) makes sense since the L1(D) has to handle small writes well, while the other caches deal in lines.
- eru 5y agoOf course, parity can only detect an odd number of bit flips.
- RL_Quine 5y agoSome systems do have two parity bits but obviously there’s efficiency loss there.
- namibj 5y agoZen3 has parity on L1I and ECC on L1D and beyond. Even ECC on consumer hardware (it's just a few extra traces on the motherboard).
- willis936 5y agoChecksumming filesystems and file transfer protocols cover many cases. SCP, rsync, btrfs, and zfs all fix this problem. As for guaranteeing the computed data is correct: I know space systems often have two redundant computers that calculate everything and compare results. It's crazy expensive and power demanding, but it all but solves the problem.
- Jolter 5y agoIIRC, EEC memory can correct for single-but flips. It can detect/warn/fail on double-bit flips. And it can not detect triple-bit flips. This might be a simplified understanding, but if this has happened only once, that seems to match up with my intuitive understanding of the probability of a triple bit flip occurring in a particular system.
- willis936 5y agoYou multiply the probability of two random events to get the probability they will happen at the same time. If the expected value of a bit flip is 10^-18 then two would be 10^-36 and three would he 10^-54. At some point it becomes a philosophical question of how much can the tails of the distribution be tolerated. We've never seen a quantum fluctuation make a whale appear in the sky.
- jeffbee 5y agoDRAM failures are not independent events, so it’s not appropriate to multiply the probabilities like that. Faults are often clustered in a row, column, bank, page or whatever structure your DRAM has, raising the probability of multi-bit errors.
- BenjiWiebe 5y agoI believe the usual concern is bit flips due to subatomic particles, and as far as I'm aware that only flips one bit per particle.
- jeffbee 5y agoI don't see why a high-energy particle strike would confine itself to a single bit. The paper I posted elsewhere in this thread says that "the most likely cause of the higher single-bit, single-column, and single-bank transient fault rates in Cielo is particle strikes from high-energy neutrons". In the paper, both single-bit and multi-bit errors are sensitive to altitude.
- the8472 5y agoInstead of ECC it would also be possible to run the log machine redundantly and have each replica cross-check the others before making an update public. I assume the log calculation is deterministic.
- jacquesm 5y agoProcess enough data and even ECC can - and will - fail undetected. Any kind of mechanism you come up with is going to have some rate of undetected errors.
- rob_c 5y agoGiven the rate required for this its not a reasonable assumption. It's like saying Amazon sees sha256 collisions between S3 buckets. Just doesn't happen in practice.
- jeffbee 5y agoUndetected ECC errors are common enough to see from time to time in the wild. This paper estimates that a supercomputer sees one undetected error per day. https://www.cs.virginia.edu/~gurumurthi/papers/asplos15.pdf https://www.cs.virginia.edu/~gurumurthi/papers/asplos15.pdf
- jacquesm 5y agoThose two are many orders of magnitude apart, undetected bit flips in spite of ECC are a fact or life in any large computing installation.
- hbogert 5y agoECC does parity for memory, there's more between memory and cpu registers. Easy candidates are the memory controller, the cpu registers.
- dathinab 5y agoIn my experience it's better to run redundancy on a higher abstraction level. I.e. (simplified) you do the computation twice on different systems then interchange hashes of the result and if they match you continue.
- deleted 5y ago[deleted]
- teddyh 5y agoUser ‘JoshTriplett’ here suggests using forward error correction (FEC) in RAM: https://news.ycombinator.com/item?id=11604918 https://news.ycombinator.com/item?id=11604918
- sgtnoodle 5y ago"Forward error correction" is really just the application of ECC while transmitting data over a lossy link in order to tolerate errors without two-way communication. The ECC used in memory is likely relatively space inefficient at the benefit of being computationally simple so it can be done quickly in hardware. More redundancy could be added to tolerate more bit flips, but it would either add a lot of memory overhead, or a lot of computational complexity. In particular, something really good like reed solomon would likely be very difficult to encode on every single memory write, at least not without taking a several order of magnitude performance hit. It would likely be easier just to have 2x ECC memory, or 3x non-ECC memory and do majority voting.