11 ms·
I beg this, every time this conversation comes up it’s the same answer “I don’t see a problem”. It’s so easy to chalk these kind of errors to other issues, a l
by dijit 6y ago
I beg this, every time this conversation comes up it’s the same answer “I don’t see a problem”.
It’s so easy to chalk these kind of errors to other issues, a little corruption here, a running program goes bezerk there- could be a buggy program or a little accidental memory overwrite. Reboot will fix it.
But I ran many thousands of physical machines, petabytes of RAM, I tracked memory flip errors and they were _common_; common even in: less dense memory, in thick metal enclosures surrounded by mesh. Where density and shielding impacts bitflips a lot.
My own experience tracking bitflips across my fleet led me to buy a Xeon laptop with ECC memory (precision 5520) and it has (anecdotally) been significantly more reliable than my desktop.
- lighttower 6y agoCan you get decent battery life with this ecc memory in a laptop?
- dijit 6y agoYes. ECC memory uses only marginally more power than non-ECC memory. And memory isn’t the largest consumer of battery life by a country mile. Screen, Wi-Fi, and to a much lesser extent (unless under load) the CPU are the most major culprits of low battery life.
- indolering 6y agoIt can actually reduce power consumption, because refresh rates don't need to be so high: https://media-www.micron.com/-/media/client/global/documents/products/white-paper/ecc_for_mobile_devices_white_paper.pdf?la=en-in&rev=b892d7cdf84a495d83b2560b2fe523ef https://media-www.micron.com/-/media/client/global/documents...
- loeg 6y agoYeah, it's real obnoxious of Intel to silo ECC support off into the Xeon line, isn't it? I switched to ECC memory in 2013 or 2014 with a Xeon E3 (fundamentally a Core i7 without the ECC support fused off) and of course a Xeon-supporting motherboard (with weird "server board" quirks: e.g., no on-board sound device). I love that AMD doesn't intentionally break ECC on its consumer desktop platforms and upgraded to the Threadripper in 2017.
- rhn_mk1 6y agoDoesn't intel make ECC available on the i3 line of CPUs?
- minot 6y agoI was going to say no but I just checked and at least ONE latest generation i3 processor supports ECC https://ark.intel.com/content/www/us/en/ark/compare.html?productIds=208656,209735,209736,208074,208079,208920,208652,208651 https://ark.intel.com/content/www/us/en/ark/compare.html?pro... https://ark.intel.com/content/www/us/en/ark/products/208074/intel-core-i3-1115gre-processor-6m-cache-up-to-3-90-ghz.html https://ark.intel.com/content/www/us/en/ark/products/208074/... Problem is this processor is an Embedded processor so probably not for us > Industrial Extended Temp, Embedded Broad Market Extended Temp My understanding is Intel does not support ECC on the desktop unless you pay extra.
- makomk 6y agoYeah, that appears to be a BGA-packaged processor designed to be permanently soldered to the board of some embedded device, not something that you can install in your desktop at all. I'm not sure why Intel decided to brand their embedded processors with ECC as i3, though I suspect the reason this range exists at all is because companies were going with competitors like AMD instead due to their across-the-board ECC support.
- hollerith 6y agoThat i3 is for file servers.
- opencl 6y agoThey used to support ECC in the desktop i3 lineup, current gen does not have ECC except in some embedded SKUs. https://ark.intel.com/content/www/us/en/ark/products/199280/intel-core-i3-10320-processor-8m-cache-up-to-4-60-ghz.html https://ark.intel.com/content/www/us/en/ark/products/199280/...
- xxs 6y ago
- deleted 6y ago[deleted]
- smoyer 6y agoThere is no guarantee of state at the quantum level ... just a high-degree of assurance in a state. After 40 years in the electronics, optics, software business, I've learned that there is absolutely the possibility for unexplained "blips".
- hosteur 6y agoHow did you track memory errors across thousands of physical machines?
- core-questions 6y agohttps://github.com/netdata/netdata/issues/1508 https://github.com/netdata/netdata/issues/1508 Looks like `mcelog --client` might be a starting place? Feed that into your metrics pipeline and alert on it like anything else...
- jeffbee 6y agoNewer linux have replaced mcelog with edac-util. I think most shops operating systems at that scale are getting their ECC errors out of band with IPMI SEL, though.
- gsvelto 6y agoIt's rasdaemon these days: https://www.setphaserstostun.org/posts/monitoring-ecc-memory-on-linux-with-rasdaemon/ https://www.setphaserstostun.org/posts/monitoring-ecc-memory...
- ikiris 6y agoThe same way you do it with everything else, export the telemetry and store it in time series...
- incrudible 6y agoWhen you say bitflips were "common" on thousands of physical machines, does that mean you observed thousands of bitflips? Otherwise, I would think that an unlikely event becoming 1000x more likely by sheer numbers would have warped your perception. I believe that hardware reliability is mostly irrelevant, because software reliability is already far worse. It doesn't matter whether a bitflip (unlikely) or some bug (likely) causes a node to spuriously fail, what matters is that this failure is handled gracefully.
- felixhandte 6y agoYes. I work at Facebook on data compression. The libraries we maintain (1) are responsible for a non-trivial part of Facebook's overall compute footprint, (2) should basically never fail of their own accord, and (3) have pretty good error monitoring. So my team is operating what is effectively (among other things) a very sensitive detector for hardware failure. And indeed we see examples all the time of blobs that fail to decompress, and usually when we dig in we find that the blob is only a single bit-flip away from a blob that decompresses successfully into a syntactically correct message. I can't share numbers, but, off the top of my head, I think it's the largest source of failures we see. It happens frequently enough that I wrote a tool to automate checking [0]. So yes. It happens. Pretty frequently, in the sense that if you're doing xillions of operations a day, a one-in-a-xillion failure happens all the time. [0] https://github.com/facebook/zstd/tree/dev/contrib/diagnose_corruption https://github.com/facebook/zstd/tree/dev/contrib/diagnose_c...
- dkersten 6y agoAnother comment[1] mentioned 1 bitflip per gigabyte per month. If you have a lot of RAM, that's rather a lot. > It doesn't matter whether a bitflip (unlikely) or some bug (likely) causes a node to spuriously fail Except that a bitflip can go undetected. It may crash your software or system, but it also may simply leak errors into your data, which can be far more catastrophic. [1] https://news.ycombinator.com/item?id=25623206 https://news.ycombinator.com/item?id=25623206
- thequux 6y ago
- derefr 6y agoWere you around for enough DRAM generations to notice an effect of DRAM density / cell-size on reported ECC error rate? I’ve always believed that, ECC aside, DRAM made intentionally with big cells would be less prone to spurious bit-flips (and that this is one of the things NASA means when they talk about “radiation hardening” a computer: sourcing memory with ungodly-large DRAM cells, willingly trading off lower memory capacity for higher per-cell level-shift activation-energy.) If that’s true, then that would mean that the per-cell error rate would have actually been increasing over the years, as DRAM cell-size decreased, in the same way cell-size decrease and voltage-level tightening have increased error rate for flash memory. Combined with the fact that we just have N times more memory now, you’d think we’d be seeing a quadratic increase in faults compared to 40 years ago. But do we? It doesn’t seem like it. I’ve also heard a counter-effect proposed, though: maybe there really are far more “raw” bit-flips going on — but far less of main memory is now in the causal chain for corrupting a workload than it used to be. In the 80s, on an 8-bit micro, POKEing any random address might wreck a program, since there’s only 64k addresses to POKE and most of the writable ones are in use for something critical. Today, most RAM is some sort of cache or buffer that’s going to be used once to produce some ephemeral IO effect (e.g. the compressed data for a video frame, that might decompress incorrectly, but only cause 16ms of glitchiness before the next frame comes along to paper over it); or, if it’s functional data, it’s part of a fault-tolerant component (e.g. a TCP packet, that’s going to checksum-fail when passed to the Ethernet controller and so not even be sent, causing the client to need to retry the request; or, even if accidentally checksums correctly, the server will choke on the malformed request, send an error... and the client will need to retry the request. One generic retry-on-exception handler around your net request, and you get memory fault-tolerance for free!) If both effects are real, this would imply that regular PCs without ECC should still seem quite stable — but that it would be a far worse idea to run a non-ECC machine as a densely-packed multitenant VM hypervisor today (i.e. to tile main memory with OS kernels), than it would have been ~20 years ago when memory densities were lower. Can anyone attest to this? (I’d just ask for actual numbers on whether per-cell per-second errors have increased over the years, but I don’t expect anyone has them.)
- mlyle 6y agoI think it's been quadratic with a pretty low contribution from the order 2 term. Think of the number of events that can flip a bit. If you make bits smaller, you get a modestly larger number of events in a given area capable of flipping a bit, spread across a larger number of bits in that area. That is, it's flip event rate * memory die area, not flip event rate * number of memory bits. In recent generations, I understand it's even been a bit paradoxical-- smaller geometries mean less of the die is actual memory bits, so you can actually end up with fewer flips from shrinking geometries. And sure, your other effect is true: there's a whole lot fewer bitflips that "matter". Flip a bit in some framebuffer used in compositing somewhere-- and that's a lot of my memory-- and I don't care.