4 ms·

I don't know folks. 2 years ago I took a laptop which I wasn't using (16 GiB RAM non-ECC) => I created in Linux with Python an array ("bytes"? Don't remember e

I don't know folks. 2 years ago I took a laptop which I wasn't using (16 GiB RAM non-ECC) => I created in Linux with Python an array ("bytes"? Don't remember exactly anymore) of ~10 or 12 GiB containing random integers => computed the array's hash and saved it. Then for ~1-2 months I recomputed from time to time the hash of that array (inbetween the laptop was in suspend-to-RAM) and compared it to the original result => it always matched, I never had any bitflips. I therefore doubt that the estimation of "1/256MB/month" is correct - I could not prove that, at least not with my laptop.

deckard1 5y ago
I've always been a bit skeptical of published numbers. I usually just chalk it up to vastly different operating conditions and scale. On my home server w/ ECC you can check the corrected and uncorrected (multibit) errors. Assuming my Ryzen is correctly reporting them to Linux, I have 0 errors corrected and 0 uncorrected with a 80 day uptime. I've checked a few other times and never seen an error. Others with ECC often report the same. My understanding of modern RAM is that it has checks built in to the modules which are somewhat equivalent to ECC already (the correcting part, not the reporting part). Which is a necessity in order to hit the density we are at today.
- dijit 5y ago
  DDR5 does and so does the raspberry pi memory. But DDR4 and before does not I think.
tclancy 5y ago
>I therefore doubt that the estimation of "1/256MB/month" is correct As someone who did incredibly poorly in high school physics, this line in the article bothered me as well: the study is from the 1990s when the density of memory would have been much lower. I would think the percentage per megabyte has dropped significantly in 30 or so years. It also assumes a constant form factor for the memory, doesn't it?
- ChrisLomont 5y ago
  The energy needed to flip a bit has drastically lowered since the 1990s, and increased density makes bit flip rates increase at some point since a single physical event can flip multiple bits. So it's not so simple.
  - tclancy 5y ago
    Thanks. So I was sort of right only in the exact opposite direction!
nomel 5y ago
> I therefore doubt that the estimation of "1/256MB/month" is correct The probability is related to the physical volume the memory takes, since it's caused by a physical particle going through that volume. So, this rate will continuously drop as memory density increases.
- Rebelgecko 5y ago
  My understanding is that newer/denser transistors are actually more susceptible to single event upsets, because it takes less energy to flip a bit.
cozzyd 5y ago
A server with 64 GB of ECC ram sitting at an altitude of 3.2 km on the Greenland ice sheet is reporting... 0 bit errors (whether correctable or uncorrectable) in the 244 days it's been up. A server with 16 GB of ECC ram at an altitude of 3.8 km in California is reporting.... 0 bit errors in the 146 days it's been up. Maybe I shouldn't believe what /sys/devices/system/edac/mc is reporting? These are EL8 systems...
- zerd 5y ago
  Bryan Cantrill mentions in this talk that they saw no correctable errors, until they suddenly got uncorrectable errors. Turns out that the firmware was hiding the fact that it had been correcting ECC errors the whole time. https://youtu.be/vvZA9n3e5pc?t=1193 https://youtu.be/vvZA9n3e5pc?t=1193
joatmon-snoo 5y ago
The last time I did research into this, this was the most concrete real world example I could find, which works out to ~41 bitflips/GiB-month: > Jaguar had 360 terabytes of main memory [and] was logging ECC errors at a rate of 350 per minute Source: https://spectrum.ieee.org/how-to-kill-a-supercomputer-dirty-power-cosmic-rays-and-bad-solder https://spectrum.ieee.org/how-to-kill-a-supercomputer-dirty-...
- exikyut 5y ago
  This is throwing up a soft paywall for me FWIW - I get up to "...couldn’t run more than an hour or so without crashing." and then the article text stops, there's a "become a member" interstitial and then I'm fed with tons of headlines for different articles instead. Because of the sharp contrast with the interstitial block I thought it was like a footer, and then a skippable/ignorable part of the layout, and spent some time confusedly scrolling up and down thinking the content was past it, then I realized the "Keep Reading" bits were links to other articles... wow what a mess. </rant>
- NGC404 5y ago
  http://energysfe.ufsc.br/slides/Paolo-Rech-260917.pdf http://energysfe.ufsc.br/slides/Paolo-Rech-260917.pdf
simne 5y ago
Agree. I have once learn subject of cosmic rays, and found estimation, that every square meter suffer HUNDREDS events per second, of course mostly weak events. What I really see - on digital photo dark current test videos, usually could see few flashes. So looks like, modern ram has some sort of ecc inside chip. And yes, digital photo sensors much better suited for detecting of cosmic rays, even simplest, sure much better (semi)professional, which could tuned to make long exposition. But even on book webcam, or on smart front camera, could see events when low light (or if close lens with some opaque material).
dijit 5y ago
In 2014 I worked on a video game that loaded 1,000 instances of a mutable game world into memory, each machine had 256G of ram (about 210G was allocated to the game server) and I ran about 2,500~ of those. The large mutable state like this tends to stress the memory bandwidth, so I’m not sure it’s related, but we used ECC memory and saw a seriously insane number of corrected bit flips. Somewhere in the order of 100/s over the whole fleet as a baseline. there were times that this rate spiked due to a faulty RAM stick, but if you look at the distribution it was fairly even, though there were fairly random hotspots. After observing this I decided I didn’t want to live without ECC on my personal life, to my surprise there is no laptop on the market that supports ECC. Or at least none that I could find. This issue goes away with DDR5 and built-in ECC though, or, our visibility of it does. So, I’m happy.

deckard1 5y ago

I've always been a bit skeptical of published numbers. I usually just chalk it up to vastly different operating conditions and scale. On my home server w/ ECC you can check the corrected and uncorrected (multibit) errors. Assuming my Ryzen is correctly reporting them to Linux, I have 0 errors corrected and 0 uncorrected with a 80 day uptime. I've checked a few other times and never seen an error. Others with ECC often report the same. My understanding of modern RAM is that it has checks built in to the modules which are somewhat equivalent to ECC already (the correcting part, not the reporting part). Which is a necessity in order to hit the density we are at today.

dijit 5y ago
DDR5 does and so does the raspberry pi memory. But DDR4 and before does not I think.

tclancy 5y ago

>I therefore doubt that the estimation of "1/256MB/month" is correct As someone who did incredibly poorly in high school physics, this line in the article bothered me as well: the study is from the 1990s when the density of memory would have been much lower. I would think the percentage per megabyte has dropped significantly in 30 or so years. It also assumes a constant form factor for the memory, doesn't it?

ChrisLomont 5y ago
The energy needed to flip a bit has drastically lowered since the 1990s, and increased density makes bit flip rates increase at some point since a single physical event can flip multiple bits. So it's not so simple.
- tclancy 5y ago
  Thanks. So I was sort of right only in the exact opposite direction!

nomel 5y ago

> I therefore doubt that the estimation of "1/256MB/month" is correct The probability is related to the physical volume the memory takes, since it's caused by a physical particle going through that volume. So, this rate will continuously drop as memory density increases.

Rebelgecko 5y ago
My understanding is that newer/denser transistors are actually more susceptible to single event upsets, because it takes less energy to flip a bit.

cozzyd 5y ago

A server with 64 GB of ECC ram sitting at an altitude of 3.2 km on the Greenland ice sheet is reporting... 0 bit errors (whether correctable or uncorrectable) in the 244 days it's been up. A server with 16 GB of ECC ram at an altitude of 3.8 km in California is reporting.... 0 bit errors in the 146 days it's been up. Maybe I shouldn't believe what /sys/devices/system/edac/mc is reporting? These are EL8 systems...

zerd 5y ago
Bryan Cantrill mentions in this talk that they saw no correctable errors, until they suddenly got uncorrectable errors. Turns out that the firmware was hiding the fact that it had been correcting ECC errors the whole time. https://youtu.be/vvZA9n3e5pc?t=1193 https://youtu.be/vvZA9n3e5pc?t=1193

joatmon-snoo 5y ago

The last time I did research into this, this was the most concrete real world example I could find, which works out to ~41 bitflips/GiB-month: > Jaguar had 360 terabytes of main memory [and] was logging ECC errors at a rate of 350 per minute Source: https://spectrum.ieee.org/how-to-kill-a-supercomputer-dirty-power-cosmic-rays-and-bad-solder https://spectrum.ieee.org/how-to-kill-a-supercomputer-dirty-...

exikyut 5y ago
This is throwing up a soft paywall for me FWIW - I get up to "...couldn’t run more than an hour or so without crashing." and then the article text stops, there's a "become a member" interstitial and then I'm fed with tons of headlines for different articles instead. Because of the sharp contrast with the interstitial block I thought it was like a footer, and then a skippable/ignorable part of the layout, and spent some time confusedly scrolling up and down thinking the content was past it, then I realized the "Keep Reading" bits were links to other articles... wow what a mess. </rant>
NGC404 5y ago
http://energysfe.ufsc.br/slides/Paolo-Rech-260917.pdf http://energysfe.ufsc.br/slides/Paolo-Rech-260917.pdf

simne 5y ago

Agree. I have once learn subject of cosmic rays, and found estimation, that every square meter suffer HUNDREDS events per second, of course mostly weak events. What I really see - on digital photo dark current test videos, usually could see few flashes. So looks like, modern ram has some sort of ecc inside chip. And yes, digital photo sensors much better suited for detecting of cosmic rays, even simplest, sure much better (semi)professional, which could tuned to make long exposition. But even on book webcam, or on smart front camera, could see events when low light (or if close lens with some opaque material).

dijit 5y ago

In 2014 I worked on a video game that loaded 1,000 instances of a mutable game world into memory, each machine had 256G of ram (about 210G was allocated to the game server) and I ran about 2,500~ of those. The large mutable state like this tends to stress the memory bandwidth, so I’m not sure it’s related, but we used ECC memory and saw a seriously insane number of corrected bit flips. Somewhere in the order of 100/s over the whole fleet as a baseline. there were times that this rate spiked due to a faulty RAM stick, but if you look at the distribution it was fairly even, though there were fairly random hotspots. After observing this I decided I didn’t want to live without ECC on my personal life, to my surprise there is no laptop on the market that supports ECC. Or at least none that I could find. This issue goes away with DDR5 and built-in ECC though, or, our visibility of it does. So, I’m happy.