4 ms·

Do the results talked about in this article scale down to small systems? "Jaguar had 360 terabytes of main memory, all protected by ECC. I and others at the la

Do the results talked about in this article scale down to small systems? "Jaguar had 360 terabytes of main memory, all protected by ECC. I and others at the lab set it up to log every time a bit was flipped incorrectly in main memory. When I asked my computing colleagues elsewhere to guess how often Jaguar saw such a bit spontaneously change state, the typical estimate was about a hundred times a day. In fact, Jaguar was logging ECC errors at a rate of 350 per minute." If I take the 360 TB and scale it down to a laptop with 16 GB of RAM am I seeing errors at a proportional rate? 350 errors / 368 640 GB * 16 GB * 60 minutes * 24 hours = 21.875 memory bits changed per day on a 16 GB laptop That just seems way too high for average users to have not noticed the lack of ECC, and especially for the Jeff Atwood-type non-ECC server crowd (http://blog.codinghorror.com/to-ecc-or-not-to-ecc/ http://blog.codinghorror.com/to-ecc-or-not-to-ecc/). What am I missing?

ryao 11y ago
That applied to all system memory rather than what was in use. Memory errors in regions not in use are harmless while there are plenty of places where memory errors can occur in memory that is in use where they would not be noticed. Take the buffer storing this webpage in your web browser for example. A bit flip there would cause a misrendering that might not even be noticeable. Additionally, when a program crashes or the kernel panics, users are more likely to think it was caused by a bug in the software than a bitflip. In the rare cases where something goes catastrophically wrong, they tend to blame the software too.
- klagermkii 11y ago
  That's the theory in the article, that this silent corruption is happening all the time and it's just happening in unimportant places or ones that don't trigger a noticeable crash. But why am I not seeing this when explicitly looking for it? If I run Memtest86 for 24 hours I certainly don't expect to see an average of 22 errors and just treat that as within spec. I expect to see no errors. Is it a problem with the memory tests we're using, do we need ones which just write data once and then wait a day to read and validate it rather than hammering RAM with reads and writes?
  - lololomg 11y ago
    Memtest86 writes patterns to memory and reads them back almost immediately. To do the test properly you need to write a known pattern to memory, wait 24 hours, and THEN read it all back and check for flipped bits. If you do not have ECC in your computer, yes you should see some flips.
tzs 11y ago
It would be interesting to test this with a Raspberry Pi. Run a program that allocates a 512 MB buffer, initializes it to some known pattern, and then simply loops periodically reading the buffer and verifying the data. (Make sure that the compiler is not optimizing away the checks). That should see about 0.7 errors per day if they occur at the rate given in the article. If it is hard to get the compiler to not optimize away the verification, put the buffer in shared memory, and have the verify be a separate program that verifies once and exits, and then periodically invoke that. I did something like this for a long time on a home Linux server and never caught an error. However, I did not check the assembly output of the compiler to verify that it was actually reading the buffer, so my test may have been invalid.

ryao 11y ago

That applied to all system memory rather than what was in use. Memory errors in regions not in use are harmless while there are plenty of places where memory errors can occur in memory that is in use where they would not be noticed. Take the buffer storing this webpage in your web browser for example. A bit flip there would cause a misrendering that might not even be noticeable. Additionally, when a program crashes or the kernel panics, users are more likely to think it was caused by a bug in the software than a bitflip. In the rare cases where something goes catastrophically wrong, they tend to blame the software too.

klagermkii 11y ago
That's the theory in the article, that this silent corruption is happening all the time and it's just happening in unimportant places or ones that don't trigger a noticeable crash. But why am I not seeing this when explicitly looking for it? If I run Memtest86 for 24 hours I certainly don't expect to see an average of 22 errors and just treat that as within spec. I expect to see no errors. Is it a problem with the memory tests we're using, do we need ones which just write data once and then wait a day to read and validate it rather than hammering RAM with reads and writes?
- lololomg 11y ago
  Memtest86 writes patterns to memory and reads them back almost immediately. To do the test properly you need to write a known pattern to memory, wait 24 hours, and THEN read it all back and check for flipped bits. If you do not have ECC in your computer, yes you should see some flips.

tzs 11y ago

It would be interesting to test this with a Raspberry Pi. Run a program that allocates a 512 MB buffer, initializes it to some known pattern, and then simply loops periodically reading the buffer and verifying the data. (Make sure that the compiler is not optimizing away the checks). That should see about 0.7 errors per day if they occur at the rate given in the article. If it is hard to get the compiler to not optimize away the verification, put the buffer in shared memory, and have the verify be a separate program that verifies once and exits, and then periodically invoke that. I did something like this for a long time on a home Linux server and never caught an error. However, I did not check the assembly output of the compiler to verify that it was actually reading the buffer, so my test may have been invalid.