5 ms·

When was the last time your computer crashed because of a cosmic ray?

by cronz 8y ago

Scramblejams 8y ago
The problem isn't a crash, the problem is silent data corruption. I have a system with 128 gigs of ECC and it catches a soft bit flip or two about every 6 months.
- cronz 8y ago
  Most people don't have that amount of RAM. What are the chances that a soft bit flip will cause silent data corruption that matters for a user with, say, 8 GB of RAM?
  - Scramblejams 8y ago
    The answer to the question you've asked depends heavily on the particular user and their work, of course. If the user is just playing games, who cares? But maybe they're doing financial calculations or compiling software that lots of users run or maybe they're doing aircraft structural analysis. In any case, bit flips are much more common than were suspected: https://arstechnica.com/information-technology/2009/10/dram-study-turns-assumptions-about-errors-upside-down/ https://arstechnica.com/information-technology/2009/10/dram-... I believe strongly that ECC should be standard, because you can't safely assume that your users are doing worthless work. Apple got this right on (non-Mini) desktops a long time ago. Not yet on laptops, unfortunately.
    - monocasa 8y ago
      That study is faulty. The intern who did that study didn't know that Google would buy DRAM chips that failed manufacturers QA, but them on DIMMs themselves, and retest them at lower frequencies and with ECC turned on. When they already have to be tolerant of any node failing because of their scale, they can start playing fast and loose with this sort of thing if it makes financial sense. EDIT: At -3 so far, does anyone want to explain the downvotes? I saw the google slides first hand, and there are comments from 2009 in that article saying the same thing.
      - Scramblejams 8y ago
        I didn't downvote, but I'd guess that they (a) want an authoritative reference to the story you're sharing, or (b) figure the point is moot because these errors still happen, even if less often than depicted by the study, and that ought to be enough to justify using ECC. Like I said though, just a guess.
      - ianai 8y ago
        All ram should be ecc. No one would accept this incorrectness BS if the precedent hadn’t been set by the monopolist.
        leoc 8y ago
        It's really the memory manufacturers who are enforcing and profiting from the ECC shakedown though, AFAICS.
        ianai 8y ago
        Intel probably gets more from selling a Xeon.
      - dgacmu 8y ago
        Didn't down it either, but: "The intern" is a well-respected CS professor; the paper had two Google authors whom one would expect to have knowledge of Google's oddities. https://ai.google/research/pubs/pub35162 https://ai.google/research/pubs/pub35162 Your comment provided no substantiation of your claim, merely hand-waving, while casting aspersions on someone else's work.
    - ianai 8y ago
      If you’ve got files you care about then care about bit rot. I want ECC so my ZFS volumes don’t silently corrupt.
  - geezerjay 8y ago
    > Most people don't have that amount of RAM. Most people don't buy desktops with 16 or 32 or 64-thread processors designed to maximize throughput. Those who do tend to want to max out how much RAM they can shove in their box.
  - patrickg_zill 8y ago
    Bit flip in the dirty cache that gets written to disk? Depends on what data structure was corrupt...
  - eloff 8y ago
    As a counter example, my desktop has had 128 gb of ram since two years ago. I've never needed that much, 64GB would have been enough for the number of vms and containers that I run. But I couldn't pass up a good deal. I got the ram for $600.
    - ianai 8y ago
      Real statement of the times that I had a tough time finding 128 gig ram sets (on a short search - this should be easy).
  - berti 8y ago
    Run a filesystem that checksums data (e.g. ZFS) on a system without ECC RAM for a few weeks and then come back to us... I've done it, the results were surprising, and I replaced the RAM with ECC.
  - lathiat 8y ago
    If you think bit flipping is rare, check out this write-up: http://dinaburg.org/bitsquatting.html http://dinaburg.org/bitsquatting.html Bitsquatting: DNS Hijacking without exploitation When bit-errors occur they can change memory content. Computer memory content has semantic meaning. Sometimes, that meaning will be a domain name. And applications utilizing that memory will use the wrong domain name.
    - ianai 8y ago
      Nice point. As with all things that follow a distribution and given the large number of machines this error will occur.
Rebelgecko 8y ago
most bit flips won't cause a full-on crash, but may corrupt data in funny ways, or affect unused memory. I think people underestimate how common they are[1] [1]https://ai.google/research/pubs/pub35162 https://ai.google/research/pubs/pub35162
bzbarsky 8y ago
You computer? Not sure. Computers in general? A noticeable fraction of Firefox crash reports are due to single-bit memory flips.
- kjeetgill 8y ago
  Huh, this is really interesting. Can you go more into this? How do you know the cause of the error? What kinda numbers are we talking here? Also, are cosmic rays really the main source of single bit flips as apposed to just bad ram maybe?
  - bzbarsky 8y ago
    Firefox crash reports include the contents of registers and a few instructions around the instruction pointer. If the crash is in compiled code, not JIT code, you also know where in the binary you were and can get symbols from a symbol server, then compare what the instructions should be to what they actually were. Some of the time the instructions don't match up, indicating corruption _somewhere_. For the specific case of crashes in JIT-generated code, the contents of registers and the instructions can be related in various ways (e.g. if you have a jmp instruction the register better contain your code location). And if you know where your code locations might be (because you're a JIT, and are generating the code and aligning it in memory yourself) and the register with the code location looks like the sort of address you would end up with but with one extra low bit set, say... I am having trouble right now finding the bug report where some of the JIT engineers were analyzing crashes in jitcode, but about 1/3 of those were due to bitflips if I recall correctly. What that means in terms of absolute numbers (or numbers per user-hour, which would be even more useful), I don't know. Note, by the way, single-bit flips can be a consequence of a bad memory chip, not just of cosmic rays.
    - kjeetgill 8y ago
      Thanks! Incredibly interesting.

When was the last time your computer crashed because of a cosmic ray?

by cronz 8y ago

When was the last time your computer crashed because of a cosmic ray?

The problem isn't a crash, the problem is silent data corruption. I have a system with 128 gigs of ECC and it catches a soft bit flip or two about every 6 months.

cronz 8y ago
Most people don't have that amount of RAM. What are the chances that a soft bit flip will cause silent data corruption that matters for a user with, say, 8 GB of RAM?
- Scramblejams 8y ago
  The answer to the question you've asked depends heavily on the particular user and their work, of course. If the user is just playing games, who cares? But maybe they're doing financial calculations or compiling software that lots of users run or maybe they're doing aircraft structural analysis. In any case, bit flips are much more common than were suspected: https://arstechnica.com/information-technology/2009/10/dram-study-turns-assumptions-about-errors-upside-down/ https://arstechnica.com/information-technology/2009/10/dram-... I believe strongly that ECC should be standard, because you can't safely assume that your users are doing worthless work. Apple got this right on (non-Mini) desktops a long time ago. Not yet on laptops, unfortunately.
  - monocasa 8y ago
    That study is faulty. The intern who did that study didn't know that Google would buy DRAM chips that failed manufacturers QA, but them on DIMMs themselves, and retest them at lower frequencies and with ECC turned on. When they already have to be tolerant of any node failing because of their scale, they can start playing fast and loose with this sort of thing if it makes financial sense. EDIT: At -3 so far, does anyone want to explain the downvotes? I saw the google slides first hand, and there are comments from 2009 in that article saying the same thing.
    - Scramblejams 8y ago
      I didn't downvote, but I'd guess that they (a) want an authoritative reference to the story you're sharing, or (b) figure the point is moot because these errors still happen, even if less often than depicted by the study, and that ought to be enough to justify using ECC. Like I said though, just a guess.
    - ianai 8y ago
      All ram should be ecc. No one would accept this incorrectness BS if the precedent hadn’t been set by the monopolist.
      - leoc 8y ago
        It's really the memory manufacturers who are enforcing and profiting from the ECC shakedown though, AFAICS.
        ianai 8y ago
        Intel probably gets more from selling a Xeon.
    - dgacmu 8y ago
      Didn't down it either, but: "The intern" is a well-respected CS professor; the paper had two Google authors whom one would expect to have knowledge of Google's oddities. https://ai.google/research/pubs/pub35162 https://ai.google/research/pubs/pub35162 Your comment provided no substantiation of your claim, merely hand-waving, while casting aspersions on someone else's work.
  - ianai 8y ago
    If you’ve got files you care about then care about bit rot. I want ECC so my ZFS volumes don’t silently corrupt.
- geezerjay 8y ago
  > Most people don't have that amount of RAM. Most people don't buy desktops with 16 or 32 or 64-thread processors designed to maximize throughput. Those who do tend to want to max out how much RAM they can shove in their box.
- patrickg_zill 8y ago
  Bit flip in the dirty cache that gets written to disk? Depends on what data structure was corrupt...
- eloff 8y ago
  As a counter example, my desktop has had 128 gb of ram since two years ago. I've never needed that much, 64GB would have been enough for the number of vms and containers that I run. But I couldn't pass up a good deal. I got the ram for $600.
  - ianai 8y ago
    Real statement of the times that I had a tough time finding 128 gig ram sets (on a short search - this should be easy).
- berti 8y ago
  Run a filesystem that checksums data (e.g. ZFS) on a system without ECC RAM for a few weeks and then come back to us... I've done it, the results were surprising, and I replaced the RAM with ECC.
- lathiat 8y ago
  If you think bit flipping is rare, check out this write-up: http://dinaburg.org/bitsquatting.html http://dinaburg.org/bitsquatting.html Bitsquatting: DNS Hijacking without exploitation When bit-errors occur they can change memory content. Computer memory content has semantic meaning. Sometimes, that meaning will be a domain name. And applications utilizing that memory will use the wrong domain name.
  - ianai 8y ago
    Nice point. As with all things that follow a distribution and given the large number of machines this error will occur.

Rebelgecko 8y ago

most bit flips won't cause a full-on crash, but may corrupt data in funny ways, or affect unused memory. I think people underestimate how common they are[1] [1]https://ai.google/research/pubs/pub35162 https://ai.google/research/pubs/pub35162

bzbarsky 8y ago

You computer? Not sure. Computers in general? A noticeable fraction of Firefox crash reports are due to single-bit memory flips.

kjeetgill 8y ago
Huh, this is really interesting. Can you go more into this? How do you know the cause of the error? What kinda numbers are we talking here? Also, are cosmic rays really the main source of single bit flips as apposed to just bad ram maybe?
- bzbarsky 8y ago
  Firefox crash reports include the contents of registers and a few instructions around the instruction pointer. If the crash is in compiled code, not JIT code, you also know where in the binary you were and can get symbols from a symbol server, then compare what the instructions should be to what they actually were. Some of the time the instructions don't match up, indicating corruption _somewhere_. For the specific case of crashes in JIT-generated code, the contents of registers and the instructions can be related in various ways (e.g. if you have a jmp instruction the register better contain your code location). And if you know where your code locations might be (because you're a JIT, and are generating the code and aligning it in memory yourself) and the register with the code location looks like the sort of address you would end up with but with one extra low bit set, say... I am having trouble right now finding the bug report where some of the JIT engineers were analyzing crashes in jitcode, but about 1/3 of those were due to bitflips if I recall correctly. What that means in terms of absolute numbers (or numbers per user-hour, which would be even more useful), I don't know. Note, by the way, single-bit flips can be a consequence of a bad memory chip, not just of cosmic rays.
  - kjeetgill 8y ago
    Thanks! Incredibly interesting.