4 ms·

The idea that cosmic rays are responsible for DRAM failures seems laughable. I'm surprised anybody ever thought that. I worked in a DRAM fab; I saw how it was

The idea that cosmic rays are responsible for DRAM failures seems laughable. I'm surprised anybody ever thought that. I worked in a DRAM fab; I saw how it was made and tested. It's a month-long process involving dozens of machines. Particulate contamination, watermarks, process variation -- there are tons of things that can go wrong. And then testing. There's just no way to test thoroughly enough to catch all of the possible issues with timing and crosstalk. Not to mention electromigration and dielectric breakdown. You can do burn-in, but it's not going to catch everything. We even know which die are most failure-prone (usually edge die, due to process uniformity issues.) If they pass test, they get shipped! Of course these issues are manufacturing related.

josaka 11y ago
This is true, re. manufacturing defects. I used to work in a DRAM fab, too, and observed similar patterns. Re. cosmic rays, it's surprising, but I think they can cause errors: http://stackoverflow.com/questions/2580933/cosmic-rays-what-is-the-probability-they-will-affect-a-program http://stackoverflow.com/questions/2580933/cosmic-rays-what-...
- mud_dauber 11y ago
  The risk is significant: JEDEC has a standard testing methodology for them: http://www.jedec.org/sites/default/files/docs/jesd89a.pdf http://www.jedec.org/sites/default/files/docs/jesd89a.pdf
  - sevensor 11y ago
    Interesting. Shows how my point of view colors the facts -- I never once heard one of our test engineers complain about cosmic rays, but then it was their job to blame process when defects came up. We used to try to blame dirty test heads, but that's another story. I'm sure in the real world cosmic rays actually do flip bits sometimes, but I'm equally sure that a significant proportion of the DRAM in production has latent defects. Given our observed defect rates and the limits of test coverage, it's unavoidable.
    - mikeash 11y ago
      For manufacturing defects, there are no doubt processes you can do to make them manifest more frequently so you can detect them. I imagine a lot of defects are temperature-dependent, so you're probably testing hot chips to force that. Errors while reading and writing can be made to happen more often by repeatedly reading and writing in a tight cycle. Errors from radio interference could be tested with artificial radio noise. The idea, I imagine, is to try to test the equivalent of X years of real-world use in only Y days of lab time. But cosmic rays are just a constant background thing. If an errant cosmic ray flips a bit, that doesn't happen any more often if you hammer on the chips or heat them up or expose them to radio waves or whatever. The only way to simulate X years of real-world use is to use them for X years, short of setting up a particle accelerator or something. (Technically, you could simulate X years of real-world cosmic ray exposure in X/N years by testing N sets of RAM in parallel, but then that requires you to have more ram under test than you give to your customers!) All the same, I'm sure more mundane explanations are behind the vast majority of errors, as you say.
      - sevensor 11y ago
        Yeah, we did hot tests as well as cold. The problem isn't whether you could detect defects given all the time and resources in the world, but whether you can maximize the probability of detection given finite resources. If you're in test, you've got to deal with an avalanche of material coming out of the fab, as well as constant pressure to get it out of the door and off the books. Upper management doesn't see you as adding value, but they'll come down on you like a ton of bricks if there's an incident and you don't catch it. It's a tough job, and I'm glad it wasn't mine.
- mud_dauber 11y ago
  Apparently it's a sufficiently common occurence that JEDEC has a standard testing methodology for just these types of events: http://www.jedec.org/sites/default/files/docs/jesd89a.pdf http://www.jedec.org/sites/default/files/docs/jesd89a.pdf
sufiyan 11y ago
If so, why dont they increase refresh rates given that most of the memory (>95%) can do refreshes at numbers of over 3x existing refresh rates. This could potentially increase the energy savings and also potentially make the system faster because of lesser waiting time during Refresh and Row Activation.
- yuhong 11y ago
  I think they already did. The 2x refresh option has existed since DDR2, and was originally designed for very high temperatures BTW.
- sevensor 11y ago
  Faster refresh will mitigate leakage, so you can run your DRAM hotter, but I don't see how that helps energy savings. Refresh means current flows out of the cell, through the sense amplifier, and then current flows back in, through the word driver. More refresh means more current. More current flow, more power consumption. This does not seem to argue for energy savings.
  - sufiyan 11y ago
    I meant reduce the refresh rate. My bad. Should have proof read what I typed. Essentially, what I meant was that many dram chips needn't be refreshed as often as they are now.
- Osiris 11y ago
  I read an article that explained that DRAM cannot be read or written to during a refresh cycle, so if you triple the frequency of the refresh, you'll substantially harm the performance and throughput of the RAM.
- PhantomGremlin 11y ago
  why dont they increase refresh rates Apple did exactly this to the EFI of some machines, to mitigate Rowhammer[1]: Description: A disturbance error, also known as Rowhammer, exists with some DDR3 RAM that could have led to memory corruption. This issue was mitigated by increasing memory refresh rates. [1] https://support.apple.com/en-us/HT204934 https://support.apple.com/en-us/HT204934

josaka 11y ago

This is true, re. manufacturing defects. I used to work in a DRAM fab, too, and observed similar patterns. Re. cosmic rays, it's surprising, but I think they can cause errors: http://stackoverflow.com/questions/2580933/cosmic-rays-what-is-the-probability-they-will-affect-a-program http://stackoverflow.com/questions/2580933/cosmic-rays-what-...

mud_dauber 11y ago
The risk is significant: JEDEC has a standard testing methodology for them: http://www.jedec.org/sites/default/files/docs/jesd89a.pdf http://www.jedec.org/sites/default/files/docs/jesd89a.pdf
- sevensor 11y ago
  Interesting. Shows how my point of view colors the facts -- I never once heard one of our test engineers complain about cosmic rays, but then it was their job to blame process when defects came up. We used to try to blame dirty test heads, but that's another story. I'm sure in the real world cosmic rays actually do flip bits sometimes, but I'm equally sure that a significant proportion of the DRAM in production has latent defects. Given our observed defect rates and the limits of test coverage, it's unavoidable.
  - mikeash 11y ago
    For manufacturing defects, there are no doubt processes you can do to make them manifest more frequently so you can detect them. I imagine a lot of defects are temperature-dependent, so you're probably testing hot chips to force that. Errors while reading and writing can be made to happen more often by repeatedly reading and writing in a tight cycle. Errors from radio interference could be tested with artificial radio noise. The idea, I imagine, is to try to test the equivalent of X years of real-world use in only Y days of lab time. But cosmic rays are just a constant background thing. If an errant cosmic ray flips a bit, that doesn't happen any more often if you hammer on the chips or heat them up or expose them to radio waves or whatever. The only way to simulate X years of real-world use is to use them for X years, short of setting up a particle accelerator or something. (Technically, you could simulate X years of real-world cosmic ray exposure in X/N years by testing N sets of RAM in parallel, but then that requires you to have more ram under test than you give to your customers!) All the same, I'm sure more mundane explanations are behind the vast majority of errors, as you say.
    - sevensor 11y ago
      Yeah, we did hot tests as well as cold. The problem isn't whether you could detect defects given all the time and resources in the world, but whether you can maximize the probability of detection given finite resources. If you're in test, you've got to deal with an avalanche of material coming out of the fab, as well as constant pressure to get it out of the door and off the books. Upper management doesn't see you as adding value, but they'll come down on you like a ton of bricks if there's an incident and you don't catch it. It's a tough job, and I'm glad it wasn't mine.
mud_dauber 11y ago
Apparently it's a sufficiently common occurence that JEDEC has a standard testing methodology for just these types of events: http://www.jedec.org/sites/default/files/docs/jesd89a.pdf http://www.jedec.org/sites/default/files/docs/jesd89a.pdf

sufiyan 11y ago

If so, why dont they increase refresh rates given that most of the memory (>95%) can do refreshes at numbers of over 3x existing refresh rates. This could potentially increase the energy savings and also potentially make the system faster because of lesser waiting time during Refresh and Row Activation.

yuhong 11y ago
I think they already did. The 2x refresh option has existed since DDR2, and was originally designed for very high temperatures BTW.
sevensor 11y ago
Faster refresh will mitigate leakage, so you can run your DRAM hotter, but I don't see how that helps energy savings. Refresh means current flows out of the cell, through the sense amplifier, and then current flows back in, through the word driver. More refresh means more current. More current flow, more power consumption. This does not seem to argue for energy savings.
- sufiyan 11y ago
  I meant reduce the refresh rate. My bad. Should have proof read what I typed. Essentially, what I meant was that many dram chips needn't be refreshed as often as they are now.
Osiris 11y ago
I read an article that explained that DRAM cannot be read or written to during a refresh cycle, so if you triple the frequency of the refresh, you'll substantially harm the performance and throughput of the RAM.
PhantomGremlin 11y ago
why dont they increase refresh rates Apple did exactly this to the EFI of some machines, to mitigate Rowhammer[1]: Description: A disturbance error, also known as Rowhammer, exists with some DDR3 RAM that could have led to memory corruption. This issue was mitigated by increasing memory refresh rates. [1] https://support.apple.com/en-us/HT204934 https://support.apple.com/en-us/HT204934