4 ms·

Here I was hoping this was something about repurposing RAM as an actual geiger counter by measuring radiation-induced bit flips.

s9w 6y ago
Actually me too. I did write a program a while back to investigate the rumors about this though. Spoiler alert: Doesn't really happen. (https://github.com/s9w/MCC https://github.com/s9w/MCC)
- faeyanpiraat 6y ago
  I’m mildly disappointed aswell.
- bArray 6y ago
  Well, I'm also happy this doesn't happen! RAM is designed to be stable! If you have the resources, have you considered getting a source of radiation and putting it by the RAM? In a plane for example, background radiation can be 30x or so. It might also reduce the time to see a flip... Other considerations: 1. The act of checking the RAM (I believe) refreshes the memory cells. 2. I suspect the act of regularly changing the RAM bits to have an effect on the data stored. I think when the bit transitions from a 0 to 1 or vice versa, some inconvenient piece of solar radiation could bias it to a different state. That said, you would have to separate that from accidental row-hammer events. 3. You would think that higher temperature RAM was more likely to bit flip (the Scientific term is more wiggly electrons). 4. RAM clock speed. The higher it is, the less time it has to stabilize at the transistor level. 5. Memory cell size. The smaller it is, the more likely it will be affected. There are probably tonnes of variables, but these are the ones I can think of for now.
  - s9w 6y ago
    That's actually a couple of interesting ideas. I don't think however that 1) is completely correct? As far as I remember from my classes, RAMs are refreshed all the time on very small time scales exactly because they're volatile. Therefore a manual read shouldn't influence things. Temperatures and clock speed would be very interesting. Not sure I want to even google for radiation sources though :D
    - formerly_proven 6y ago
      DRAM reads are destructive, so reading a row always refreshes it. On the other hand, every row is refreshed every ~64 ms. (This is governed by tREFI, the interval between successive rows being refreshed. For example a module with 8K rows would require a tREFI of 64 ms / 8K = 7.8 us = 12480 clocks (7.8 us / (1600 MHz)^-1)). Increasing tREFI is one of those things people do to improve memory performance, because less time spent refreshing means more time available for useful work. Some memory chips can use very high values, which would be equivalent to ~200 ms, others corrupt data consistently much before that. So while rows are refreshed very often, a specific row is only refreshed ten times per second or so.
      - s9w 6y ago
        Interesting, thanks for the numbers. That would mean that it would probably be feasible to significantly influence the mean refresh times.
- apankrat 6y ago
  Bit flips in RAM happen often enough to be an issue. I make a desktop program with a sizeable installation base and it uses a form paged on-disk storage for larger data sets. The code has a fair amount of asserts in it and one specific assert would keep failing in a very tiny percentage of the cases. I kept reviewing the code and scratching my head before finally adding some extra logging for this case. Lo and behold, in all failed cases an argument passed to a function had one of the bits set for no apparent reason. I.e. the offsets of pages in the cache would be: 00000000000000 ... 00000000100000 ... 00000000200000 ... 00000000300000 ... But the args would come in as 80000000100000 00040000200000 01000000300000 ... Followed up with the people who reported these, asked to run an overnight RAM test and ALL of them found issues with their RAM. That was quite an eye-opener.
  - Someone 6y ago
    You weren’t explicitly saying it, but some might infer it, so I think it’s worth mentioning that bit flips in RAM aren’t necessarily radiation-induced. Some other causes are heat and aging (both increase capacitor leakage, and DRAM cells are tiny capacitors), or lack of robustness in refresh logic.
    - quercusa 6y ago
      Another cause, at least in the old days, was bad board design, in which badly-placed traces could induce currents in other traces. Of course this only happened two+ hours into the test.
  - s9w 6y ago
    I remember a blog post by Riot (league of legends developer) also about this happening. But I couldn't find it again when I was searching for it. I can't imagine the havoc that regular bit flips must cause.
  - tyingq 6y ago
    One of the more interesting ways to test this is to register a one character typo of a very popular domain where the typo is one bit off. Like, say, dacebook.com or facebooo.com Spin up a web server, and you'll get traffic, but typically only one of the requests from a specific client.
- dnautics 6y ago
  but did you put a radiation source next to your non-ecc ram?
01100011 6y ago
Probably better to use a CCD with electrical tape over the lens. I'm curious to see the difference in sensitivity between RAM and a CCD. I would think RAM is pretty resistant to bit flips or measures to counteract it, like ECC, would be more common.
- 7373737373 6y ago
  This sounds like it could be a cheap, software-based alternative to expensive radiometric hardware, interesting!
- 4gotunameagain 6y ago
  At CERN I used to work on radiation measurement and characterization using ram chips (two different types with different sensitivities to neutron energies), but also characterisation with a ccd. With the ccd you can distinguish the type of the particle based on the track that it leaves behind, but the problem is the very low cross section (area, sensitivity) of the chip.
  - pletnes 6y ago
    To get a track the trajectory has to be in the CCD plane? Doesn’t that mean abysmal efficiency?
    - corty 6y ago
      You can also get a trajectory with a stack of CCDs where the trajectory is not in the plane of the CCD. Actually the stack may be very high with quite some distance between individual CCDs, because you need trajectory in and out of detector elements like calorimeters that are a few meters in size. Also, trajectory may be bent because of the particle's charge and a magnetic field, for that you need a large distance to get the "amount of bend" if it is small. But yes, there always is an efficiency, precision and identification tradeoff.
  - mcejp 6y ago
    Hello, colleague! For folks who might be interested in more details on this: The _cross section_ allows you to quantify the frequency of a specific event occurring in your chip when bombarded by a specific particle (or class of particles). For example, in a case of an SRAM, you might want to calculate the cross-section for bit flips due to 24-GeV protons. The result that you get will be in units of cm^2; you can also interpret it as the physical surface area of the chip, multiplied by the probability (0..1) that any 1 particle passing through will trigger an event. Let's say you measure a cross section of 0.01 mm^2. Then if you expose that component to a flux of 10^3 particles/cm^2/sec, you will get an average of 1 event every 10 seconds. The cross-section is only valid for one combination of the (event type, particle, energy) tuple. But often in dosimetry, errors on the order of 20% are acceptable, so we can "abuse" the results a bit. Within one manufacturing lot of chips, the cross-sections should be reasonably consistent. Typically you would estimate the mean and variance by measuring a few samples from the lot. Then, to accurately measure a radiation field using these findings, you would generally need to measure the flux of every particle-energy combination separately. This is impractical, and for SEE* concerns we generally get away with distinguishing thermal neutrons and high-energy hadrons (which are separated by several orders of magnitude in energy). For this, it is sufficient to have 2 types of SRAM where each has a dominant cross-section for one of the particle classes of interest. To measure the type of radiation that causes long-term gradual damage (TID) such as X-ray and gamma rays, you would use a different principle of measurement altogether (RadFET, P-i-N diode, floating-gate dosimeter...) Radiation measurement can be fun! [*] single-event effects, including bit flips and latch-ups
gebruikersnaam 6y ago
One of my favorite programming stories : http://archive.is/O70Rg http://archive.is/O70Rg
segfaultbuserr 6y ago
According to HN user idoh, it's indeed possible, and it's used in serious military systems. https://news.ycombinator.com/item?id=3179852 https://news.ycombinator.com/item?id=3179852 > So, my mom is a former Maxwell employee who now does her own defense contracting, and makes this same type of part. > As I understand it, when there is a nuclear event, it generates x-rays followed by the EMP. The goal is to have warheads in flight to be able to continue to their target, so the strategy is to employ an NED. When an event is detected, the warhead shuts down its electronics for the duration of the EMP, and then powers back up. > With her NED, she uses an ASIC for detection. I might get some details wrong, but the ASIC has a physical array in it, and the x-rays flip bits in the array. When enough bits get flipped, one can infer a nuclear event. Because it is an ASIC it is really small, which has important advantages for space-born avionics.
- deleted 6y ago
  [deleted]
- coolspot 6y ago
  Thanks, Кomrade, we now know that electronics of latest americaН nukes can be dizabЛed by x-ray emissИons.
dnautics 6y ago
I had exactly the same thought.
mshockwave 6y ago
same here

s9w 6y ago

Actually me too. I did write a program a while back to investigate the rumors about this though. Spoiler alert: Doesn't really happen. (https://github.com/s9w/MCC https://github.com/s9w/MCC)

faeyanpiraat 6y ago
I’m mildly disappointed aswell.
bArray 6y ago
Well, I'm also happy this doesn't happen! RAM is designed to be stable! If you have the resources, have you considered getting a source of radiation and putting it by the RAM? In a plane for example, background radiation can be 30x or so. It might also reduce the time to see a flip... Other considerations: 1. The act of checking the RAM (I believe) refreshes the memory cells. 2. I suspect the act of regularly changing the RAM bits to have an effect on the data stored. I think when the bit transitions from a 0 to 1 or vice versa, some inconvenient piece of solar radiation could bias it to a different state. That said, you would have to separate that from accidental row-hammer events. 3. You would think that higher temperature RAM was more likely to bit flip (the Scientific term is more wiggly electrons). 4. RAM clock speed. The higher it is, the less time it has to stabilize at the transistor level. 5. Memory cell size. The smaller it is, the more likely it will be affected. There are probably tonnes of variables, but these are the ones I can think of for now.
- s9w 6y ago
  That's actually a couple of interesting ideas. I don't think however that 1) is completely correct? As far as I remember from my classes, RAMs are refreshed all the time on very small time scales exactly because they're volatile. Therefore a manual read shouldn't influence things. Temperatures and clock speed would be very interesting. Not sure I want to even google for radiation sources though :D
  - formerly_proven 6y ago
    DRAM reads are destructive, so reading a row always refreshes it. On the other hand, every row is refreshed every ~64 ms. (This is governed by tREFI, the interval between successive rows being refreshed. For example a module with 8K rows would require a tREFI of 64 ms / 8K = 7.8 us = 12480 clocks (7.8 us / (1600 MHz)^-1)). Increasing tREFI is one of those things people do to improve memory performance, because less time spent refreshing means more time available for useful work. Some memory chips can use very high values, which would be equivalent to ~200 ms, others corrupt data consistently much before that. So while rows are refreshed very often, a specific row is only refreshed ten times per second or so.
    - s9w 6y ago
      Interesting, thanks for the numbers. That would mean that it would probably be feasible to significantly influence the mean refresh times.
apankrat 6y ago
Bit flips in RAM happen often enough to be an issue. I make a desktop program with a sizeable installation base and it uses a form paged on-disk storage for larger data sets. The code has a fair amount of asserts in it and one specific assert would keep failing in a very tiny percentage of the cases. I kept reviewing the code and scratching my head before finally adding some extra logging for this case. Lo and behold, in all failed cases an argument passed to a function had one of the bits set for no apparent reason. I.e. the offsets of pages in the cache would be: 00000000000000 ... 00000000100000 ... 00000000200000 ... 00000000300000 ... But the args would come in as 80000000100000 00040000200000 01000000300000 ... Followed up with the people who reported these, asked to run an overnight RAM test and ALL of them found issues with their RAM. That was quite an eye-opener.
- Someone 6y ago
  You weren’t explicitly saying it, but some might infer it, so I think it’s worth mentioning that bit flips in RAM aren’t necessarily radiation-induced. Some other causes are heat and aging (both increase capacitor leakage, and DRAM cells are tiny capacitors), or lack of robustness in refresh logic.
  - quercusa 6y ago
    Another cause, at least in the old days, was bad board design, in which badly-placed traces could induce currents in other traces. Of course this only happened two+ hours into the test.
- s9w 6y ago
  I remember a blog post by Riot (league of legends developer) also about this happening. But I couldn't find it again when I was searching for it. I can't imagine the havoc that regular bit flips must cause.
- tyingq 6y ago
  One of the more interesting ways to test this is to register a one character typo of a very popular domain where the typo is one bit off. Like, say, dacebook.com or facebooo.com Spin up a web server, and you'll get traffic, but typically only one of the requests from a specific client.
dnautics 6y ago
but did you put a radiation source next to your non-ecc ram?

01100011 6y ago

Probably better to use a CCD with electrical tape over the lens. I'm curious to see the difference in sensitivity between RAM and a CCD. I would think RAM is pretty resistant to bit flips or measures to counteract it, like ECC, would be more common.

7373737373 6y ago
This sounds like it could be a cheap, software-based alternative to expensive radiometric hardware, interesting!
4gotunameagain 6y ago
At CERN I used to work on radiation measurement and characterization using ram chips (two different types with different sensitivities to neutron energies), but also characterisation with a ccd. With the ccd you can distinguish the type of the particle based on the track that it leaves behind, but the problem is the very low cross section (area, sensitivity) of the chip.
- pletnes 6y ago
  To get a track the trajectory has to be in the CCD plane? Doesn’t that mean abysmal efficiency?
  - corty 6y ago
    You can also get a trajectory with a stack of CCDs where the trajectory is not in the plane of the CCD. Actually the stack may be very high with quite some distance between individual CCDs, because you need trajectory in and out of detector elements like calorimeters that are a few meters in size. Also, trajectory may be bent because of the particle's charge and a magnetic field, for that you need a large distance to get the "amount of bend" if it is small. But yes, there always is an efficiency, precision and identification tradeoff.
- mcejp 6y ago
  Hello, colleague! For folks who might be interested in more details on this: The _cross section_ allows you to quantify the frequency of a specific event occurring in your chip when bombarded by a specific particle (or class of particles). For example, in a case of an SRAM, you might want to calculate the cross-section for bit flips due to 24-GeV protons. The result that you get will be in units of cm^2; you can also interpret it as the physical surface area of the chip, multiplied by the probability (0..1) that any 1 particle passing through will trigger an event. Let's say you measure a cross section of 0.01 mm^2. Then if you expose that component to a flux of 10^3 particles/cm^2/sec, you will get an average of 1 event every 10 seconds. The cross-section is only valid for one combination of the (event type, particle, energy) tuple. But often in dosimetry, errors on the order of 20% are acceptable, so we can "abuse" the results a bit. Within one manufacturing lot of chips, the cross-sections should be reasonably consistent. Typically you would estimate the mean and variance by measuring a few samples from the lot. Then, to accurately measure a radiation field using these findings, you would generally need to measure the flux of every particle-energy combination separately. This is impractical, and for SEE* concerns we generally get away with distinguishing thermal neutrons and high-energy hadrons (which are separated by several orders of magnitude in energy). For this, it is sufficient to have 2 types of SRAM where each has a dominant cross-section for one of the particle classes of interest. To measure the type of radiation that causes long-term gradual damage (TID) such as X-ray and gamma rays, you would use a different principle of measurement altogether (RadFET, P-i-N diode, floating-gate dosimeter...) Radiation measurement can be fun! [*] single-event effects, including bit flips and latch-ups

gebruikersnaam 6y ago

One of my favorite programming stories : http://archive.is/O70Rg http://archive.is/O70Rg

segfaultbuserr 6y ago

According to HN user idoh, it's indeed possible, and it's used in serious military systems. https://news.ycombinator.com/item?id=3179852 https://news.ycombinator.com/item?id=3179852 > So, my mom is a former Maxwell employee who now does her own defense contracting, and makes this same type of part. > As I understand it, when there is a nuclear event, it generates x-rays followed by the EMP. The goal is to have warheads in flight to be able to continue to their target, so the strategy is to employ an NED. When an event is detected, the warhead shuts down its electronics for the duration of the EMP, and then powers back up. > With her NED, she uses an ASIC for detection. I might get some details wrong, but the ASIC has a physical array in it, and the x-rays flip bits in the array. When enough bits get flipped, one can infer a nuclear event. Because it is an ASIC it is really small, which has important advantages for space-born avionics.

deleted 6y ago
[deleted]
coolspot 6y ago
Thanks, Кomrade, we now know that electronics of latest americaН nukes can be dizabЛed by x-ray emissИons.

dnautics 6y ago

I had exactly the same thought.

mshockwave 6y ago

same here