4 ms·

When you say bitflips were "common" on thousands of physical machines, does that mean you observed thousands of bitflips? Otherwise, I would think that an unli

When you say bitflips were "common" on thousands of physical machines, does that mean you observed thousands of bitflips? Otherwise, I would think that an unlikely event becoming 1000x more likely by sheer numbers would have warped your perception. I believe that hardware reliability is mostly irrelevant, because software reliability is already far worse. It doesn't matter whether a bitflip (unlikely) or some bug (likely) causes a node to spuriously fail, what matters is that this failure is handled gracefully.

felixhandte 6y ago
Yes. I work at Facebook on data compression. The libraries we maintain (1) are responsible for a non-trivial part of Facebook's overall compute footprint, (2) should basically never fail of their own accord, and (3) have pretty good error monitoring. So my team is operating what is effectively (among other things) a very sensitive detector for hardware failure. And indeed we see examples all the time of blobs that fail to decompress, and usually when we dig in we find that the blob is only a single bit-flip away from a blob that decompresses successfully into a syntactically correct message. I can't share numbers, but, off the top of my head, I think it's the largest source of failures we see. It happens frequently enough that I wrote a tool to automate checking [0]. So yes. It happens. Pretty frequently, in the sense that if you're doing xillions of operations a day, a one-in-a-xillion failure happens all the time. [0] https://github.com/facebook/zstd/tree/dev/contrib/diagnose_corruption https://github.com/facebook/zstd/tree/dev/contrib/diagnose_c...
dkersten 6y ago
Another comment[1] mentioned 1 bitflip per gigabyte per month. If you have a lot of RAM, that's rather a lot. > It doesn't matter whether a bitflip (unlikely) or some bug (likely) causes a node to spuriously fail Except that a bitflip can go undetected. It may crash your software or system, but it also may simply leak errors into your data, which can be far more catastrophic. [1] https://news.ycombinator.com/item?id=25623206 https://news.ycombinator.com/item?id=25623206
- thequux 6y ago
  I have a server with 384GB of RAM sitting next to me, and in two months of uptime with the only shielding being a pile of IBM iron on top of it, it has detected a grand total of 0 errors. This is, of course, an anecdote rather than data, but 0 is different enough from the expected 768 that it makes me doubt that statistic.
- jhasse 6y ago
  So can a bug.
  - dkersten 6y ago
    Yes. And? That doesn't suddenly make bitflips benign.
    - incrudible 6y ago
      The point is that you can't prevent failure by just buying something. You have to deal with the fact that failure can not be prevented. In other words, if a single defective DIMM somewhere in your deployment is causing catastraphic failure, your mistake was not buying the wrong RAM modules. Your mistake was relying on a single point of failure for mission critical data.
      - hda2 6y ago
        People might not be preventing failure by using ECC, but they significantly decrease the likelihood of having to deal with hard-to-debug problems caused by bitflips.
tyoma 6y ago
It depends where the failure happens. Sometimes you really lose the “failure in the wrong place” lottery. For example, in a domain name: http://dinaburg.org/bitsquatting.html http://dinaburg.org/bitsquatting.html
jjeaff 6y ago
Ya, I'm not buying that biyflips are a problem. Or maybe modern software can correct better for this? Because I use my desktop all day everyday running tons of software on 64 gb of ram and I don't get errors or crashes often enough to remember ever having one.
- ChrisLomont 6y ago
  > I'm not buying that biyflips are a problem. Google and read up - it is a problem, has killed people, has thrown election results, and much more. It's such a common problem than bitsquatting is a real thing :) Want to do an experiment? Pick a bitsquatted domain for a common site, and see how often you get hits. https://en.wikipedia.org/wiki/Bitsquatting https://en.wikipedia.org/wiki/Bitsquatting
  - incrudible 6y ago
    Nobody denies that bitflips happen. On the whole, you fail to make a case that preventing bitflips is the solution to a problem. Bitsquatting is not a real problem, it's a curiosity. As for the case of bitflips killing someone: Bitflips are not the root cause here. The root cause is that somebody engineered something life-critical that mistakenly assumed hardware can not fail. Bitflips are just one of many reasons for hardware failure.
    - ChrisLomont 6y ago
      >Bitflips are not the root cause here. So those systems didn't fail when a bitflip happened? > The root cause is that somebody engineered something life-critical that mistakenly assumed hardware can not fail. The systems I am aware of were designed with bitflips in mind. NO software can handle arbitrary amounts of bitflips. ALL software designed to mitigate bitflips only lower the odds via various forms of redundancy. (For context, I've written code for NASA, written a few proposals on making things more radiation hardened, and my PhD thesis was on a new class of error correcting codes - so I do know a little about making redundant software and hardware specifically designed to mitigate bitflips). By claiming a bitflip didn't kick off the problems, and trying to push the cause elsewhere, you may as well blame all of engineering for making a device that can kill on failure. So your argument is a red herring >On the whole, you fail to make a case that preventing bitflips is the solution to a problem Yes, had those bitflips been prevented, or not happened, those fatalities would not have happened. >Ya, I'm not buying that biyflips are a problem. If bitflips are not a problem then we don't need ECC ram (or ECC almost anything!) which is clearly used a lot. So bitflips are enough of a problem that a massively widespread technology is in place to handle precisely that problem. I guess you've never written a program and watched bits flip on computers you control? You should try it - it's a good exercise to see how often it does happen. I guess you define something being a problem differently than I or the ECC ram industry do.
ikiris 6y ago
Its enough that graphs can show you solar weather. I can't give my source, but its far higher than most people think. Just pay the money.
adrian_b 6y ago
On a single computer with a large memory, e.g. 32 GB or more, the time between errors can be of a few months, if you are lucky to have good modules. Moreover, some of the errors will have no effect, if they happened to affect free memory. Nevertheless, anyone who uses the computer for anything else besides games or movie watching, will greatly benefit from having ECC memory, because that is the only way to learn when the memory modules become defective. Modern memories have a shorter lifetime than old memories and very frequently they begin to have bit errors from time to time long before breaking down completely. Without ECC, you will become aware that a memory module is defective only when the computer crashes or no longer boots and severe data corruption in your files could have happened some months before that. For myself, this was the most obvious reason why ECC was useful, because I was able in several cases to replace memory modules that began to have frequent correctable errors, after many years with little or no errors, without losing any precious data and without downtime.
- ikiris 6y ago
  The good modules bit is important. I'm told by some colleagues that most of the bit flips are from alpha particles from the ram casings surprisingly enough.
- dannyw 6y ago
  Plus, ECC RAM is so accessible these days thanks to Ryzen. All the asrock mobos, from the lowest end to the highest end, advertise official support for it.

felixhandte 6y ago

Yes. I work at Facebook on data compression. The libraries we maintain (1) are responsible for a non-trivial part of Facebook's overall compute footprint, (2) should basically never fail of their own accord, and (3) have pretty good error monitoring. So my team is operating what is effectively (among other things) a very sensitive detector for hardware failure. And indeed we see examples all the time of blobs that fail to decompress, and usually when we dig in we find that the blob is only a single bit-flip away from a blob that decompresses successfully into a syntactically correct message. I can't share numbers, but, off the top of my head, I think it's the largest source of failures we see. It happens frequently enough that I wrote a tool to automate checking [0]. So yes. It happens. Pretty frequently, in the sense that if you're doing xillions of operations a day, a one-in-a-xillion failure happens all the time. [0] https://github.com/facebook/zstd/tree/dev/contrib/diagnose_corruption https://github.com/facebook/zstd/tree/dev/contrib/diagnose_c...

dkersten 6y ago

Another comment[1] mentioned 1 bitflip per gigabyte per month. If you have a lot of RAM, that's rather a lot. > It doesn't matter whether a bitflip (unlikely) or some bug (likely) causes a node to spuriously fail Except that a bitflip can go undetected. It may crash your software or system, but it also may simply leak errors into your data, which can be far more catastrophic. [1] https://news.ycombinator.com/item?id=25623206 https://news.ycombinator.com/item?id=25623206

thequux 6y ago
I have a server with 384GB of RAM sitting next to me, and in two months of uptime with the only shielding being a pile of IBM iron on top of it, it has detected a grand total of 0 errors. This is, of course, an anecdote rather than data, but 0 is different enough from the expected 768 that it makes me doubt that statistic.
jhasse 6y ago
So can a bug.
- dkersten 6y ago
  Yes. And? That doesn't suddenly make bitflips benign.
  - incrudible 6y ago
    The point is that you can't prevent failure by just buying something. You have to deal with the fact that failure can not be prevented. In other words, if a single defective DIMM somewhere in your deployment is causing catastraphic failure, your mistake was not buying the wrong RAM modules. Your mistake was relying on a single point of failure for mission critical data.
    - hda2 6y ago
      People might not be preventing failure by using ECC, but they significantly decrease the likelihood of having to deal with hard-to-debug problems caused by bitflips.

tyoma 6y ago

It depends where the failure happens. Sometimes you really lose the “failure in the wrong place” lottery. For example, in a domain name: http://dinaburg.org/bitsquatting.html http://dinaburg.org/bitsquatting.html

jjeaff 6y ago

Ya, I'm not buying that biyflips are a problem. Or maybe modern software can correct better for this? Because I use my desktop all day everyday running tons of software on 64 gb of ram and I don't get errors or crashes often enough to remember ever having one.

ChrisLomont 6y ago
> I'm not buying that biyflips are a problem. Google and read up - it is a problem, has killed people, has thrown election results, and much more. It's such a common problem than bitsquatting is a real thing :) Want to do an experiment? Pick a bitsquatted domain for a common site, and see how often you get hits. https://en.wikipedia.org/wiki/Bitsquatting https://en.wikipedia.org/wiki/Bitsquatting
- incrudible 6y ago
  Nobody denies that bitflips happen. On the whole, you fail to make a case that preventing bitflips is the solution to a problem. Bitsquatting is not a real problem, it's a curiosity. As for the case of bitflips killing someone: Bitflips are not the root cause here. The root cause is that somebody engineered something life-critical that mistakenly assumed hardware can not fail. Bitflips are just one of many reasons for hardware failure.
  - ChrisLomont 6y ago
    >Bitflips are not the root cause here. So those systems didn't fail when a bitflip happened? > The root cause is that somebody engineered something life-critical that mistakenly assumed hardware can not fail. The systems I am aware of were designed with bitflips in mind. NO software can handle arbitrary amounts of bitflips. ALL software designed to mitigate bitflips only lower the odds via various forms of redundancy. (For context, I've written code for NASA, written a few proposals on making things more radiation hardened, and my PhD thesis was on a new class of error correcting codes - so I do know a little about making redundant software and hardware specifically designed to mitigate bitflips). By claiming a bitflip didn't kick off the problems, and trying to push the cause elsewhere, you may as well blame all of engineering for making a device that can kill on failure. So your argument is a red herring >On the whole, you fail to make a case that preventing bitflips is the solution to a problem Yes, had those bitflips been prevented, or not happened, those fatalities would not have happened. >Ya, I'm not buying that biyflips are a problem. If bitflips are not a problem then we don't need ECC ram (or ECC almost anything!) which is clearly used a lot. So bitflips are enough of a problem that a massively widespread technology is in place to handle precisely that problem. I guess you've never written a program and watched bits flip on computers you control? You should try it - it's a good exercise to see how often it does happen. I guess you define something being a problem differently than I or the ECC ram industry do.

ikiris 6y ago

Its enough that graphs can show you solar weather. I can't give my source, but its far higher than most people think. Just pay the money.

adrian_b 6y ago

On a single computer with a large memory, e.g. 32 GB or more, the time between errors can be of a few months, if you are lucky to have good modules. Moreover, some of the errors will have no effect, if they happened to affect free memory. Nevertheless, anyone who uses the computer for anything else besides games or movie watching, will greatly benefit from having ECC memory, because that is the only way to learn when the memory modules become defective. Modern memories have a shorter lifetime than old memories and very frequently they begin to have bit errors from time to time long before breaking down completely. Without ECC, you will become aware that a memory module is defective only when the computer crashes or no longer boots and severe data corruption in your files could have happened some months before that. For myself, this was the most obvious reason why ECC was useful, because I was able in several cases to replace memory modules that began to have frequent correctable errors, after many years with little or no errors, without losing any precious data and without downtime.

ikiris 6y ago
The good modules bit is important. I'm told by some colleagues that most of the bit flips are from alpha particles from the ram casings surprisingly enough.
dannyw 6y ago
Plus, ECC RAM is so accessible these days thanks to Ryzen. All the asrock mobos, from the lowest end to the highest end, advertise official support for it.