4 ms·

This problem has no ultimate solution. I've seen all components flip bits, CPUs, networking cards, RAM, most often you just can't know for sure what did it. You

This problem has no ultimate solution. I've seen all components flip bits, CPUs, networking cards, RAM, most often you just can't know for sure what did it. You can remedy it a bit (like with ECC), but ultimately there will always be corruption if you process hundreds of petabytes of data. Get used to it, your computer executes an instruction with a probability extremely close to 1, but not equal to 1. Deep in the archives of a well known tech company is a very well documented case of a bit flip that caused the wrong function to be executed in a C++ v-table. The big oof was that this function was the equivalent of an SQL "drop table", and just happened to be 32 bytes off of a very benign function that did something like stat(). Really funny stuff once the crisis is over :)

water8 4y ago
[dead]
don-code 4y ago
Is this something that's documented publicly? I'd love to read more.
dale_glass 4y ago
ECC isn't a terribly complicated technology, and can be used in all those cases. In limited cases, a checksum is good enough. If you checksum outgoing data, and verify it on reception, then it being corrupted in transit whether on the network card or the cable can be detected and transparently compensated for. Really, we can do much better than to "get used to it".
- H8crilA 4y ago
  You are under the impression that CPUs and other chips always perform the same instructions as are written in the code, and only RAM can flip bits because DRAM is DRAM :) It can (and should! whenever possible) be improved, not fixed. There's always that pesky gamma that can hit a specific transistor, even if it is deep underground. Gamma cannot be fully stopped. At certain scales data corruption becomes directly measurable. And yes, corruption levels vary between pieces of hardware.
  - dale_glass 4y ago
    Of course not. I'm not saying we can have perfection. I'm saying that we can do much better, using methods and technologies that are very old at this point. The reason why we don't is laziness and market segmentation, mostly.
  - sobriquet9 4y ago
    The probability of a bit flip depends on the size of the transistor used. RAM tends to pack many small transistors.
  - ilyt 4y ago
    Sure but once your registers, cache, data bus and address bus has ECC you have vastly smaller area that can flip. You can even just buy (well, chipaggedon aside) ARM cores that have 2 chips running in parallel and faulting when the result is different
    - my123 4y ago
      > You can even just buy (well, chipaggedon aside) ARM cores that have 2 chips running in parallel and faulting when the result is different See dual-core lock-step Arm chips (used for automotive).
  - keybored 4y ago
    OP said: We can do better You replied: it should be improved, not (it cannot be completely 100%) fixed Your error: replying to a strawman
  - to11mtm 4y ago
    > You are under the impression that CPUs and other chips always perform the same instructions as are written in the code, and only RAM can flip bits because DRAM is DRAM :) I thought L2/L3 is ECC (at least on Intel, though L1 I think is parity only)
_0w8t 4y ago
One can process petabytes without bit flips if one use proper checksums and error correction codes. While that does imposes overhead, it is not big and, thanks to Shannon theorem, can be made arbitrary small with sufficiently big blocks.
nlewycky 4y ago
If this is the same issue at the same well known tech company that I recall, it wasn't a hardware failure but a case of undefined behaviour. The existing code used memset to initialize a struct with zeroes, then someone else added a virtual method (to it? to a base? or member?) which caused the struct to grow a vptr. Now the memset was clobbering the vptr. They compiled this program with some form of link time optimization and the compiler emitted assembly that tested the vptr against one of two known values, and when it didn't match (because it was NULL) it must be the other known value! Either way it didn't use the vtable, it just made direct calls to the appropriate member functions, which led to a "drop table". As part of resolution it also led to https://reviews.llvm.org/rL130299 https://reviews.llvm.org/rL130299 for compile-time checking and also UBSan's `-fsanitize=vptr` for runtime checking.
TomSwirly 4y ago
> ultimately there will always be corruption if you process hundreds of petabytes of data. Your statement is in practical terms false. Error correction exists, and adding enough redundancy to make any corruption have such a tiny chance of being undetected that the whole history of the universe could go by without it happening is a problem that was solved over 60 years ago with the invention of the Reed-Solomon code,

water8 4y ago

[dead]

don-code 4y ago

Is this something that's documented publicly? I'd love to read more.

dale_glass 4y ago

ECC isn't a terribly complicated technology, and can be used in all those cases. In limited cases, a checksum is good enough. If you checksum outgoing data, and verify it on reception, then it being corrupted in transit whether on the network card or the cable can be detected and transparently compensated for. Really, we can do much better than to "get used to it".

H8crilA 4y ago
You are under the impression that CPUs and other chips always perform the same instructions as are written in the code, and only RAM can flip bits because DRAM is DRAM :) It can (and should! whenever possible) be improved, not fixed. There's always that pesky gamma that can hit a specific transistor, even if it is deep underground. Gamma cannot be fully stopped. At certain scales data corruption becomes directly measurable. And yes, corruption levels vary between pieces of hardware.
- dale_glass 4y ago
  Of course not. I'm not saying we can have perfection. I'm saying that we can do much better, using methods and technologies that are very old at this point. The reason why we don't is laziness and market segmentation, mostly.
- sobriquet9 4y ago
  The probability of a bit flip depends on the size of the transistor used. RAM tends to pack many small transistors.
- ilyt 4y ago
  Sure but once your registers, cache, data bus and address bus has ECC you have vastly smaller area that can flip. You can even just buy (well, chipaggedon aside) ARM cores that have 2 chips running in parallel and faulting when the result is different
  - my123 4y ago
    > You can even just buy (well, chipaggedon aside) ARM cores that have 2 chips running in parallel and faulting when the result is different See dual-core lock-step Arm chips (used for automotive).
- keybored 4y ago
  OP said: We can do better You replied: it should be improved, not (it cannot be completely 100%) fixed Your error: replying to a strawman
- to11mtm 4y ago
  > You are under the impression that CPUs and other chips always perform the same instructions as are written in the code, and only RAM can flip bits because DRAM is DRAM :) I thought L2/L3 is ECC (at least on Intel, though L1 I think is parity only)

_0w8t 4y ago

One can process petabytes without bit flips if one use proper checksums and error correction codes. While that does imposes overhead, it is not big and, thanks to Shannon theorem, can be made arbitrary small with sufficiently big blocks.

nlewycky 4y ago

If this is the same issue at the same well known tech company that I recall, it wasn't a hardware failure but a case of undefined behaviour. The existing code used memset to initialize a struct with zeroes, then someone else added a virtual method (to it? to a base? or member?) which caused the struct to grow a vptr. Now the memset was clobbering the vptr. They compiled this program with some form of link time optimization and the compiler emitted assembly that tested the vptr against one of two known values, and when it didn't match (because it was NULL) it must be the other known value! Either way it didn't use the vtable, it just made direct calls to the appropriate member functions, which led to a "drop table". As part of resolution it also led to https://reviews.llvm.org/rL130299 https://reviews.llvm.org/rL130299 for compile-time checking and also UBSan's `-fsanitize=vptr` for runtime checking.

TomSwirly 4y ago

> ultimately there will always be corruption if you process hundreds of petabytes of data. Your statement is in practical terms false. Error correction exists, and adding enough redundancy to make any corruption have such a tiny chance of being undetected that the whole history of the universe could go by without it happening is a problem that was solved over 60 years ago with the invention of the Reed-Solomon code,