5 ms·

I get what you mean and mostly agree with it, but compiler / OS / hardware errors are ultimately not that rare. For example: Not too long ago, the latest Apple

I get what you mean and mostly agree with it, but compiler / OS / hardware errors are ultimately not that rare. For example: Not too long ago, the latest Apple clang simply crashed when compiling some C++ codebases I was working on. It was an obvious compiler bug, and we got a fix a few months later. During the 32-bit to 64-bit transition, it was common that large disk writes would fail silently because some layer truncated the length to 32 bits. Usually the culprit was the standard library, but I once saw it happening in the file system. For a long time, I assumed that OpenMP had some nontrivial memory overhead, because I often saw simple multithreaded data processing code using more memory than I would have expected. Then one time the overhead was tens of gigabytes and continued growing. When I investigated, it turned out to be degenerate behavior in glibc malloc/free. If you allocated and freed memory in different threads, you could end up with many fragmented arenas where the memory freed by the user could not be reused or released back to the OS. Bit flips and other memory errors seem to have become more common in the recent years, but only on consumer hardware. Maybe it's time to start using ECC memory everywhere.

anyfoo 5y ago
Yes, these bugs exist. But for the majority of developers they are usually so rare, that your working hypothesis should always be that something wrong happened on some upper layer, starting with your own code, unless you are downright forced to accept that there is no way that e.g. the CPU is behaving correctly. As for bit flips, I tend to wonder how many of those were in reality rarely triggering concurrency bugs, especially with memory ordering involved. I definitely saw some issues that could only be explained through a real bit flip or other spooky hardware failure (and we know of their definitive existence anyway), but it takes a very specific set of circumstances to fully exclude, say, a rare memory stomper. For example, an erratic bit set instructions whose operand address was fetched without the proper barrier or exclusive instruction, getting the address wrong in only 0.2% of common cases, and even then most of the time it would hit a word in memory where it doesn't matter or become apparent.
- aidenn0 5y ago
  > that your working hypothesis should always be that something wrong happened on some upper layer, starting with your own code, unless you are downright forced to accept that there is no way that e.g. the CPU is behaving correctly This is true for production hardware. I work with a lot of pre-release hardware so I start every bug with the latest errata open. Even then it's wrong to squint at the errata documentation until it might kind of look like your bug. Rather you do a quick look to make sure there's no clear smoking gun, and then continue to debug at the upper layer. It's ultimately usually not that hard to demonstrate an actual hardware bug with a reduced test-case once you have a general idea of what is wrong; if I had a nickle for each time I've heard "this is so random it must be a hardware bug" only to demonstrate it was a software race-condition, I would be a wealthy man.
  - anyfoo 5y ago
    Yup, I work with pre-release hardware, too, and you are spot on. The general audience I was addressing usually is not hit by CPU errata, and anyone who is should know about it.
Salgat 5y ago
Usually those bugs have such a large surface area that you can google and find a bunch of other people with the exact same issue. The issue are the folks who blame it without having concrete proof of that being the actual cause. "The Runtime is the issue? Alright point me to the Github issue or Stackoverflow entry that shows exactly that".
pronlover723 5y ago
large writes still fail. MacOS 12.3's finder can't reliably 0.3gig or larger files to network shares. 12.2 still works.

anyfoo 5y ago

Yes, these bugs exist. But for the majority of developers they are usually so rare, that your working hypothesis should always be that something wrong happened on some upper layer, starting with your own code, unless you are downright forced to accept that there is no way that e.g. the CPU is behaving correctly. As for bit flips, I tend to wonder how many of those were in reality rarely triggering concurrency bugs, especially with memory ordering involved. I definitely saw some issues that could only be explained through a real bit flip or other spooky hardware failure (and we know of their definitive existence anyway), but it takes a very specific set of circumstances to fully exclude, say, a rare memory stomper. For example, an erratic bit set instructions whose operand address was fetched without the proper barrier or exclusive instruction, getting the address wrong in only 0.2% of common cases, and even then most of the time it would hit a word in memory where it doesn't matter or become apparent.

aidenn0 5y ago
> that your working hypothesis should always be that something wrong happened on some upper layer, starting with your own code, unless you are downright forced to accept that there is no way that e.g. the CPU is behaving correctly This is true for production hardware. I work with a lot of pre-release hardware so I start every bug with the latest errata open. Even then it's wrong to squint at the errata documentation until it might kind of look like your bug. Rather you do a quick look to make sure there's no clear smoking gun, and then continue to debug at the upper layer. It's ultimately usually not that hard to demonstrate an actual hardware bug with a reduced test-case once you have a general idea of what is wrong; if I had a nickle for each time I've heard "this is so random it must be a hardware bug" only to demonstrate it was a software race-condition, I would be a wealthy man.
- anyfoo 5y ago
  Yup, I work with pre-release hardware, too, and you are spot on. The general audience I was addressing usually is not hit by CPU errata, and anyone who is should know about it.

Salgat 5y ago

Usually those bugs have such a large surface area that you can google and find a bunch of other people with the exact same issue. The issue are the folks who blame it without having concrete proof of that being the actual cause. "The Runtime is the issue? Alright point me to the Github issue or Stackoverflow entry that shows exactly that".

pronlover723 5y ago

large writes still fail. MacOS 12.3's finder can't reliably 0.3gig or larger files to network shares. 12.2 still works.