3 ms·
Yes, these bugs exist. But for the majority of developers they are usually so rare, that your working hypothesis should always be that something wrong happened
by anyfoo 4y ago
Yes, these bugs exist. But for the majority of developers they are usually so rare, that your working hypothesis should always be that something wrong happened on some upper layer, starting with your own code, unless you are downright forced to accept that there is no way that e.g. the CPU is behaving correctly.
As for bit flips, I tend to wonder how many of those were in reality rarely triggering concurrency bugs, especially with memory ordering involved. I definitely saw some issues that could only be explained through a real bit flip or other spooky hardware failure (and we know of their definitive existence anyway), but it takes a very specific set of circumstances to fully exclude, say, a rare memory stomper. For example, an erratic bit set instructions whose operand address was fetched without the proper barrier or exclusive instruction, getting the address wrong in only 0.2% of common cases, and even then most of the time it would hit a word in memory where it doesn't matter or become apparent.
- aidenn0 4y ago> that your working hypothesis should always be that something wrong happened on some upper layer, starting with your own code, unless you are downright forced to accept that there is no way that e.g. the CPU is behaving correctly This is true for production hardware. I work with a lot of pre-release hardware so I start every bug with the latest errata open. Even then it's wrong to squint at the errata documentation until it might kind of look like your bug. Rather you do a quick look to make sure there's no clear smoking gun, and then continue to debug at the upper layer. It's ultimately usually not that hard to demonstrate an actual hardware bug with a reduced test-case once you have a general idea of what is wrong; if I had a nickle for each time I've heard "this is so random it must be a hardware bug" only to demonstrate it was a software race-condition, I would be a wealthy man.
- anyfoo 4y agoYup, I work with pre-release hardware, too, and you are spot on. The general audience I was addressing usually is not hit by CPU errata, and anyone who is should know about it.