3 ms·

> A so-called "senior" developer started screaming at the compiler, then at the IDE, then at the operating system, then at his colleagues. He was frustrated. T

by anyfoo 4y ago

> A so-called "senior" developer started screaming at the compiler, then at the IDE, then at the operating system, then at his colleagues. He was frustrated. This is one of the worst traps to fall into. I call it out whenever I can to people who fall into this: It's never the compiler, it's never the CPU, and if you're an application developer, it's never the OS. And if it is you can only get to that conclusion by assuming it still isn't, unless, Sherlock Holmes style, you are left with no choice. Never let it be your working hypothesis, always try to find out how those things working correctly matches your observations instead. Working on very low level code, I do run into actual compiler and CPU bugs, and just two weeks ago or so I deeply regretted assuming something to be a CPU bug in an obscure part of it towards the end of a lengthy bug investigation, after the gathered data clearly suggested it was the CPU misbehaving. It still wasn't: I missed a crucial half-sentence in the spec.

CGamesPlay 4y ago
I got very skilled by learning this lesson! Then I started coding in JavaScript, and now I find it’s only my code about half the time :-/
SemanticStrengh 4y ago
In some niche cases it can be the tooling. I remember having code that ran well with ASAN and ~segfaulted (or code was non-functional, don't remember) when I finally decided to disable it. The irony.. I won't say the cause yet, so that you can try to guess. Indice: there was a problem in the code, but the runtime behavior was correct and ASAN didn't show any error/warning.
jltsiren 4y ago
I get what you mean and mostly agree with it, but compiler / OS / hardware errors are ultimately not that rare. For example: Not too long ago, the latest Apple clang simply crashed when compiling some C++ codebases I was working on. It was an obvious compiler bug, and we got a fix a few months later. During the 32-bit to 64-bit transition, it was common that large disk writes would fail silently because some layer truncated the length to 32 bits. Usually the culprit was the standard library, but I once saw it happening in the file system. For a long time, I assumed that OpenMP had some nontrivial memory overhead, because I often saw simple multithreaded data processing code using more memory than I would have expected. Then one time the overhead was tens of gigabytes and continued growing. When I investigated, it turned out to be degenerate behavior in glibc malloc/free. If you allocated and freed memory in different threads, you could end up with many fragmented arenas where the memory freed by the user could not be reused or released back to the OS. Bit flips and other memory errors seem to have become more common in the recent years, but only on consumer hardware. Maybe it's time to start using ECC memory everywhere.
- anyfoo 4y ago
  Yes, these bugs exist. But for the majority of developers they are usually so rare, that your working hypothesis should always be that something wrong happened on some upper layer, starting with your own code, unless you are downright forced to accept that there is no way that e.g. the CPU is behaving correctly. As for bit flips, I tend to wonder how many of those were in reality rarely triggering concurrency bugs, especially with memory ordering involved. I definitely saw some issues that could only be explained through a real bit flip or other spooky hardware failure (and we know of their definitive existence anyway), but it takes a very specific set of circumstances to fully exclude, say, a rare memory stomper. For example, an erratic bit set instructions whose operand address was fetched without the proper barrier or exclusive instruction, getting the address wrong in only 0.2% of common cases, and even then most of the time it would hit a word in memory where it doesn't matter or become apparent.
  - aidenn0 4y ago
    > that your working hypothesis should always be that something wrong happened on some upper layer, starting with your own code, unless you are downright forced to accept that there is no way that e.g. the CPU is behaving correctly This is true for production hardware. I work with a lot of pre-release hardware so I start every bug with the latest errata open. Even then it's wrong to squint at the errata documentation until it might kind of look like your bug. Rather you do a quick look to make sure there's no clear smoking gun, and then continue to debug at the upper layer. It's ultimately usually not that hard to demonstrate an actual hardware bug with a reduced test-case once you have a general idea of what is wrong; if I had a nickle for each time I've heard "this is so random it must be a hardware bug" only to demonstrate it was a software race-condition, I would be a wealthy man.
    - anyfoo 4y ago
      Yup, I work with pre-release hardware, too, and you are spot on. The general audience I was addressing usually is not hit by CPU errata, and anyone who is should know about it.
- 4y ago
hibikir 4y ago
Sometimes really it is the tool underneath you that is broken: I've been fortunate enough to not hit a CPU bug, but I sure have hit OS bugs, and definitely database bugs, and runtime environment bugs. In those cases, however, the bugs have to be understood as features, because your chances of changing them might be very low. If your database doesn't match documented behavior, or sometimes refuses to return the right thing, the application developer has to find a way to get the result they need anyway. You can report the bug, or if you are lucky send a PR, but until it's all approved and released, bugs have to be worked around and accepted: Little is fixed by screaming.
mike_hock 4y ago
It's never the compiler, the OS or the CPU. But if you're working with a shit framework and after countless times prudently assuming the bug must be in your code only to be forced to finally conclude (with a stripped down repro) that some basic functionality is broken in the framework, then let it be your working hypothesis that it's the framework.
hackeraccount 4y ago
This is not limited to software debugging, it's a life problem. When facing a problem you can either look to yourself (painful because it means both work and facing your own personal limitations) or to outside things that you have not responsibility for and limited control over. The worst part is that sometimes it really is one of those outside things.

by anyfoo 4y ago

CGamesPlay 4y ago

I got very skilled by learning this lesson! Then I started coding in JavaScript, and now I find it’s only my code about half the time :-/

SemanticStrengh 4y ago

In some niche cases it can be the tooling. I remember having code that ran well with ASAN and ~segfaulted (or code was non-functional, don't remember) when I finally decided to disable it. The irony.. I won't say the cause yet, so that you can try to guess. Indice: there was a problem in the code, but the runtime behavior was correct and ASAN didn't show any error/warning.

jltsiren 4y ago

I get what you mean and mostly agree with it, but compiler / OS / hardware errors are ultimately not that rare. For example: Not too long ago, the latest Apple clang simply crashed when compiling some C++ codebases I was working on. It was an obvious compiler bug, and we got a fix a few months later. During the 32-bit to 64-bit transition, it was common that large disk writes would fail silently because some layer truncated the length to 32 bits. Usually the culprit was the standard library, but I once saw it happening in the file system. For a long time, I assumed that OpenMP had some nontrivial memory overhead, because I often saw simple multithreaded data processing code using more memory than I would have expected. Then one time the overhead was tens of gigabytes and continued growing. When I investigated, it turned out to be degenerate behavior in glibc malloc/free. If you allocated and freed memory in different threads, you could end up with many fragmented arenas where the memory freed by the user could not be reused or released back to the OS. Bit flips and other memory errors seem to have become more common in the recent years, but only on consumer hardware. Maybe it's time to start using ECC memory everywhere.

anyfoo 4y ago
Yes, these bugs exist. But for the majority of developers they are usually so rare, that your working hypothesis should always be that something wrong happened on some upper layer, starting with your own code, unless you are downright forced to accept that there is no way that e.g. the CPU is behaving correctly. As for bit flips, I tend to wonder how many of those were in reality rarely triggering concurrency bugs, especially with memory ordering involved. I definitely saw some issues that could only be explained through a real bit flip or other spooky hardware failure (and we know of their definitive existence anyway), but it takes a very specific set of circumstances to fully exclude, say, a rare memory stomper. For example, an erratic bit set instructions whose operand address was fetched without the proper barrier or exclusive instruction, getting the address wrong in only 0.2% of common cases, and even then most of the time it would hit a word in memory where it doesn't matter or become apparent.
- aidenn0 4y ago
  > that your working hypothesis should always be that something wrong happened on some upper layer, starting with your own code, unless you are downright forced to accept that there is no way that e.g. the CPU is behaving correctly This is true for production hardware. I work with a lot of pre-release hardware so I start every bug with the latest errata open. Even then it's wrong to squint at the errata documentation until it might kind of look like your bug. Rather you do a quick look to make sure there's no clear smoking gun, and then continue to debug at the upper layer. It's ultimately usually not that hard to demonstrate an actual hardware bug with a reduced test-case once you have a general idea of what is wrong; if I had a nickle for each time I've heard "this is so random it must be a hardware bug" only to demonstrate it was a software race-condition, I would be a wealthy man.
  - anyfoo 4y ago
    Yup, I work with pre-release hardware, too, and you are spot on. The general audience I was addressing usually is not hit by CPU errata, and anyone who is should know about it.
4y ago

hibikir 4y ago

Sometimes really it is the tool underneath you that is broken: I've been fortunate enough to not hit a CPU bug, but I sure have hit OS bugs, and definitely database bugs, and runtime environment bugs. In those cases, however, the bugs have to be understood as features, because your chances of changing them might be very low. If your database doesn't match documented behavior, or sometimes refuses to return the right thing, the application developer has to find a way to get the result they need anyway. You can report the bug, or if you are lucky send a PR, but until it's all approved and released, bugs have to be worked around and accepted: Little is fixed by screaming.

mike_hock 4y ago

It's never the compiler, the OS or the CPU. But if you're working with a shit framework and after countless times prudently assuming the bug must be in your code only to be forced to finally conclude (with a stripped down repro) that some basic functionality is broken in the framework, then let it be your working hypothesis that it's the framework.

hackeraccount 4y ago

This is not limited to software debugging, it's a life problem. When facing a problem you can either look to yourself (painful because it means both work and facing your own personal limitations) or to outside things that you have not responsibility for and limited control over. The worst part is that sometimes it really is one of those outside things.