6 ms·

It’s been known for more than 35 years that most bugs found in production are “Heisenbugs”: https://www.cs.columbia.edu/~junfeng/08fa-e6998/sched/readings/why-

It’s been known for more than 35 years that most bugs found in production are “Heisenbugs”: https://www.cs.columbia.edu/~junfeng/08fa-e6998/sched/readings/why-do-computers-stop.pdf https://www.cs.columbia.edu/~junfeng/08fa-e6998/sched/readin... Gray’s paper provides a logical reason why: If a bug can be reproduced reliably it’s much more likely to be found during testing and fixed, so appearing seemingly at random is a kind of Darwinian survival adaptation.

chii 4y ago
> If a bug can be reproduced reliably it’s much more likely to be found during testing and fixed which means in order to find as many bugs as possible during development, the dev environment must match the prod environment as exactly as possible. Bugs don't "evolve" like a life form - unless you code is self-modifying! It is caused by a permutation of all possible inputs (where the environment is also an "input"). Limiting possible inputs (such as functional programming) means you limit the possible differences between dev and prod - so functional programming ought to produce less bugs!
- WoahNoun 4y ago
  Or go to the opposite extreme and use fuzzing.
- angry_octet 4y ago
  Bugs don't evolve but the bug generators do. We build code with more and more complexity until we reach an equilibrium with our ability to debug the systems. If there is no competition in the evolution then systems can become arbitrarily baroque and fragile, which leads to rules like 'no software updates/patches' and 'all config changes via the CCB and pass mandatory tests' or 'your problem is handled manually', the wire wrap of systems development. Rebooting (the process or the computer) is trimming the state space. Model checking driven design really helps tame the tendency to let program state evolve without bound. FP tends to follow a bottom up design which makes this easier too.
- Akronymus 4y ago
  Or, if it is a system based on many components that can be independantly updated, some bugs may be introduced because a mismatch caused one variable to corrupt and then that one just keeps being propagated to newer versions until it triggers a bug because you assumed it can never have that specific value. Which, it actually can't in the current state, but only because of the system itself changin, or evolving, over time.
- paskozdilar 4y ago
  > Bugs don't "evolve" like a life form - unless you code is self-modifying! Bugs live in the codebase, and the codebase evolves (through human modifications). Every change in the codebase has a probability of introducing a new bug. If the introduced bug is deterministic and easy to catch, the probability of its survival is lower. If the introduced bug is nondeterministic and hard to catch, the probability of its survival is higher. So eventually, as time approaches infinity, the probability of a random bug in the codebase being nondeterministic approaches 100%.
fnordpiglet 4y ago
This is why I only test in production.
- midasuni 4y ago
  You test? How modern. I develop in prod, editing the code direct on the server. Users are the best testers anyway, they’ll phone if there’s a problem.
  - cardiffspaceman 4y ago
    The best is when you push your changes when your users are most likely to be invested in the correct behavior of the system.
    - fnordpiglet 4y ago
      The code is the correct behavior of the system. Users are deluded in their investments, sort of like crypto bros.
logbiscuitswave 4y ago
Heisenbugs are the worst. Just last week I was on the verge of literally ripping my hair out trying to figure out a frustrating bug that never occurred when running in the debugger. After a lot of frustration I had an idea: rather than starting the application in the debugger, I’d attach later. When doing this, I began to see a huge amount of interesting first-chance exceptions (this was managed C# code) that clued me into the root cause of the issue and I was finally able to solve the problem. I came to learn that an underlying library I was using had some code that ran on startup that actually detected if a debugger was attached and would turn off the problematic code path in this case! I came to learn that this was an intentional optimization the initialization of this code caused a bunch of noise during regular debugging (lots of first-chance exceptions that were ignorable). Unfortunately, this also meant that the interesting failure case I was running into was now completely hidden.
- bentcorner 4y ago
  Production code that checks if a debugger is attached should be illegal. Ideally there would be a way to get the debugger to lie to the process that a debugger is indeed not attached. Sure, maybe there's a reason today why you actually need to know a debugger isn't attached but at least give future-developers an escape hatch if they need it.
  - 10000truths 4y ago
    On Linux, you can "lie" to the process by running it with the LD_PRELOAD environment variable set to a library that hijacks ptrace(PTRACE_TRACEME, ...) to return 0. If the process is smart enough to check for LD_PRELOAD as well, you might be able to use something like seccomp to hijack the return value at the OS level.
    - virchau13 4y ago
      You don't need LD_PRELOAD; the ptrace mechanism by itself can intercept system calls and modify them on the fly.
  - 4y ago
legalcorrection 4y ago
Heisenbug is a very specific concept that refers to bugs that don’t reproduce because you are attempting to debug them. E.g. your debug printf statement changes the compiler output enough such that the bug no longer occurs with the same inputs. Or running the program in the debugger changes the thread scheduling, so the race condition is no longer hit. General difficulty reproducing is not what makes a heisenbug. What’s hard about heisenbugs is investigating their cause, not necessarily discovering and reproducing them.
- magicalhippo 4y ago
  One such instance is when debug mode causes memory allocations to be zeroed, while release mode doesn't. Always a recipe for a good time.
  - gjadi 4y ago
    Or your printf changes: - the timing of execution so nothing works => hello USB driver! - the scheduling of your threads - the memory layout => a memory overflow won't crash your program the same way or not at all I feel lucky I've enough hairs to pull them out on this kind of bugs. Fun time indeed.

chii 4y ago

> If a bug can be reproduced reliably it’s much more likely to be found during testing and fixed which means in order to find as many bugs as possible during development, the dev environment must match the prod environment as exactly as possible. Bugs don't "evolve" like a life form - unless you code is self-modifying! It is caused by a permutation of all possible inputs (where the environment is also an "input"). Limiting possible inputs (such as functional programming) means you limit the possible differences between dev and prod - so functional programming ought to produce less bugs!

WoahNoun 4y ago
Or go to the opposite extreme and use fuzzing.
angry_octet 4y ago
Bugs don't evolve but the bug generators do. We build code with more and more complexity until we reach an equilibrium with our ability to debug the systems. If there is no competition in the evolution then systems can become arbitrarily baroque and fragile, which leads to rules like 'no software updates/patches' and 'all config changes via the CCB and pass mandatory tests' or 'your problem is handled manually', the wire wrap of systems development. Rebooting (the process or the computer) is trimming the state space. Model checking driven design really helps tame the tendency to let program state evolve without bound. FP tends to follow a bottom up design which makes this easier too.
Akronymus 4y ago
Or, if it is a system based on many components that can be independantly updated, some bugs may be introduced because a mismatch caused one variable to corrupt and then that one just keeps being propagated to newer versions until it triggers a bug because you assumed it can never have that specific value. Which, it actually can't in the current state, but only because of the system itself changin, or evolving, over time.
paskozdilar 4y ago
> Bugs don't "evolve" like a life form - unless you code is self-modifying! Bugs live in the codebase, and the codebase evolves (through human modifications). Every change in the codebase has a probability of introducing a new bug. If the introduced bug is deterministic and easy to catch, the probability of its survival is lower. If the introduced bug is nondeterministic and hard to catch, the probability of its survival is higher. So eventually, as time approaches infinity, the probability of a random bug in the codebase being nondeterministic approaches 100%.

fnordpiglet 4y ago

This is why I only test in production.

midasuni 4y ago
You test? How modern. I develop in prod, editing the code direct on the server. Users are the best testers anyway, they’ll phone if there’s a problem.
- cardiffspaceman 4y ago
  The best is when you push your changes when your users are most likely to be invested in the correct behavior of the system.
  - fnordpiglet 4y ago
    The code is the correct behavior of the system. Users are deluded in their investments, sort of like crypto bros.

logbiscuitswave 4y ago

Heisenbugs are the worst. Just last week I was on the verge of literally ripping my hair out trying to figure out a frustrating bug that never occurred when running in the debugger. After a lot of frustration I had an idea: rather than starting the application in the debugger, I’d attach later. When doing this, I began to see a huge amount of interesting first-chance exceptions (this was managed C# code) that clued me into the root cause of the issue and I was finally able to solve the problem. I came to learn that an underlying library I was using had some code that ran on startup that actually detected if a debugger was attached and would turn off the problematic code path in this case! I came to learn that this was an intentional optimization the initialization of this code caused a bunch of noise during regular debugging (lots of first-chance exceptions that were ignorable). Unfortunately, this also meant that the interesting failure case I was running into was now completely hidden.

bentcorner 4y ago
Production code that checks if a debugger is attached should be illegal. Ideally there would be a way to get the debugger to lie to the process that a debugger is indeed not attached. Sure, maybe there's a reason today why you actually need to know a debugger isn't attached but at least give future-developers an escape hatch if they need it.
- 10000truths 4y ago
  On Linux, you can "lie" to the process by running it with the LD_PRELOAD environment variable set to a library that hijacks ptrace(PTRACE_TRACEME, ...) to return 0. If the process is smart enough to check for LD_PRELOAD as well, you might be able to use something like seccomp to hijack the return value at the OS level.
  - virchau13 4y ago
    You don't need LD_PRELOAD; the ptrace mechanism by itself can intercept system calls and modify them on the fly.
- 4y ago

legalcorrection 4y ago

Heisenbug is a very specific concept that refers to bugs that don’t reproduce because you are attempting to debug them. E.g. your debug printf statement changes the compiler output enough such that the bug no longer occurs with the same inputs. Or running the program in the debugger changes the thread scheduling, so the race condition is no longer hit. General difficulty reproducing is not what makes a heisenbug. What’s hard about heisenbugs is investigating their cause, not necessarily discovering and reproducing them.

magicalhippo 4y ago
One such instance is when debug mode causes memory allocations to be zeroed, while release mode doesn't. Always a recipe for a good time.
- gjadi 4y ago
  Or your printf changes: - the timing of execution so nothing works => hello USB driver! - the scheduling of your threads - the memory layout => a memory overflow won't crash your program the same way or not at all I feel lucky I've enough hairs to pull them out on this kind of bugs. Fun time indeed.