5 ms·

Multiple times I've had bugs that disappeared when print debugging was added. To say that that was frustrating is an understatement. (But where there's a will,

Multiple times I've had bugs that disappeared when print debugging was added. To say that that was frustrating is an understatement. (But where there's a will, there's a way. It just takes longer, sigh.)

layer8 5y ago
Those are known as Heisenbugs. https://en.wikipedia.org/wiki/Heisenbug https://en.wikipedia.org/wiki/Heisenbug
8note 5y ago
Sounds like you solved the problem though. Leave the printing in there:P
loopz 5y ago
In multi-threaded code or using interrupts (ie. clock or I/O), it's pretty obvious what causes this. Fix can be hard though, but absolutely required for robust execution. In single-thread code, I've often noticed my assumptions about the code paths were wrong. So print debugging actually improved my understanding of what the logic really is doing. Then, either improving the process, or my assumptions/memory of it. Many times debugging like that removes subtle bugs and mistakes that otherwise would've remained invisible and dormant. Usually when something is off about the execution, however long it takes, it's usually my own assumptions that's wrong. The program logic just does what it's told, and is usually void of off by one errors and the like. If there's subtle bugs, it's often shallow object clones or mistakes handling complex datastructures that seemingly works. Debugging is just a tool though, use whatever suits you best. I do like the idea of improved tools like RR. However, you often delve into languages that don't have them, so print debugging remains a useful hammer lying around.
- jes 5y ago
  Fully agree with this. More often than not bug is in the computer between my ears rather than the computer on my bench.
slaymaker1907 5y ago
One trick to minimize debugging footprint is to store logs in a ring buffer/array list and defer printing. It's not perfect, but if you use a ring buffer, you can avoid making any syscalls until you are truly ready to print. Thread synchronization can obviously still be a problem, but that can be mitigated by using thread local buffers or by using a lockless ring buffer. Ring buffers are also nice because you can also use them for cases where you don't normally want some information printed because it would introduce too much overhead, but you want the option to print it out in case of crashes/errors. On an engineering side, you just need to be careful not to put too much through this system so the buffer stays useful and also avoid putting something only in the ring buffer system that should really just be logged normally.
- nefitty 5y ago
  This is genius. I hadn’t heard of this logging style before. I’m going to try it tomorrow morning.
  - pjc50 5y ago
    A variant I developed in an embedded system, which had weirdly defective debugging facilities: define an area of memory which is not cleared on startup. Log into a circular buffer there. On startup, output the contents of the buffer over the serial link (or similar).
  - hyperman1 5y ago
    There are many variants. One is to put the ringbuffer in a shared memory segment or mmapped file, and use another process to read and print the content.

layer8 5y ago

Those are known as Heisenbugs. https://en.wikipedia.org/wiki/Heisenbug https://en.wikipedia.org/wiki/Heisenbug

8note 5y ago

Sounds like you solved the problem though. Leave the printing in there:P

loopz 5y ago

In multi-threaded code or using interrupts (ie. clock or I/O), it's pretty obvious what causes this. Fix can be hard though, but absolutely required for robust execution. In single-thread code, I've often noticed my assumptions about the code paths were wrong. So print debugging actually improved my understanding of what the logic really is doing. Then, either improving the process, or my assumptions/memory of it. Many times debugging like that removes subtle bugs and mistakes that otherwise would've remained invisible and dormant. Usually when something is off about the execution, however long it takes, it's usually my own assumptions that's wrong. The program logic just does what it's told, and is usually void of off by one errors and the like. If there's subtle bugs, it's often shallow object clones or mistakes handling complex datastructures that seemingly works. Debugging is just a tool though, use whatever suits you best. I do like the idea of improved tools like RR. However, you often delve into languages that don't have them, so print debugging remains a useful hammer lying around.

jes 5y ago
Fully agree with this. More often than not bug is in the computer between my ears rather than the computer on my bench.

slaymaker1907 5y ago

One trick to minimize debugging footprint is to store logs in a ring buffer/array list and defer printing. It's not perfect, but if you use a ring buffer, you can avoid making any syscalls until you are truly ready to print. Thread synchronization can obviously still be a problem, but that can be mitigated by using thread local buffers or by using a lockless ring buffer. Ring buffers are also nice because you can also use them for cases where you don't normally want some information printed because it would introduce too much overhead, but you want the option to print it out in case of crashes/errors. On an engineering side, you just need to be careful not to put too much through this system so the buffer stays useful and also avoid putting something only in the ring buffer system that should really just be logged normally.

nefitty 5y ago
This is genius. I hadn’t heard of this logging style before. I’m going to try it tomorrow morning.
- pjc50 5y ago
  A variant I developed in an embedded system, which had weirdly defective debugging facilities: define an area of memory which is not cleared on startup. Log into a circular buffer there. On startup, output the contents of the buffer over the serial link (or similar).
- hyperman1 5y ago
  There are many variants. One is to put the ringbuffer in a shared memory segment or mmapped file, and use another process to read and print the content.