4 ms·

The crashes/bugs I deal with are rarely straight down failures, they are often the 1 out of 100 runs kind, so printf debugging is the only way to go really. And

The crashes/bugs I deal with are rarely straight down failures, they are often the 1 out of 100 runs kind, so printf debugging is the only way to go really. And I used to be big on using debuggers, but now I’m horribly out of practice.

skissane 2y ago
> The crashes/bugs I deal with are rarely straight down failures, they are often the 1 out of 100 runs kind, so printf debugging is the only way to go really. Another thing I’ve found helpful, is to write out a “system state dump” (say in JSON) to a file whenever certain errors happen. Like we had a production system that was randomly hanging and running out of DB connections. So now whenever the DB connection pool is exhausted, it dumps a JSON file to S3 listing the status of every DB connection, including the stack dump of the thread that owns it, the HTTP request that thread is serving, the user account, etc. Once we did that, it went from “we don’t understand why this application is randomly falling over” to “oh this is the thing that is consistently triggering it” When it writes a dump, it then starts a “lockout period” in which it won’t write any further dumps even if the error reoccurs. Don’t want to make a meltdown worse by getting bogged down endlessly writing out diagnostics.
- alextingle 2y ago
  Why not just dump a core, rather than going to all that effort?
  - skissane 2y ago
    Because it is a lot easier to analyse a few megabytes of JSON than a heap dump which is gigabytes in size. Not my original idea, I got it from some Oracle products which have a similar feature (most notably Oracle Database)
roca 2y ago
Record-and-replay debuggers are often better than anything else for debugging intermittent failures: run the program with recording many times until you eventually get the bug, then debug that recording at your leisure; you'll never have to waste time reproducing it again.
- johnisgood 2y ago
  I'm very rusty when it comes to debuggers. Where may I read about particular workflows involving debuggers, e.g. gdb? (Mainly for programs written in C.) Most of my issues are related to issues with concurrency though, deadlocks and whatnot.
throwaway2037 2y ago
> they are often the 1 out of 100 runs kind Can you share an example? In my whole career, I have only seen one or two of them, but most of my work is CRUD type of stuff, not really gaming or systems programming where such a thing might happen.
- arter4 2y ago
  Let's say your application talks to a database. You reuse connections with a connection pool, but you accidentally reuse connections with different privileges and scopes. As a result, sometimes you get to read some data you shouldn't read and sometimes you don't. Or, concurrency bugs. You don't properly serialize transactions and sometimes two transactions overlap in time leading to conflicts.
- seanmcdirmid 2y ago
  In Android they are way too common. Like an animation that decides to fire after the UI element it was made for has been destroyed, or a race condition in destroying a resource, or VE logging.

skissane 2y ago

> The crashes/bugs I deal with are rarely straight down failures, they are often the 1 out of 100 runs kind, so printf debugging is the only way to go really. Another thing I’ve found helpful, is to write out a “system state dump” (say in JSON) to a file whenever certain errors happen. Like we had a production system that was randomly hanging and running out of DB connections. So now whenever the DB connection pool is exhausted, it dumps a JSON file to S3 listing the status of every DB connection, including the stack dump of the thread that owns it, the HTTP request that thread is serving, the user account, etc. Once we did that, it went from “we don’t understand why this application is randomly falling over” to “oh this is the thing that is consistently triggering it” When it writes a dump, it then starts a “lockout period” in which it won’t write any further dumps even if the error reoccurs. Don’t want to make a meltdown worse by getting bogged down endlessly writing out diagnostics.

alextingle 2y ago
Why not just dump a core, rather than going to all that effort?
- skissane 2y ago
  Because it is a lot easier to analyse a few megabytes of JSON than a heap dump which is gigabytes in size. Not my original idea, I got it from some Oracle products which have a similar feature (most notably Oracle Database)

roca 2y ago

Record-and-replay debuggers are often better than anything else for debugging intermittent failures: run the program with recording many times until you eventually get the bug, then debug that recording at your leisure; you'll never have to waste time reproducing it again.

johnisgood 2y ago
I'm very rusty when it comes to debuggers. Where may I read about particular workflows involving debuggers, e.g. gdb? (Mainly for programs written in C.) Most of my issues are related to issues with concurrency though, deadlocks and whatnot.

throwaway2037 2y ago

> they are often the 1 out of 100 runs kind Can you share an example? In my whole career, I have only seen one or two of them, but most of my work is CRUD type of stuff, not really gaming or systems programming where such a thing might happen.

arter4 2y ago
Let's say your application talks to a database. You reuse connections with a connection pool, but you accidentally reuse connections with different privileges and scopes. As a result, sometimes you get to read some data you shouldn't read and sometimes you don't. Or, concurrency bugs. You don't properly serialize transactions and sometimes two transactions overlap in time leading to conflicts.
seanmcdirmid 2y ago
In Android they are way too common. Like an animation that decides to fire after the UI element it was made for has been destroyed, or a race condition in destroying a resource, or VE logging.