3 ms·

I don't think the author has ever debugged a complex semantic/logic state bug, maybe even across multiple components or systems. Relatedly, all they look at is

I don't think the author has ever debugged a complex semantic/logic state bug, maybe even across multiple components or systems. Relatedly, all they look at is error logging. Admittedly, there's a blurry line here between logging and tracing, but the latter word generally evokes externally attaching additional tools. But that "far end" of tracing is frequently not what's needed... there's a reason logging systems have "info" and "debug" severities. Either way good (i.e. configurable, low overhead) logging is worth its weight in gold.

xvector 7y ago
I don't even want to count how many nightmares distributed tracing has saved me from.
hinkley 7y ago
When I noticed that some of the people who complained loudly about certain practices also tended to disappear when a problem was too difficult for them to solve, I stopped letting what they think bother me. Bosses notice exceptional events. As the team or cluster get bigger, they start happening closer together. Depending on context, any event that happens once a week or once a day will be characterized as happening “all the time” and that creates a new set of problems. You need tools to head these off. And yes they are a pain but so is telling your boss, “we don’t know.”
- notduncansmith 7y ago
  > You need tools to head these off. And yes they are a pain but so is telling your boss, “we don’t know.” That is one of the feelings I dislike the most as someone who takes pride in building quality software. It’s important to be honest when you don’t know why something is wrong, but even more important to minimize the number of times it happens. Logging is incredibly beneficial for the latter, and complements metrics + exception-tracking.
hodgesrm 7y ago
> I don't think the author has ever debugged a complex semantic/logic state bug, maybe even across multiple components or systems. That includes rare conditions where you have to backtrack from error to root cause based on information already in the log. I worked for years on data replication and got into the habit of logging anything that might affect data integrity at INFO level to ensure it would be available to guide debugging. Simple examples: where did you read config values from on startup? Log the file name including the full path. Rebooting? Log all steps in the boot sequence in detail. Retries of calls to remote components? Log that too--put in a counter to log every N failures so you don't flood the log. Transaction aborts? Log them as well using the same counter method. Extra credit: Use ring buffers to dump current transaction state on fatal errors. If things are OK the log is fairly small. This kind of logging is essential in transaction processing systems. You can't log every transaction because (at least in my case) there were billions of them. But you should log everything that gives insight into whether the system is meeting preconditions for safety and liveness. If you need more info you can turn on traces but this will at least give you an idea where [not] to look.
j88439h84 7y ago
> there's a reason logging systems have "info" What is it? Performance?

xvector 7y ago

I don't even want to count how many nightmares distributed tracing has saved me from.

hinkley 7y ago

When I noticed that some of the people who complained loudly about certain practices also tended to disappear when a problem was too difficult for them to solve, I stopped letting what they think bother me. Bosses notice exceptional events. As the team or cluster get bigger, they start happening closer together. Depending on context, any event that happens once a week or once a day will be characterized as happening “all the time” and that creates a new set of problems. You need tools to head these off. And yes they are a pain but so is telling your boss, “we don’t know.”

notduncansmith 7y ago
> You need tools to head these off. And yes they are a pain but so is telling your boss, “we don’t know.” That is one of the feelings I dislike the most as someone who takes pride in building quality software. It’s important to be honest when you don’t know why something is wrong, but even more important to minimize the number of times it happens. Logging is incredibly beneficial for the latter, and complements metrics + exception-tracking.

hodgesrm 7y ago

> I don't think the author has ever debugged a complex semantic/logic state bug, maybe even across multiple components or systems. That includes rare conditions where you have to backtrack from error to root cause based on information already in the log. I worked for years on data replication and got into the habit of logging anything that might affect data integrity at INFO level to ensure it would be available to guide debugging. Simple examples: where did you read config values from on startup? Log the file name including the full path. Rebooting? Log all steps in the boot sequence in detail. Retries of calls to remote components? Log that too--put in a counter to log every N failures so you don't flood the log. Transaction aborts? Log them as well using the same counter method. Extra credit: Use ring buffers to dump current transaction state on fatal errors. If things are OK the log is fairly small. This kind of logging is essential in transaction processing systems. You can't log every transaction because (at least in my case) there were billions of them. But you should log everything that gives insight into whether the system is meeting preconditions for safety and liveness. If you need more info you can turn on traces but this will at least give you an idea where [not] to look.

j88439h84 7y ago

> there's a reason logging systems have "info" What is it? Performance?