4 ms·
> I don't think the author has ever debugged a complex semantic/logic state bug, maybe even across multiple components or systems. That includes rare condition
by hodgesrm 7y ago
> I don't think the author has ever debugged a complex semantic/logic state bug, maybe even across multiple components or systems.
That includes rare conditions where you have to backtrack from error to root cause based on information already in the log. I worked for years on data replication and got into the habit of logging anything that might affect data integrity at INFO level to ensure it would be available to guide debugging.
Simple examples: where did you read config values from on startup? Log the file name including the full path. Rebooting?
Log all steps in the boot sequence in detail. Retries of calls to remote components? Log that too--put in a counter to log every N failures so you don't flood the log. Transaction aborts? Log them as well using the same counter method. Extra credit: Use ring buffers to dump current transaction state on fatal errors.
If things are OK the log is fairly small.
This kind of logging is essential in transaction processing systems. You can't log every transaction because (at least in my case) there were billions of them. But you should log everything that gives insight into whether the system is meeting preconditions for safety and liveness. If you need more info you can turn on traces but this will at least give you an idea where [not] to look.