4 ms·
I think, like most people, I will disagree with this: > In my opinion, my thesis from GopherCon 2014 still holds: services should only log actionable informati
by jacques_chester 11y ago
I think, like most people, I will disagree with this:
> In my opinion, my thesis from GopherCon 2014 still holds: services should only log actionable information.
This assumes that I have future knowledge of which logged events will be actionable.
I used to work in the classifieds department of a newspaper. The rule of thumb was that the cover of the newspaper was worthless by 11am, the classifieds by 2pm.
But that's not quite true. Every single edition of a newspaper retains a small probability of turning out to be very important at some unknown future time. Hence the practice of archiving.
So it is with logs. In my day job I have on several occasions turned to raw logs in order to reconstruct the history of unexpected events.
What metric, to take an actual example, tells me that a user was trying to evade export control by hopping to a VPN halfway through a session? Which gauge tells me that an error was caused by the interaction of third party javascript and an obscure web browser?
For myself, I draw the boundary differently. Logging is for events with identity, events that will be inspected individually. Metrics are for events that will never have identity and will only be understood statistically.
- sagichmal 11y agoIMO the author buried the lede: > Finally, understand that logging is expensive. I’ve seen entire teams of absolutely brilliant engineers spend years building, managing, and evolving logging infrastructure. It’s a hard problem, made much harder by overburdening the pipelines with unnecessary load. It's ultimately a piece about engineering cost versus benefit. There's a real cost associated with logging every event with identity, to create an audit trail. Maybe that's necessary in your domain, for compliance reasons or otherwise. In which case it falls into the bucket of machine-read structured logging, and you design for it. But for lots of domains, it isn't strictly necessary.
- jacques_chester 11y agoIn which light, allow me to rephrase the argument: Log as much as you can afford. Given that the cost of logging is low and continuously falling, the default decision should be "log everything". "Log only actionable messages" is analogous to advice like "always use bit masks instead of booleans", or only "use two digits to encode years". It assumes algo-economics that are no longer the normal case.
- sagichmal 11y ago> Given that the cost of logging is low and continuously falling Everything gets cheaper in time, but the cost of logging is actually quite high.
- tonyarkles 11y agoI don't know why you're getting downvoted, because in my mind you're right in some circumstances. More than once, I've found logging-related performance issues (i.e. logging is near the top of a profiling-driven performance optimization task). One immediate example that comes to mind was code that had a bunch of logging statements like log.debug(boost::format("%d %x etc", param, param)). Normally the code would run with debug logging turned off, but there's a pretty brutal problem there! The formatted string is always going to be processed, which the logger will immediately discard. Turns out one of these formatted strings actually took a fair bit of processing to generate and was hurting the system.
- kedean 11y agoThat's a solved problem. Your logger should be capable of formatting on it's own, and take the arguments to the format string as variadic arguments. This is how Java's ubiquitous SLF4J interface is defined. If the log level isn't low enough, then the arguments are tossed but never composited.
- azth 11y agoAnd it's even better with languages with macros, like Rust, Scala, even C++. There won't be any function call to begin with if the corresponding logging level is not enabled, only a branch.
- tonyarkles 11y agoI replaced it with a macro!
- 11y ago
- sbov 11y agoThis is a false dichotomy. If you log things you don't need brilliant engineers spending countless hours on a logging infrastructure. All you need to do is log things to local disk, and delete some when you start to run out of space. You can write a fairly trivial script to do so.
- pjc50 11y agoThat limits your logging rate to your available disk bandwidth.
- jorangreef 11y agoThat's not a problem for 99.99% of applications.
- tyingq 11y agoYou could set up something to allow a dynamic change in logging level. Doesn't help with blip type issues that happened in the past, but helps with issues that arise and continue. There's a defacto standard for many daemons using SIGUSR1 as a prompt to close/reopen log files. You could settle on some other non-used signals to change the log level.
- oasisbob 11y agoMy primary disagreement with that thesis would be the utility of logging things that clearly represent transactions, especially to the outside world. eg, HTTP access logging is good and useful (vs non-structured debug timings on every fragment rendering) eg, A mail server reporting delivery of a particular message into a mailbox. I think your "events with identity" description is a nice succinct version of this same idea.
- wglb 11y agoI quite agree with this. In extreme circumstances, you don't know what you need until some disaster happens. Consider why there are cockpit voice recorders and flight data recorders. Almost all of that data is thrown away. In a more extreme example than the newspaper, consider some sort of data breach investigation.