5 ms·
I don't get why people get so theological about logging. If logs bring you some sort of value, then log away. If they don't, then don't. In some domains you ab
by marginalia_nu 5y ago
I don't get why people get so theological about logging.
If logs bring you some sort of value, then log away. If they don't, then don't. In some domains you absolutely crucially need logs, in others they're largely a waste of time.
Logging is probably the most foolproof way to extract debugging information from your application. The killer feature of logging is that standard out very rarely breaks, no matter how bad things get it works. It will work from the moment the application starts, it will work in an OOM situation, will work if your network switch is on fire, when the DNS records are wrong and your technicians can't even enter the building, it will work no matter how bad things are. If logging does somehow fail, then that is on you, in that scenario you have fucked up your architecture somehow.
- deleted 5y ago[deleted]
- pm90 5y agoMost applications today run as replica sets (k8s pods/VM asgs) so log aggregators are another interesting part of the problem. Most shops are moving to saas logging for log collection which presents another failure mode: your logs are only as reliable as the saas provider. It is true however that looking at the individual replica, application logs should be very reliable.
- rbanffy 5y ago> log aggregators are another interesting part of the problem. And then you hope you are not logging to a logging service that'll try to execute something that was logged.
- waffletower 5y agoDefinite up vote here. Theological is the perfect word for this. It would be helpful if the author substituted the word "blog" for the word "log" in the article, and took the new meanings that result seriously.
- geocar 5y ago> I don't get why people get so theological about logging. Because the reason to log is about people, not about technology. There's no technical reason to make logs: Just don't make bugs, and you won't need the logs to find out what went wrong because nothing will have gone wrong. Go to the beach! This is hard though, and bugs happen for all sorts of reason that have nothing to do with the programmer making a mistake-- the customer can be unsatisfied because they didn't know what they wanted. Programmers learn very early in their career to distrust the requirements, because they know that saying but that's what the ticket says gets them nowhere besides an argument: They have to do what the business/customer meant. Put another way, the user doesn't want logs: The user wants to know if everything is okay or not, and presumably they have some way to figure that out, but if it isn't working correctly (or they merely think it isn't), the programmer will probably want those logs to help them. In many environments, the loop is larger than just those two people, there's QA support people, and internal experts/operators, and triage, and all of those people are going to have different needs from those logfiles. People might also have a second use for those logs involving anticipation of failures and tracking capacity. So there are a lot of non-technical, non-functional needs for logs, and once you understand this, it may be a little easier to understand why people would have almost rabid opinions about what those logs should look like (or if they can get the cure they need another way, what they might prefer instead of logs) > [standard out] will work in an OOM situation If you use write() which doesn't allocate memory, that's almost true, but if you use printf() it might allocate memory, and it might not work in an OOM anyway depending on what "standard out" actually is. > will work if your network switch is on fire, when the DNS records are wrong and your technicians can't even enter the building And yes you can write logs that nobody will ever be able to read, but what's the point of that? When your manufacturing plant is a smoldering rubble, someone is going to want to launch an expedition for those hard drives and try and figure out what caused millions (or potentially billions) of dollars in damages, so if you just wrote everything to standard output and washed your hands of the rest, you might have to find another job anyway. I've been programming for over three decades at this point, and I'm convinced there is no "easy one-size-fits-all answer" to whether to log or not, or if to log, how to log, so I find it useful to learn about this bit of theology so that I'm better equipped to talk to others about this, and maybe convince them of my opinions.
- 5y ago
- andrewstuart2 5y agoMy personal reason for having strong opinions on logging is that nearly every performance issue I have come across has had an easy win of 2x-3x improvement (small sample size, to be fair) by simply disabling logs. Not that this tends to stick because people fear the lack of information that comes from not logging, but you need to be aware of exactly how expensive logging to a synchronized output is, and how expensive that serialization is compared to the rest of the critical path. People treat logging as if it's free, when the reality is that it very often has a significant impact on performance, exacerbated by the fact that it's treated as free. Add in the fact that today's distributed systems approaches add complexity to the logging paradigm and make it much harder (vs something like Jaeger/OTel) to understand the context of a single log message, and I would rather logging disappear almost entirely in favor of metrics (fast, coarse) and tracing (slower, fully contextualized), with some ability to fall back to just printing metrics and traces to stdout or other local debugging options as a fallback.
- clappski 5y ago> logging to a synchronized output There's your performance issue, logging doesn't need to be persisted or even processed by your main program thread or any thread/process actually doing the work. Who's logging straight to stdout or directly ::write'ing a file in a production application?
- dustymcp 5y agoUh oh it got silent in here :)
- kcb 5y agoMany many people
- andrewstuart2 5y agoRight, and depending on the actual buffer implementation when logs are buffered, given the massive volume of logging I've seen, it's likely not all that much better than synchronous because those buffers are going to fill quickly and need flushed frequently.
- outworlder 5y ago> I don't get why people get so theological about logging. > If logs bring you some sort of value, then log away. I have been on the other side of the problem for years now. Logging has a cost. And the cost benefit is also not great. It requires a human to read it(either that or a regepx hellscape awaits you). This can be fine if you are working with a small service, but it quickly gets out of hand once scale increases. I've had to deal with double digits(approaching triple) terabytes of logs daily. Most of the information logged was, like the author suggests, completely irrelevant. It makes it much harder to see what's actually relevant. And, for whatever was actually relevant, generally not enough context was provided (something the author also argues). What you really have to have in production systems are metrics. That's what's going to tell you if shit is on fire and which ass is burning. Has your app has hit a catastrophic failure and cannot continue? OOM? Sure, by all means log away. Did it have an error while trying to talk to a service? Are you retrying? Then _do not log_ until your retries (or whatever recovery mechanism) fail. Networks have become very reliable, but failures are expected and normal. Spit out metrics instead. If you don't do this, some schmuck will have to spend their days trying to keep the logging infrastructure from falling over (Cthulhu helps you if you are on an ELK stack) while management will be complaining why noone seems to notice failures before customers do. If you find yourself parsing logs to alert on issues, please revisit your architecture. METRICS FIRST. Logs are a safety net and a tool for troubleshooters.
- marginalia_nu 5y agoWhat you are describing is pretty clearly within a domain where logging doesn't bring much value. It's either superfluous debug messages from an earlier phase in development, or at the wrong granularity. If you are dealing with thousands of events per second, you can't log at every event. Your logging probably shouldn't produce more than a couple of lines/second on average unless something has gone very wrong. Logging is something that needs to be maintained (like all code), it can't be just some vestigial by-product of the development that's left to its own devices, at least not if it's to be of any use. The same is true for metrics. If your metrics are just some arbitrary selection of times and counts that seemed good at the time, they're probably not very useful. Which data to collect is something that needs to be deliberately chosen, and re-evaluated over the lifetime of the application. When you are launching a new product, then lots of logging is probably a good idea. When that product is mature, you probably should have dialed it back several notches, as you most likely already know the general areas where things go wrong.
- falcolas 5y ago> I don't get why people get so theological about logging. IMO, it’s because the right logging solution for one situation is not the right solution for another situation. That means that one person is “right” and the other person is “wrong” (because the context required to identify why one person is right is complicated and hard to express, and is thus missing). Observability (of which logging is a subset) has a performance impact, and software has performance requirements. This means that the right amount of observability is going to depend on the software, not a standard.
- hzhou321 5y ago> If logs bring you some sort of value, then log away. If they don't, then don't. The issue is -- some people find value in the logs but some other people don't. I am in the strong camp that find logging provide negative value, mostly due to the extra complexity it brings and its obstruction on code reading. And I believe the reason the other camp finds value in logging is because they lack insight to pin-point issues and lack effort to gain that insight. But I can't really use that argument because it can easily be deemed offensive and solicit slash backs rather than constructive discussions and debates. Logging is my most frustrating pet peeves.