3 ms·
I have no idea what the authors qualifications are. There have been uncountable cases where a GOOD log trail has been the difference between a 20 hour journey
by kfsdfjdf 5y ago
I have no idea what the authors qualifications are.
There have been uncountable cases where a GOOD log trail has been the difference between a 20 hour journey through a code base and a short stop to immediately discovering the problem. Common failure points should be logged, especially when they are deeply nested, or they involve complex hard-to-diagnose processes.
Overlogging is just as bad. But to say "do not log" and then suggest using services like sentry or datadog instead is really reaching around your nose to touch your face. You're doing yourself a disservice by lacking the final bit of correlation - the approximate location and nature of the failure. There have been times when I've been on a service so complex that just building a harness to reproduce the case takes hours or days. Not everything is a nice, clean, small and simple microservice where you only have N guesses at where the problem could originate. On a large enough, or old enough, service the upper bound could approach N! very quickly.
I'm not sure why this was posted to HN as it seems like a very low-quality ranty post by someone whom I'm not sure has ever worked on a large enough project where it becomes non-obvious what the origin of some error alerts are.
- smikhanov 5y ago> what authors qualifications are The author is definitely not very qualified, yes.
- masklinn 5y ago> Overlogging is just as bad. It really is not: you can trim overlogging, you can't add missing logs. Now one thing I am very much in favor of, but which I didn't see anywhere in TFA, is the ability to dynamically instrument instances (add or remove log points). It's not necessarily easy to know what to log or not to log. With "normal" logging adding new log points is a complicated process (they're static so you have to introduce them, ensure they're correct, and then go through whatever deployment process you have), so there's a natural tendency to overlog because if you underlog you may miss critical information. In systems where the capability exists, instrumenting log points "from the outside" without having to update the program being observed is much better, not only does it let you refine your logging more easily, it also lets you brutally increase logging density (possibly locally) without downtime.
- kfsdfjdf 5y agoThat's true. Annoying and pointless logs at least tell you where they are :). I suppose what I meant by this is in the heat of the moment having to sift through overlogging probably somewhat justifies the "no log" rant on the surface. But the underlying problem isn't logging but rather the abuse of it so it seems a little premature to blame logging itself for this.
- wizzwizz4 5y agoFor trimming logs, I load the whole log file into Python's REPL and repeatedly add >>> l = [x for x in l if ...] lines. I imagine there's a better way.
- icedchai 5y agoMaybe try "grep -v" ?
- spaetzleesser 5y agoI try to produce many small log files by hour or day. This makes it pretty easy to discard old logs or reduce the amount of data you have to look at if you know the problem was within a certain timeframe.
- pm90 5y agoAre there ways to do this? It sounds like an application restart would be warranted at the very least, which can be disruptive.
- masklinn 5y ago> Are there ways to do this? In some systems / runtimes, yes. For instance every browser supports "logging breakpoints" which can be added and removed on the fly (create a breakpoint then right click, you should get an option or sub-option about logging). I would also expect that there's Java instrumentation things which do it, though I haven't done any java for a very long time. It's pretty rare though, obviously so for runtime-less programs, somewhat less so for those with a runtime but while "external" profiling is common logging is much less so (e.g. Python supports updating the configuration of the existing static logging system on the fly, but not defining the log points). > It sounds like an application restart would be warranted at the very least, which can be disruptive. See above, if you can profile an application on the fly without restarting it you should be able to instrument for logging. And tools like dtrace/ebpf should allow this, but I've yet to take the plunge and actually try and look at that, it's only been an idle thought so far.
- deleted 5y ago[deleted]
- newaccount2021 5y ago
- spaetzleesser 5y ago“the difference between a 20 hour journey through a code base and a short stop to immediately discovering the problem.” I had situations where the difference was between having an intermittent problem for months or years that nobody can figure out and pouring over logs for days and finding the problem. I do pretty extensive logging in my code and it has saved my ass many times. Before people complain about performance or storage problems I want to see hard profiling numbers. In most cases you it’s only a few places where the logging is causing problems and you can reduce it there. Overlogging is almost never a real problem in my experience. You can always discard log data you have but you can’t create log data you may need but didn’t produce.
- outworlder 5y ago> Overlogging is almost never a real problem in my experience I would disagree, after managing a 100 node ELK monstrosity to try to process the terabytes of logs generated by our systems. Overlogging is not a problem if log levels can be easily changed (preferably without an app restart). One app logging like crazy is not a problem. A thousand of microservices, is.