8 ms·
The Python logging module is one of the worst modules in the stdlib. You can see it’s Java inspiration all over it, as being one of the oldest modules it was cr
by orf 4y ago
The Python logging module is one of the worst modules in the stdlib. You can see it’s Java inspiration all over it, as being one of the oldest modules it was created before anyone really knew what “pythonic” code was.
Things I really dislike about it is the lack of “context”. Usually log messages are about something - often a request. You can’t attach a request ID or something else a nested set of function calls that log.
The whole library is just really rubbish and has not aged well. There’s no standard way of outputting JSON logs, but there’s a built in way to spawn a socket server that allows the logging configuration to be updated remotely via a bespoke protocol? And everything is built mostly around logging to files.
It also hits that perfect sweet spot of being both overly engineered and totally inflexible. I wanted to funnel our log messages to s3 via the “s3fs” module, which supports a “file like” object that gets written to s3. I had to hack around the various handlers to support this because it assumes it’s a “real” file.
Hate it.
- formerly_proven 4y agoIt's unfortunate how blind copying of log4j into many languages seems to have inhibited improvements in logging in the last ~20ish years.
- systemvoltage 4y agoAgreed. One of the most confusing libraries with a ton of magic going on behind the scenes. I also detest PyTest, although not a std lib, pytest is one of the worst in terms of explicitness. It’s very useful but that’s orthogonal. I’ve seen newcomers to python struggle to grasp how pytest just manipulates a billion things behind the scenes. Overall, I prefer less magic and more explicitness.
- oefrha 4y agoPSL unittest is very explicit and has no magic. Ironically it also has Java written all over it. You must be the first person I met who prefer unittest to pytest.
- txru 4y agoI sometimes prefer unittest too! Pytest is very opinionated about how and when it's going to be run. If you want to put a test in say, Jupyter or a Markdown block, unittest is much more amenable to just doing what you want without having to read through documentation to break a whole bunch of magic behavior.
- dragonwriter 4y ago> PSL unittest is very explicit and has no magic. Ironically it also has Java written all over it. By “Java written all over it” do you mean that like JUnit for Java, it's an implementation of the xUnit pattern derived from SUnit for Smalltalk?
- oefrha 4y agoIt didn't have to copy everything down to the camelCase if it was just an implementation of the xUnit pattern with no Java influence.
- dragonwriter 4y agocamelCase is idiomatic in Smalltalk, which as a language and a community was a pretty big influence on other dynamic OO languages like Python and Ruby, and from whence SUnit comes, and thus the whole xUnit world, derives. So, it's maybe not the best evidence that the Python version is particularly a clone of the Java port.
- deleted 4y ago[deleted]
- simonw 4y agoDisagree on pytest: it's the first testing framework I've used in ANY language which has genuinely made me enjoy writing tests. It has a bit of an initial learning curve, but once you understand how fixtures work it's amazing how productive you can be with it.
- deleted 4y ago[deleted]
- miohtama 4y agoThe implicit magic makes writing pytest tests painless. Also with improvement of type hinting, tooling like PyCharm starts to understand pytest better and make it easier to read and manage and a lot of the magic can be dispellt by written out explicitly. Though this need some discipline from the dev team.
- mtlguitarist 4y agoI wish I could agree. Every aspect of pytest that I like (fixtures, plugins, etc.) is marred by a boneheaded move to make them autoloaded by default. Loading should be explicit/opt-in. It's dumv that if you pip install something that adds a new pytest plugin it'll automatically modify your runtime behavior. Some of the plugins are great (shoutout to pytest-socket), but it has the same issue flake8 has with the plugin ecosystem where the quality varies wildly. I might be in the minority but I also find python's mock implementation awful. Mockito in Java is way better where you explicitly state which calls you expect and if you get an unexpected call it errors out.
- codethief 4y agoStrong agree on all points! The number of times I had to debug into the pytest source code because developers (the ones using pytest) thought it would be cool to use some decorator black magic to manipulate global state behind the scenes which pytest not just makes very easy but actually encourages and which was obviously going to break sooner or later… And regarding the sibling comment: Yes, I prefer unittest as well.
- bornfreddy 4y agoWhat do you mean? Pytest rules! Yes there is some magic, but if you let it take control, it is very convenient to write and run tests with it. Just... you need to trust it and it will do its job.
- systemvoltage 4y agoI use pytest all the time. Doesn't mean it is well thought out. It could have been so much better with more visibility and control, and less magic.
- thechao 4y agoI can't agree with you more. I switched to Python's logger off of my janky thing about 6 months ago, and I'm hating every second of it. I still have no clue how to turn it on, turn it off, get it to fork output (stdout, stderr, file), nor how to override its behavior. It's literally easier to just have an IOBuffer laying around.
- oefrha 4y agoThere are valid complaints in the grandparent, but if you can’t figure out how to fork output to stdout, stderr and file in six months, that’s really on you. You should know how to register multiple handlers if you spent five minutes reading the documentation.
- thechao 4y agoI can easily fork output manually; I can't do it reliably using Python's logger.
- maleldil 4y agoEchoing grandparent's comment, it's not that hard: https://docs.python.org/3/library/logging.html#logging.Logger.addHandler https://docs.python.org/3/library/logging.html#logging.Logge... For dictConfig, the documentation has examples https://docs.python.org/3/library/logging.config.html#object-connections https://docs.python.org/3/library/logging.config.html#object... Python's logging has its faults, but you're complaining about something I could find in 5 minutes of Google.
- thechao 4y agoNone of which work reliably across multiple modules all fighting for control of the logger. Reliable is the key term, here.
- oblvious-earth 4y agoDon't write code where every module tries to make a root logger or set the logger config then. Each module should use `logger = logging.getLogger(__name__)` and the logger config can set in one place, conventionally in the `__main__` script. https://docs.python.org/3/howto/logging.html#advanced-logging-tutorial https://docs.python.org/3/howto/logging.html#advanced-loggin...
- paulfurtado 4y agoIt's definitely sad that there isn't built-in context support, but if you're looking for a solution to this, it is actually very easy to attach context to logs using threadlocal and a logging filter. When running a webserver though, an even simpler trick is to add middleware that sets the current request's info as the current thread's name, and then including threadName in your log format.
- andrewingram 4y agoContextVars are seemingly more robust and work with async. We use them quite extensively for adding context to logs.
- nerdwaller 4y agoI was going to say the same, I combine ContextVars with LoggerAdapters[1]. [1] https://docs.python.org/3/library/logging.html#logging.LoggerAdapter https://docs.python.org/3/library/logging.html#logging.Logge...
- MapleWalnut 4y agoTIL that ContextVars can work in place of thread locals too. Thanks for pointing that out! I had mistakenly assumed ContextVars only worked for async code.
- memco 4y agoThe contextual support is lacking but it can be done. See this comment from a previous discussion where I briefly described how I’ve used the record factory to help: https://news.ycombinator.com/item?id=31484527 https://news.ycombinator.com/item?id=31484527
- 1337shadow 4y agoTake a look at pylogctx maybe it'll do what you need.
- deleted 4y ago[deleted]
- drzoltar 4y agoAre there any good 3rd party alternatives that work better?
- Jamie9912 4y agologuru perhaps
- sage92 4y agoThis is the only library I that both can use out of the box and extend to my needs.
- linsomniac 4y agoA few months ago I built a project and wanted to add logging. After searching around, I ended up selecting loguru and it definitely is a nicer user experience. But I also didn't really love it, for reasons that I don't recall now. I would look to it again, simply for lack of a better choice. Sadly, my social media is subscribed to by pretty heavy hitters in the Python community, but when I asked about the best logging for python it was all crickets. Maybe that says something. :-)
- charlieyu1 4y agoOne missing feature of loguru is filter by logging levels. You can still get around it by modifying an internal variable that you can specifically not to
- kortex 4y agoWhat do you mean? You just add a custom handler. It's like a few lines to write a handler that does some action on specific levels.
- seabrookmx 4y agostructlog
- 4y ago
- akhmatova 4y agoThe Python logging module is one of the worst modules in the stdlib. One has to remember that logging was never intended to be some infinitely generalizable metaprotocol for massively streaming parallel notifications to the cloud or whatever. It was meant as a replacement for "if (DEBUG > 4): print(...)". For which I think it does quite well, thank you. The whole library is just really rubbish and has not aged well. Hate it. So what have you written that has made people's (not just your bosses' or your clients') lives easier, for nearly 20 years now (if only incrementally)? Do share. I see and grant your points, but I find this strong emotional rebuke of something that was never intended to be anything more than a simple convenience library to be well, strange.
- orf 4y agoIt clearly is supposed to be more than a convenience library. It’s the blessed, stdlib way to do logging with Python. The module isn’t called “print_with_levels”. So now everything needs to work with it, it’s internals and it’s way of doing things. And to top it off it’s near impossible to refactor or replace.
- akhmatova 4y agoIs there an alternative that you would prefer to bless?
- LtWorf 4y agoI normally just import syslog or journald. The latter allows for arbitrary fields to be defined and then searched, but it's not in the stdlib.
- 1337shadow 4y agoGreat idea but how do you work this out in a container where the python process is PID 1?
- LtWorf 4y agobind mount /dev/log in the container of course. Also logging can be set up remotely, but it's easier to do like that from the container and then configure the hos to send everything to a single log machine.
- helsinki 4y agoIt’s rock solid, at least. I’ve also had the displeasure of working in the logging package. Totally outdated. I created a JSON formatter (also support for context). Huge pain, but always works.
- 1337shadow 4y agoMaybe worth trying pylogctx and python-logging-loki or see how it's done there.
- emmelaich 4y agoThat might explain why it lacks a NOTICE level. That bugged the hell out of me with Java. Presumably these levels derive from syslog or older. I've no idea why they omitted NOTICE.
- miohtama 4y agoYou can quite easily add your own logging levels - it is not hardcoded. I think it’s even covered in the documentation, as I have done it before.
- MichaelRazum 4y agoYep. Have to agree. It works, but is really messy.
- rgovostes 4y agoI've been writing Python daily for 15 years but when I have to do any configuration beyond `basicConfig()`, I still groan. In many instances I have wanted to do something like temporarily turn on logging of HTTP requests, which turns into a much more involved effort than it ought to be. I just recently discovered that there's no documented way to dump the current configuration (https://stackoverflow.com/q/72624883 https://stackoverflow.com/q/72624883) so you can see what loggers exist and where their output is going.
- miohtama 4y agoNot to defend the legacy of good old logging, but if you have ever tried to log in Node.js applications Python logging starts to feel godsent.
- remram 4y agoIf you're talking about console.log() and console.error(), I don't think it's trying to be a logging system. The name of that first function is unfortunate but not fooling me. Those are just println() and eprintln()
- sorokod 4y agoFor whatever it's worth, java logging has mapped diagnostic context (MDC).
- doliveira 4y ago> there’s a built in way to spawn a socket server that allows the logging configuration to be updated remotely via a bespoke protocol That sounds like there's a Log4shell-like vulnerability waiting to be found. But I couldn't even find it, I found the SocketHandler but that's just a log destination. EDIT: shockingly, I did find it: https://docs.python.org/3/howto/logging-cookbook.html#configuration-server-example https://docs.python.org/3/howto/logging-cookbook.html#config...
- kortex 4y agoYou should try the loguru library. I was able to roll a rolling-upload-to-s3 adapter in under an hour. Switching to json logs is one bool flag away. Plus it's gorgeous https://github.com/Delgan/loguru https://github.com/Delgan/loguru Also iirc s3's "file-like interface" does not actually obey the file protocol, which is obnoxious.
- dikei 4y ago>> You can see it’s Java inspiration all over it, as being one of the oldest modules it was created before anyone really knew what “pythonic” code was. The irony is Java logging moved on and with newer libraries offers better supports for lazy evaluation, structured logging and multiple output formats, while Python logging seems stuck with an ancient copy of log4j 1.x
- wheelerof4te 4y agoDo you know about any functional-style logging libraries? Or just plain old procedural ones?