3 ms·
Binary logging would be fine if it Just Worked, but my experience is that journald/journalctl does not just work, and when it breaks I don't have logs. For exa
by gwillen 5y ago
Binary logging would be fine if it Just Worked, but my experience is that journald/journalctl does not just work, and when it breaks I don't have logs.
For example, if the system clock is not monotonic, my text logs are still written sequentially in order, and it's easy to figure out what happens. It is the stated view of the journald maintainers that, if your clock is not monotonic, you get to keep both pieces. https://github.com/systemd/systemd/issues/662 https://github.com/systemd/systemd/issues/662 (The specific issue described there sounds complicated, but the fundamental problem behind it, which I have hit multiple times, is "if your system can't maintain a monotonic clock at all times, including early boot, then your logs will be mangled.")
Similarly, a text log line is complete the moment it's written, even if the process that wrote it goes away immediately afterwards. Journald, by contrast, performs asynchronous metadata lookups for _each_ log line at some later time, which it apparently does not cache. This means that, when a process dies suddenly, the final messages will not appear in "journalctl -u", because they failed to get tagged with metadata. (I can't find the bug for this, so I haven't verified that it's still open and unfixed at this time. But it has certainly existed for years at this point.)
So no, in principle I have no objection to _competent_ binary logs. Journald does not meet that bar.
- saurik 5y ago> (I can't find the bug for this, so I haven't verified that it's still open and unfixed at this time. But it has certainly existed for years at this point.) https://github.com/systemd/systemd/issues/2913 https://github.com/systemd/systemd/issues/2913 I am so happy you mentioned this, as I am in a position of developing a daemon used by people who insist on using systemd, and I keep asking them for logs when things fail and, in fact, the most critical logs -- arguably the only ones that ever truly matter: the ones that come immediately before the daemon terminates for some reason -- are often missing. Now I know why :/. https://unix.stackexchange.com/questions/291483/how-to-fix-unit-attribution-of-last-lines-of-systemd-logging-before-exit https://unix.stackexchange.com/questions/291483/how-to-fix-u... > So no, in principle I have no objection to _competent_ binary logs. Journald does not meet that bar. Yeah... I hadn't even gotten to the end of your message before immediately going "omg I have to look into this" and the most apt description that was coming to my mind for a logging system that fails to actually store logs -- and particularly the most critical death throws to the log before termination -- is 100% "incompetent" :/.