3 ms·

Love/hate systemd as I might, it's been rock solid everywhere I've used it, and I've used it heavily. It has it's quirks, as does the init-scripts that came bef

by antonyh 9mo ago

Love/hate systemd as I might, it's been rock solid everywhere I've used it, and I've used it heavily. It has it's quirks, as does the init-scripts that came before, and launchd on OSX (not sure what the modern equivalent is for MacOS). However, the systemd journal raw format is binary data and would much rather a plain text log. All things being equal I'd rather deal with human readable files.

embedding-shape 9mo ago
> However, the systemd journal raw format is binary data and would much rather a plain text log Yeah, I also wish that at least was an option, would make some things easier. Also wished the remote log sending was easier, not sure if it's just me but was a huge hassle to setup properly, and really hard to properly validate it works as expected in all cases. Finally got it working, but it isn't as easy as the other parts of systemd/journald.
- antonyh 9mo ago
  It should have been an option, even dumb old CSV if it needed structured data. I didn't bother trying to natively ship and used promtail instead, to get it into Loki, so I could query via Grafana.
e2le 9mo ago
Personally I would much rather they had simply used an existing database file format. For example, sqlite3 which is robust and already present in the default installation of most Linux distributions. Querying system logs with SQL would be cool and make things a bit easier unlike the sd_journal API with it's strange/bizarre quirks.
- antonyh 9mo ago
  Wow what an idea, that would work so well and give me a trivial way to build apps to do alerts, monitoring, shipping to remotes, and whatnot in virtually any language.
- direwolf20 9mo ago
  You could write this. But you see the problem with systemd — because it integrates the journal with the init manager, it's hard to replace just one or the other.
PunchyHamster 9mo ago
> However, the systemd journal raw format is binary data and would much rather a plain text log. All things being equal I'd rather deal with human readable files. It happens to be worse than text logs, worse than just spewing JSON to files, and entirely worse than most binary log formats. Every log line takes several times it would under text, duplicates a ton of info (like writing boot id in every entry) and somehow is not crash-proof (at least every non-clean shutdown gets me journald complaining about corrupt log files). It's also dog slow in commands that matter (well documented in systemd bugs on github). It should be just sqlite with some strategic indexes and tables. It's so bad.
- e2le 9mo ago
  One feature of the Journal format which I don't believe to be a great design choice is that entries can contain non-unique fields. Unfortunately this doesn't seem to be something that is handled well by all tools, instead only returning the value for the first field of an entry with the provided name.
Grimm665 9mo ago
I'm probably in the minority for preferring journald's binary logging, especially alleviating the need for things like log-rotate, which I have always fought issues with. I like how RedHat distros have it setup, where journald collects the logs, but rsyslog is there parsing them into the traditional /var/log/messages and /var/log/secure, so you get some logs in plain text as well as being able to send them along to an rsyslog server the traditional way. I haven't run into a situation with corrupt binary logs, and any crashed system I've booted with a rescue disk I can connect to the binary logs from the rescue distro's journalctl. That being said, I imagine one bad experience with a corrupt log or a non-booting system I can't get logs from would change my mind pretty quickly, but that hasn't been the case for almost a decade, so *shrug*

by antonyh 9mo ago

embedding-shape 9mo ago

> However, the systemd journal raw format is binary data and would much rather a plain text log Yeah, I also wish that at least was an option, would make some things easier. Also wished the remote log sending was easier, not sure if it's just me but was a huge hassle to setup properly, and really hard to properly validate it works as expected in all cases. Finally got it working, but it isn't as easy as the other parts of systemd/journald.

antonyh 9mo ago
It should have been an option, even dumb old CSV if it needed structured data. I didn't bother trying to natively ship and used promtail instead, to get it into Loki, so I could query via Grafana.

e2le 9mo ago

Personally I would much rather they had simply used an existing database file format. For example, sqlite3 which is robust and already present in the default installation of most Linux distributions. Querying system logs with SQL would be cool and make things a bit easier unlike the sd_journal API with it's strange/bizarre quirks.

antonyh 9mo ago
Wow what an idea, that would work so well and give me a trivial way to build apps to do alerts, monitoring, shipping to remotes, and whatnot in virtually any language.
direwolf20 9mo ago
You could write this. But you see the problem with systemd — because it integrates the journal with the init manager, it's hard to replace just one or the other.

PunchyHamster 9mo ago

> However, the systemd journal raw format is binary data and would much rather a plain text log. All things being equal I'd rather deal with human readable files. It happens to be worse than text logs, worse than just spewing JSON to files, and entirely worse than most binary log formats. Every log line takes several times it would under text, duplicates a ton of info (like writing boot id in every entry) and somehow is not crash-proof (at least every non-clean shutdown gets me journald complaining about corrupt log files). It's also dog slow in commands that matter (well documented in systemd bugs on github). It should be just sqlite with some strategic indexes and tables. It's so bad.

e2le 9mo ago
One feature of the Journal format which I don't believe to be a great design choice is that entries can contain non-unique fields. Unfortunately this doesn't seem to be something that is handled well by all tools, instead only returning the value for the first field of an entry with the provided name.

Grimm665 9mo ago

I'm probably in the minority for preferring journald's binary logging, especially alleviating the need for things like log-rotate, which I have always fought issues with. I like how RedHat distros have it setup, where journald collects the logs, but rsyslog is there parsing them into the traditional /var/log/messages and /var/log/secure, so you get some logs in plain text as well as being able to send them along to an rsyslog server the traditional way. I haven't run into a situation with corrupt binary logs, and any crashed system I've booted with a rescue disk I can connect to the binary logs from the rescue distro's journalctl. That being said, I imagine one bad experience with a corrupt log or a non-booting system I can't get logs from would change my mind pretty quickly, but that hasn't been the case for almost a decade, so *shrug*