5 ms·

You use a tool like cat, vim , emacs, or tail to see logs. You can't physically read the file off the disk, how is this different than using journalctl ? # jo

You use a tool like cat, vim , emacs, or tail to see logs. You can't physically read the file off the disk, how is this different than using journalctl ? # journalctl -u ssh

rnhmjoj 5y ago
I think the issue is not that it's binary, but that it's a custom half-assed format. It's actually pretty slow because there is not enough indexing, this is why `systemctl status` can takes seconds to show the last 10 log lines[1]. Even worse, systemd can detect corruption but can't repair the logs[2], in practice a power failure can mean months of logs thrown away. Thankfully it can skip over corruption, so logging continues to work. I would have no problem if systemd used a database like sqlite, lmdb or anything else with some valid tooling, indexing, caching and proper recovery solutions. [1]: https://github.com/systemd/systemd/issues/2460 https://github.com/systemd/systemd/issues/2460 [2]: https://unix.stackexchange.com/questions/86206 https://unix.stackexchange.com/questions/86206
- zxzax 5y ago
  I looked into this a while ago -- a traditional transactional CRUD database is totally unnecessary here, these are not files that you want to be editable by random tools. The journald log files have the important property that they are append-only by a single-writer. I agree there are some outstanding issues but those issues should just be fixed, instead of throwing it all out and moving to an even more complicated database layer that brings with it all its own additional problems. I think it would be perfectly feasible for someone to improve the indexing to show the latest 10 lines, or to write a fsck-type tool to repair damaged logs.
  - throw0101a 5y ago
    > instead of throwing it all out and moving to an even more complicated database layer that brings with it all its own additional problems. Would SQLite be any more complicated than what they have now? Or OpenLDAP's Lightning Memory-Mapped Database (LMDB)? OpenLDAP/Symas' Howard Chu on the latter: * https://www.linkedin.com/pulse/20140924071300-170035-why-you-cannot-trust-lennart-poettering-systemd https://www.linkedin.com/pulse/20140924071300-170035-why-you...
lmm 5y ago
Conventional logs are files, the fundamental unifying abstraction of Unix, and so you can read them with any tool, even a tool that was written before your logging system and that your logging system's author didn't know about. Having to take them just as piped input is a much more limited interface that doesn't allow you to do all the things you can do with a real file.
- kaba0 5y ago
  But allows you to do things like properly filter based on multiple properties, and you can still use it to output text logs as well.
  - lmm 5y ago
    There are plenty of unix tools for filtering files based on all manner of things. That's the unix way of doing things, and for many users the ability to mix and match unrelated tools is where a lot of the value of linux-like systems comes from.
    - kaba0 5y ago
      I don’t find grepping text logs for a service file name with many false positives better in any way than just filtering on an actual column of a quasi-db. Also, you can pipe the output of journalctl to do whatever you want with it with those unix tools.
- Chris2048 5y ago
  > so you can read them with any tool what tool can't you read them with? Why is the fs the fundamental unifying abstraction, as opposed to pipes? > doesn't allow you to do all the things you can do with a real file like what?
  - regularfry 5y ago
    Pipes are files. From the point of view of the consuming process, you're just reading from a file descriptor. Files are more fundamental than pipes.

You use a tool like cat, vim , emacs, or tail to see logs. You can't physically read the file off the disk, how is this different than using journalctl ? # journalctl -u ssh

rnhmjoj 5y ago

I think the issue is not that it's binary, but that it's a custom half-assed format. It's actually pretty slow because there is not enough indexing, this is why `systemctl status` can takes seconds to show the last 10 log lines[1]. Even worse, systemd can detect corruption but can't repair the logs[2], in practice a power failure can mean months of logs thrown away. Thankfully it can skip over corruption, so logging continues to work. I would have no problem if systemd used a database like sqlite, lmdb or anything else with some valid tooling, indexing, caching and proper recovery solutions. [1]: https://github.com/systemd/systemd/issues/2460 https://github.com/systemd/systemd/issues/2460 [2]: https://unix.stackexchange.com/questions/86206 https://unix.stackexchange.com/questions/86206

zxzax 5y ago
I looked into this a while ago -- a traditional transactional CRUD database is totally unnecessary here, these are not files that you want to be editable by random tools. The journald log files have the important property that they are append-only by a single-writer. I agree there are some outstanding issues but those issues should just be fixed, instead of throwing it all out and moving to an even more complicated database layer that brings with it all its own additional problems. I think it would be perfectly feasible for someone to improve the indexing to show the latest 10 lines, or to write a fsck-type tool to repair damaged logs.
- throw0101a 5y ago
  > instead of throwing it all out and moving to an even more complicated database layer that brings with it all its own additional problems. Would SQLite be any more complicated than what they have now? Or OpenLDAP's Lightning Memory-Mapped Database (LMDB)? OpenLDAP/Symas' Howard Chu on the latter: * https://www.linkedin.com/pulse/20140924071300-170035-why-you-cannot-trust-lennart-poettering-systemd https://www.linkedin.com/pulse/20140924071300-170035-why-you...

lmm 5y ago

Conventional logs are files, the fundamental unifying abstraction of Unix, and so you can read them with any tool, even a tool that was written before your logging system and that your logging system's author didn't know about. Having to take them just as piped input is a much more limited interface that doesn't allow you to do all the things you can do with a real file.

kaba0 5y ago
But allows you to do things like properly filter based on multiple properties, and you can still use it to output text logs as well.
- lmm 5y ago
  There are plenty of unix tools for filtering files based on all manner of things. That's the unix way of doing things, and for many users the ability to mix and match unrelated tools is where a lot of the value of linux-like systems comes from.
  - kaba0 5y ago
    I don’t find grepping text logs for a service file name with many false positives better in any way than just filtering on an actual column of a quasi-db. Also, you can pipe the output of journalctl to do whatever you want with it with those unix tools.
Chris2048 5y ago
> so you can read them with any tool what tool can't you read them with? Why is the fs the fundamental unifying abstraction, as opposed to pipes? > doesn't allow you to do all the things you can do with a real file like what?
- regularfry 5y ago
  Pipes are files. From the point of view of the consuming process, you're just reading from a file descriptor. Files are more fundamental than pipes.