4 ms·

And that's why journald is such a cool thing. * Want to parse stuff? journalctl -o json * Lots of stuff going on, need more precise timestamps? -o short-preci

And that's why journald is such a cool thing. * Want to parse stuff? journalctl -o json * Lots of stuff going on, need more precise timestamps? -o short-precise * Want metadata, like the pid? It's in there. * Want to know where to continue parsing? It supports cursors. * Want to save disk space? It uncompresses logs transparently and can trim logs to whatever size you want.

InvertedRhodium 5y ago
Want to filter logs at collection time? Well, too bad - your use case is impure and you should just fix everything immediately, even that over which you have no direct control. https://github.com/systemd/systemd/issues/6432 https://github.com/systemd/systemd/issues/6432
- Poiesis 5y ago
  I realize that the linked thread explains that decreasing the log level wouldn't work because everything else would be affected—but couldn't someone combine decreasing the log level with using a different journal namespace?
- dale_glass 5y ago
  I actually mostly agree with Poettering here that it's kind of a hack and that the ideal is to collect everything and filter afterwards. You can't get back stuff you exclude by mistake. Programs should be logging because there's some value in the information being sent to the log. If it annoys everyone and serves no purpose, the program needs fixing.
  - eximius 5y ago
    The programs needing fixing and supporting filtering are not exclusive options.
  - InvertedRhodium 5y ago
    I agree that they _should_ be fixed - but life isn't perfect, and we don't always have a way to fix the issue ourselves. > You can't get back stuff you exclude by mistake. Ok. That's fine - it's my system, my data, and my ass on the line if we lose data by mistake. I don't know see why a developer that knows nothing about my problem domain and the constraints under which I'm operating gets to dictate to me that my choice is wrong.
    - Spivak 5y ago
      Because solving the problem you’re describing means having to do work to implement and maintain a new feature. This kind of framework would put journald on the complexity level of rsyslog when currently journald is purposely very dumb. If you went ahead and implemented thoughtfully and sent a patch there’s a good chance it would get merged. Everyone has a idea about what “I just need basic filtering” and so accommodating anyone but not everyone is a recipe for making even more people mad. Telling people to just disable journald on-disk storage and forward to rsyslog and friends if you need complex filtering and don’t want to pay for double storage is a solution that feels bad but is actually generic enough to be useful.
      - InvertedRhodium 5y ago
        > If you went ahead and implemented thoughtfully and sent a patch there’s a good chance it would get merged. Poettering has made it abundantly clear that he will not accept patches that implement arbitrary filtering. There has been very little engagement on his part with people who have legitimate issues that this would resolve, and appears unwilling to accept that there are use-cases where arbitrarily filtering log data at ingestion time has benefits. Why anyone, myself included, would put the time and effort in to developing a patch that it's clear is ideologically opposed by the gatekeepers of that project is beyond me.
stonogo 5y ago
* Want to recover from a corrupted write? Too bad. You get a brand new log! * Want log input from processes that exited? Nope, information is lost (bug 2913) * Want to validate the certificate of a remote log recipient? Nope! (bug 4092) * Want to submit changes to the specification to help fix any of the above problems? What specification? The code is the spec! I am a huge journald fan, but some of these bugs have been around for years and years. It's frustrating as hell, so I don't blame the syslog diehards.
- dale_glass 5y ago
  What's wrong with the first one? I see the approach as a very reasonable one -- don't touch a thing that's already broken. Leave it alone, and start a new good log. If you've ever used reiserfs you'll know fsck isn't guaranteed to make things better.
usr1106 5y ago
The article talks also a bit about networking. A colleague of mine run on evaluation of remote log collection using journald's remote support and syslog. He found many problems with jornald remote logging. We have mobile connections and they can be poor occasionally. Syslog scored better. Whether the evaluation was fair I have no it's idea. It would have required me the same time to do my own one. But I would have preferred to see systemd win. At least in the local case the benefits mentioned by the parent comment have convinced me years ago.
Apaec 5y ago
Also, you can send the logs to a centralized server with systemd-journal-remote[1]. [1]: https://www.digitalocean.com/community/tutorials/how-to-centralize-logs-with-journald-on-debian-10 https://www.digitalocean.com/community/tutorials/how-to-cent...

InvertedRhodium 5y ago

Want to filter logs at collection time? Well, too bad - your use case is impure and you should just fix everything immediately, even that over which you have no direct control. https://github.com/systemd/systemd/issues/6432 https://github.com/systemd/systemd/issues/6432

Poiesis 5y ago
I realize that the linked thread explains that decreasing the log level wouldn't work because everything else would be affected—but couldn't someone combine decreasing the log level with using a different journal namespace?
dale_glass 5y ago
I actually mostly agree with Poettering here that it's kind of a hack and that the ideal is to collect everything and filter afterwards. You can't get back stuff you exclude by mistake. Programs should be logging because there's some value in the information being sent to the log. If it annoys everyone and serves no purpose, the program needs fixing.
- eximius 5y ago
  The programs needing fixing and supporting filtering are not exclusive options.
- InvertedRhodium 5y ago
  I agree that they _should_ be fixed - but life isn't perfect, and we don't always have a way to fix the issue ourselves. > You can't get back stuff you exclude by mistake. Ok. That's fine - it's my system, my data, and my ass on the line if we lose data by mistake. I don't know see why a developer that knows nothing about my problem domain and the constraints under which I'm operating gets to dictate to me that my choice is wrong.
  - Spivak 5y ago
    Because solving the problem you’re describing means having to do work to implement and maintain a new feature. This kind of framework would put journald on the complexity level of rsyslog when currently journald is purposely very dumb. If you went ahead and implemented thoughtfully and sent a patch there’s a good chance it would get merged. Everyone has a idea about what “I just need basic filtering” and so accommodating anyone but not everyone is a recipe for making even more people mad. Telling people to just disable journald on-disk storage and forward to rsyslog and friends if you need complex filtering and don’t want to pay for double storage is a solution that feels bad but is actually generic enough to be useful.
    - InvertedRhodium 5y ago
      > If you went ahead and implemented thoughtfully and sent a patch there’s a good chance it would get merged. Poettering has made it abundantly clear that he will not accept patches that implement arbitrary filtering. There has been very little engagement on his part with people who have legitimate issues that this would resolve, and appears unwilling to accept that there are use-cases where arbitrarily filtering log data at ingestion time has benefits. Why anyone, myself included, would put the time and effort in to developing a patch that it's clear is ideologically opposed by the gatekeepers of that project is beyond me.

stonogo 5y ago

* Want to recover from a corrupted write? Too bad. You get a brand new log! * Want log input from processes that exited? Nope, information is lost (bug 2913) * Want to validate the certificate of a remote log recipient? Nope! (bug 4092) * Want to submit changes to the specification to help fix any of the above problems? What specification? The code is the spec! I am a huge journald fan, but some of these bugs have been around for years and years. It's frustrating as hell, so I don't blame the syslog diehards.

dale_glass 5y ago
What's wrong with the first one? I see the approach as a very reasonable one -- don't touch a thing that's already broken. Leave it alone, and start a new good log. If you've ever used reiserfs you'll know fsck isn't guaranteed to make things better.

usr1106 5y ago

The article talks also a bit about networking. A colleague of mine run on evaluation of remote log collection using journald's remote support and syslog. He found many problems with jornald remote logging. We have mobile connections and they can be poor occasionally. Syslog scored better. Whether the evaluation was fair I have no it's idea. It would have required me the same time to do my own one. But I would have preferred to see systemd win. At least in the local case the benefits mentioned by the parent comment have convinced me years ago.

Apaec 5y ago

Also, you can send the logs to a centralized server with systemd-journal-remote[1]. [1]: https://www.digitalocean.com/community/tutorials/how-to-centralize-logs-with-journald-on-debian-10 https://www.digitalocean.com/community/tutorials/how-to-cent...