3 ms·
Many of those things are not all or nothing. For example you can use chrony with systemd. RHEL 8 does that by default. You can even install rsyslog to consume l
by freedomben 7y ago
Many of those things are not all or nothing. For example you can use chrony with systemd. RHEL 8 does that by default. You can even install rsyslog to consume logs if you want, and it works just fine.
I agree systemd tries to do a lot (maybe too much) and often isn't better than alternatives, but I don't think that's a big deal since you can still use the alternatives quite easily.
- newnewpdro 7y agoYou can't replace journald with rsyslog. You can use rsyslog in addition to journald, but there's no way to get journald out of the picture - it's intertwined with the service manager.
- freedomben 7y agoYes, sorry if what I wrote was misleading. You can't replace journald with rsyslog, but the only complaints I hear about journald are that it's binary so you have to use journalctl to read them. By installing rsyslog you basically get the classic log behavior back. So technically under the hood you're still using journald, but you never have to know it.
- pferde 7y agoMy main complaint is that journald is unreliable and is losing log entries (or lines, if you will). I had an automation recipe that would start a bunch of cloud instances, configure them and start certain job via a systemd service unit, and collect logs from this job 24 hours later. Initially, we were pulling these logs via journald as logs for that service unit. But then we noticed that random lines (1-5 lines out of maybe 40 for a single job) would be missing in about half the instances. We ended up pulling those logs from files in /var/log instead, made courtesy of rsyslog. Not a single missed line. Again journald is unreliable and as a whole should be nuked from the orbit.
- freedomben 7y agoInteresting, this is the first time I've ever heard this. How long ago did this happen?
- pferde 7y agoSometimes between August and September, and it was on Suse Linux Enterprise Server 12. The lines would go missing mostly if the logs were collected two or three days later instead of immediately after the job finished (it was a job running for 24 hours), which leads me to think it had to do with journald expiring entries to save on disk space. A consistent log manager would expire them chronologically, but here the lines were missing from the middle or the end of the output, so not the oldest ones logged.
- newnewpdro 7y agoFiltering journal contents by metadata is known to be unreliable. It's not that the logs are missing, it's that they're not getting included by your query. The way journald associates log lines with process metadata is inherently racy. This situation has improved some with the metadata caching changes, but IIRC there are still some holes where processes can log without journald associating accurate/up-to-date metadata with the logged info.
- pferde 7y agoThat makes sense, I guess, in a rather warped and bizarre way. The point still stands - journald is unreliable, and after getting burnt like this it will be years before I start trusting it again.
- newnewpdro 7y agoFYI, in case you wanted to document your pain, it might help prioritize fixing it fully: https://github.com/systemd/systemd/issues/2913 https://github.com/systemd/systemd/issues/2913
- dannyw 7y ago
- newnewpdro 7y agoThere are a whole lot more complaints than that for journald. The most common one (and one I share) is how slow it can be. `systemctl status foo.service` takes a significant amount of time on any of my machines thanks to the journal, and that's just the reader side. It used to be quite crashy with lots of CPU overhead, but that has improved over time, the metadata caching helped a bunch with the overhead. There was also a pretty embarrassing CVE not long ago: https://nvd.nist.gov/vuln/detail/CVE-2018-16865 https://nvd.nist.gov/vuln/detail/CVE-2018-16865
- deleted 7y ago[deleted]
- ibotty 7y agoYou do still _use_ rsyslogd if you enable it. journald is fundamental to some core systemd functionality. Of course you can't get rid of it.
- newnewpdro 7y agoRight, which just turns rsyslogd into another consumer of the journal data analogous to `journalctl --follow --all`. You still have all the overhead (and surface area) of journald being in the loop of everything logged. Many people when they encounter problems with journald want to continue using systemd for their service management, but revert to the old world of low-overhead metadata-free unstructured syslog O_APPEND text files for everything logged. If systemd (the project) were truly a collection of modular components you could pick and choose, you wouldn't need journald and you'd be able to run syslog like it's the 90s. It's not an impossibility, the systemd developers just seem to have zero interest in implementing stuff that enables escaping their overall architecture. I imagine all that's needed is some minimal IPC glue to forward all the stdout/stderr output from services to syslog when journald isn't in use. Systemd's role is to wire that stuff up to journald currently, and it's unnecessarily opinionated about journald being the only supported target in that role.
- bonzini 7y ago> If systemd (the project) were truly a collection of modular components you could pick and choose, you wouldn't need journald and you'd be able to run syslog like it's the 90s. Journald and udevd are exactly the only piece of systemd that systemd's pid 1 requires, and udevd can be used independent of systemd's pid 1. I just think of journald as a piece of the init system that is split to a separate process, and provides log data to other programs.
- ibotty 7y agoIt also gets back logs from journald! How do you envision `systemctl status $unit` to show the last logs if they are in some random log file?
- 7y ago
- evil-olive 7y agoYes, certainly the ability to use other options is still there. But one of the main selling points of systemd (and one that the GP comment was extolling) was the consistency of systemd. And you can't have it both ways - is it great because it's a consistent, cohesive system that provides everything you need for a Linux system, or is it great because it's a modular toolkit that allows you to pick & choose which components to use? I was an Arch Linux user before, during, and after the systemd transition. I welcomed it initially, because pre-systemd Arch had a lot of "here's how to decide which logging daemon / cron daemon / etc to use" which I always thought was unnecessary. As I said, I think it's a great service manager. But I've also had to deal with production servers using systemd-timesyncd and pool.ntp.org that would occasionally have system clocks that were off by ~10 minutes. SNTP was deemed "good enough"; SNTP doesn't handle falsetickers well; and under dynamic Docker workloads systemd-timesyncd would decide "network configuration changed, better synchronize the clock" every time a Docker container started or restarted. If I use systemd as a service manager, why does that also imply I'm making "default" choices for boot loader, (S)NTP service, network manager, and container manager? Each of those has no reason to have any sort of coupling to my service manager. My biggest quibble with stories like this one is that they keep focusing on the decision of systemd as service manager / init system rather than the umbrella of other jobs that the service manager has no good reason to try to take over.
- brmgb 7y ago> And you can't have it both ways - is it great because it's a consistent, cohesive system that provides everything you need for a Linux system, or is it great because it's a modular toolkit that allows you to pick & choose which components to use? You absolutely can have it both way. Systemd is great because it offers a set of tools which are consistent, provide a lot of the functionality you want on a workstation in a cohesive way. It doesn't prevent you from choosing not to use one of the components developed under the project umbrella and uses something different instead. Systemd is modular enough for you to do that. > But I've also had to deal with production servers using systemd-timesyncd and pool.ntp.org that would occasionally have system clocks that were off by ~10 minutes. SNTP was deemed "good enough" Why are you using timesyncd if you need more than SNTP ? Timesyncd is not shipped by default with the systemd init system. It is not even installed by default on RHEL. Nothing prevents you from using a different NTP client with the systemd init system. > If I use systemd as a service manager, why does that also imply I'm making "default" choices for boot loader, (S)NTP service, network manager, and container manager? It does not imply that at all. You absolutely do not have to use the boot loader, timesyncd, networkd and nspawn with systemd. They are not even installed by default on some distributions. They are not coupled to the service manager.