6 ms·

1. This can, and should, all be done without systemd, and not with idiosyncratic shell scripts and guesses. 2. Some of the systemd logging is binary, so good l

1. This can, and should, all be done without systemd, and not with idiosyncratic shell scripts and guesses. 2. Some of the systemd logging is binary, so good luck with that if there's a problem. 3. Have you tried non-systemd init systems other than sysvinit? 4. Yes, it is convenient when everything below your development is cenrtalized by a single entity. It can easily provide a consistently useful underpinning. But there's a price - overly strong coupling of the init system, the kernel and part of the user-space, centralized control of, well, almost all of how things work on the system, and stagnation of the ecosystem due to there being only one game in town.

geofft 5y ago
I really don't understand the scenario where binary logging is a problem. journalctl is a command just like, I dunno, gzip, and people are fine with gzipped logs. If something goes horribly wrong with your system, you're not looking at logs with an oscilloscope, you're looking at logs by mounting the disk on some other working OS - whether it's the initramfs, or a live CD, or whatever. You can run gunzip < /mnt/brokensystem/var/log/messages.1.gz from that other working OS to read text logs; you can also run journalctl --root /mnt/brokensystem to read journald logs. (Not to mention UNIX has log files that have been in a binary format since time immemorial, like utmp and wtmp.)
- 5e92cb50239222b 5y ago
  A thoughtful comment downvoted into oblivion without any explanation. Classic HN. I too don't get the hate that binary logs receive. They aren't any harder to work with than text logs. journalctl makes it easy to read logs from an offline system, too. The compression it automatically provides saves a lot of space across our machines. I know that it can be achieved without binary logs, but does any popular distribution implement compression out of the box? I know only of the reverse — Ubuntu, which is still shipping rsyslogd alongside journald, forces me to wipe rsyslogd from every machine to save space.
  - sseagull 5y ago
    I think the argument is really about transferablilty of skills. I already know how to manipulate compressed files because I have to do that in other places. And once you realize logs are just text files, I can immediately transfer all my skills of dealing with text files to dealing with logs. But now I have to learn another set of tools (or at least another command to convert it to text files). It's not really a huge issue (I'm not really a systemd hater), but there can be a bit of dread when all the tools move off of standard formats, like simple text files, to custom formats, requiring you to learn idiosyncrasies of lots of different packages. (And the reverse is true. Knowing how to use journalctl only helps me with systemd, and nowhere else. It's a piece of knowledge helpful in only one area that I cannot transfer anywhere else)
    - lanstin 5y ago
      I think you can pump all the journal log files to a text file and then open them in emacs and start writing Python to analyze ;)
    - cout 5y ago
      I think this is exactly it. Journalctl doesn't resemble any tools I already know, so I have to look up how to use it each time. Binary logs aren't the problem; we have had those for ages (e.g lastlog). I just wish journalctl were a little more familiar.
      - shawnz 5y ago
        > Journalctl doesn't resemble any tools I already know What about "tail"?
  - josephcsible 5y ago
    > I know that it can be achieved without binary logs, but does any popular distribution implement compression out of the box? Are you saying that since distros don't optimally configure a program out of the box, we should scrap and replace the whole program instead of just fixing its default configuration?
- spinax 5y ago
  > You can run gunzip < /mnt/brokensystem/var/log/messages.1.gz from that other working OS to read text logs; you can also run journalctl --root /mnt/brokensystem to read journald logs. It's subtle, and you've accidentally missed it like many do - you made an assumption that everything logs through journald. On my personal system where I do not have rsyslog (aka trying to live the journald life) I have non-journald text logging for: httpd, sa (sar/sysstat), lightdm, audit, atop, Xorg, cups, fdsync and samba. If we just stick to httpd, sa and samba (most folks know how those work) it shows how logging is way more complex than what is captured in the journal - these apps by design maintain their own logs. So now you launch your rescue ISO (let's assume modern sysrescuecd which has journalctl) and get your filesystems mounted, you have to employ two different techniques - journalctl for that single-use format and then your traditional find/grep skills for everything else. You don't know why it crashed, how it crashed and are on a fishing expedition. Was it an RPM upgrade? (logged to yum.log or rpm.log, not journald) Was it update-initramfs running out of disk space truncating your initrd? (sometimes logged to journald depending on distro, sometimes not). Don't know until you start following breadcrumbs, find/grep is the superior toolset (much like the trebuchet is the superior siege weapon). It's not that you cannot do it, it's that journald-only (no rsyslog) forces using a specific method with specific tools to access what should be extremely easy to access data. Keeping logging all in the same format (text) is what I want, it's not that I don't know how to use journalctl; I don't want to use journalctl, it's a pig in lipstick. > (Not to mention UNIX has log files that have been in a binary format since time immemorial, like utmp and wtmp.) I agree with this, they belong over in /var/lib/ somewhere and those files bug me, always have. "Just because these other guys did it" is however a logical fallacy, they are (IMO) just as wrong because they're more like database files than they are logs in my opinion. (the secure/auth.log is more a "log")
  - zxzax 5y ago
    If you want to use journald for those programs you could just configure them to pipe the logs there, or you could just disable journald logging and have it pipe its logs to the syslog. If your distro didn't configure all those programs to log to the same place, that's more of a distro configuration problem than a problem with any specific syslogger. I personally dislike having a bunch of services that try to implement their own log rotation, I would rather have that handled by the system.
- dec0dedab0de 5y ago
  Say the computer writing the log was destroyed, you were able to recover the storage, but the only computer available did not have journalctl. Or if you're writing to network storage and would like to analyze the logs from your haiku box. Text is not perfect, but it's the one thing that is always available. Also I'm not sure about the journalctl format, but in general binary formats don't handle partial corruption well. Which is something kind of important for logs.
  - geofft 5y ago
    > the only computer available did not have journalctl This is exactly what I don't understand. This is a world where no other computer exists? I have bigger things to worry about (even if you scope the problem down to to "no other computer with journalctl installed exists on my network"). > Or if you're writing to network storage and would like to analyze the logs from your haiku box. I don't have any Haiku boxes, but I do have Windows boxes. At my day job, where I'm a Linux sysadmin for a large finance company, my workstation is Windows, and I don't have admin on it. So even with conventional UNIX logs, a much more common case is - as I mentioned - that I'd want to read them from a computer that doesn't have gzip installed. But we're fine with gzip logs, because the way I'd actually do this is to get a Linux computer running. (Also, keep in mind that the filesystem itself is a binary format. If you're really worried about reading logs from Haiku, you wouldn't put /var/log on NFS because that sounds like a terrible idea, you'd log to a FAT filesystem. But nobody actually does that. Everyone's logs are on ext4 or XFS or btrfs or whatever, and nobody says those formats are a bad idea.)
    - gwillen 5y ago
      Note that not every journald/journalctl is created equal. It's easy to end up with logs written on one computer that journalctl on another computer can't read, even if the latter is a newer version, depending on which settings each one was compiled with (which is mostly a distro question.)
      - geofft 5y ago
        All right, then that seems like a reasonable objection to the format, not simply "It's a binary format." gzip files created anywhere can be read by any version of gzip. A standalone (and cross-platform) journalctl file reader seems like a useful thing to have around and not a terribly difficult thing for someone to build.
- rsync 5y ago
  "I really don't understand the scenario where binary logging is a problem." It's not a problem - it's just not UNIX.
- gwillen 5y ago
  Binary logging would be fine if it Just Worked, but my experience is that journald/journalctl does not just work, and when it breaks I don't have logs. For example, if the system clock is not monotonic, my text logs are still written sequentially in order, and it's easy to figure out what happens. It is the stated view of the journald maintainers that, if your clock is not monotonic, you get to keep both pieces. https://github.com/systemd/systemd/issues/662 https://github.com/systemd/systemd/issues/662 (The specific issue described there sounds complicated, but the fundamental problem behind it, which I have hit multiple times, is "if your system can't maintain a monotonic clock at all times, including early boot, then your logs will be mangled.") Similarly, a text log line is complete the moment it's written, even if the process that wrote it goes away immediately afterwards. Journald, by contrast, performs asynchronous metadata lookups for _each_ log line at some later time, which it apparently does not cache. This means that, when a process dies suddenly, the final messages will not appear in "journalctl -u", because they failed to get tagged with metadata. (I can't find the bug for this, so I haven't verified that it's still open and unfixed at this time. But it has certainly existed for years at this point.) So no, in principle I have no objection to _competent_ binary logs. Journald does not meet that bar.
  - saurik 5y ago
    > (I can't find the bug for this, so I haven't verified that it's still open and unfixed at this time. But it has certainly existed for years at this point.) https://github.com/systemd/systemd/issues/2913 https://github.com/systemd/systemd/issues/2913 I am so happy you mentioned this, as I am in a position of developing a daemon used by people who insist on using systemd, and I keep asking them for logs when things fail and, in fact, the most critical logs -- arguably the only ones that ever truly matter: the ones that come immediately before the daemon terminates for some reason -- are often missing. Now I know why :/. https://unix.stackexchange.com/questions/291483/how-to-fix-unit-attribution-of-last-lines-of-systemd-logging-before-exit https://unix.stackexchange.com/questions/291483/how-to-fix-u... > So no, in principle I have no objection to _competent_ binary logs. Journald does not meet that bar. Yeah... I hadn't even gotten to the end of your message before immediately going "omg I have to look into this" and the most apt description that was coming to my mind for a logging system that fails to actually store logs -- and particularly the most critical death throws to the log before termination -- is 100% "incompetent" :/.
- citrin_ru 5y ago
  1. journalctl is painfully slow. Most frequent commands I use: journalctl -u myunit --since=today and journalctl -u myunit -n 1000 take minutes on loaded servers with large amount of logs. 1st command is instant for daily rotated plain text logs and 2nd is almost instant for any plain text logs. In theory it is possible to make a binary log database which will work fast for many queries including these, but journalctl is not fast at all. 2. People are fine with gzip because log compression is usually done in a way which makes log corruption/loss highly unlikely: gzip log.ystd, fsync log.ystd.gz, rm log.ystd At any time you can reset a server and you will either have uncompressed or compressed logs, not some partially written binary blob.
theranger 5y ago
Stagnation of the ecosystem should not be an argument here as systemd is trying to replace the sysvinit that has its roots in Unix System V released almost 40 years ago.
Nursie 5y ago
> 1. This can, and should, all be done without systemd, and not with idiosyncratic shell scripts and guesses. What other system is there, right now, that can do this so well? I'm definitely onboard with the issues around tight coupling, I'm really not a fan of binary logs etc. But the unit files are pretty awesome IMHO. So serious question - what else does those as well or better?
- cpach 5y ago
  s6 does it well https://skarnet.org/software/s6/ https://skarnet.org/software/s6/
  - Nursie 5y ago
    Looks quite neat, though perhaps just a little more complex than systemd service files, I guess they essentially boil down to similar things. Where systemd has one file with a bunch of settings in, this is split into a directory of single-purpose files in s6. I'd hesitate to call it "better", but from a surface reading it seems roughly equivalent from a usability perspective.
    - cpach 5y ago
      According to the author, Laurent Bercot, the code for s6 is much cleaner than the systemd code. For anyone curious I guess the best way to find out is to compare and make a judgment of one’s own.
      - Nursie 5y ago
        While I appreciate clean code when I have to work on it, it's somewhat orthogonal to my requirements when we're talking about running systems rather than building them. I guess it could be preferred as a second-order factor.
      - lanstin 5y ago
        I would rather have complexity in the engine code than in the unit files. A clean implementation is less important than a clean interface.

geofft 5y ago

I really don't understand the scenario where binary logging is a problem. journalctl is a command just like, I dunno, gzip, and people are fine with gzipped logs. If something goes horribly wrong with your system, you're not looking at logs with an oscilloscope, you're looking at logs by mounting the disk on some other working OS - whether it's the initramfs, or a live CD, or whatever. You can run gunzip < /mnt/brokensystem/var/log/messages.1.gz from that other working OS to read text logs; you can also run journalctl --root /mnt/brokensystem to read journald logs. (Not to mention UNIX has log files that have been in a binary format since time immemorial, like utmp and wtmp.)

5e92cb50239222b 5y ago
A thoughtful comment downvoted into oblivion without any explanation. Classic HN. I too don't get the hate that binary logs receive. They aren't any harder to work with than text logs. journalctl makes it easy to read logs from an offline system, too. The compression it automatically provides saves a lot of space across our machines. I know that it can be achieved without binary logs, but does any popular distribution implement compression out of the box? I know only of the reverse — Ubuntu, which is still shipping rsyslogd alongside journald, forces me to wipe rsyslogd from every machine to save space.
- sseagull 5y ago
  I think the argument is really about transferablilty of skills. I already know how to manipulate compressed files because I have to do that in other places. And once you realize logs are just text files, I can immediately transfer all my skills of dealing with text files to dealing with logs. But now I have to learn another set of tools (or at least another command to convert it to text files). It's not really a huge issue (I'm not really a systemd hater), but there can be a bit of dread when all the tools move off of standard formats, like simple text files, to custom formats, requiring you to learn idiosyncrasies of lots of different packages. (And the reverse is true. Knowing how to use journalctl only helps me with systemd, and nowhere else. It's a piece of knowledge helpful in only one area that I cannot transfer anywhere else)
  - lanstin 5y ago
    I think you can pump all the journal log files to a text file and then open them in emacs and start writing Python to analyze ;)
  - cout 5y ago
    I think this is exactly it. Journalctl doesn't resemble any tools I already know, so I have to look up how to use it each time. Binary logs aren't the problem; we have had those for ages (e.g lastlog). I just wish journalctl were a little more familiar.
    - shawnz 5y ago
      > Journalctl doesn't resemble any tools I already know What about "tail"?
- josephcsible 5y ago
  > I know that it can be achieved without binary logs, but does any popular distribution implement compression out of the box? Are you saying that since distros don't optimally configure a program out of the box, we should scrap and replace the whole program instead of just fixing its default configuration?
spinax 5y ago
> You can run gunzip < /mnt/brokensystem/var/log/messages.1.gz from that other working OS to read text logs; you can also run journalctl --root /mnt/brokensystem to read journald logs. It's subtle, and you've accidentally missed it like many do - you made an assumption that everything logs through journald. On my personal system where I do not have rsyslog (aka trying to live the journald life) I have non-journald text logging for: httpd, sa (sar/sysstat), lightdm, audit, atop, Xorg, cups, fdsync and samba. If we just stick to httpd, sa and samba (most folks know how those work) it shows how logging is way more complex than what is captured in the journal - these apps by design maintain their own logs. So now you launch your rescue ISO (let's assume modern sysrescuecd which has journalctl) and get your filesystems mounted, you have to employ two different techniques - journalctl for that single-use format and then your traditional find/grep skills for everything else. You don't know why it crashed, how it crashed and are on a fishing expedition. Was it an RPM upgrade? (logged to yum.log or rpm.log, not journald) Was it update-initramfs running out of disk space truncating your initrd? (sometimes logged to journald depending on distro, sometimes not). Don't know until you start following breadcrumbs, find/grep is the superior toolset (much like the trebuchet is the superior siege weapon). It's not that you cannot do it, it's that journald-only (no rsyslog) forces using a specific method with specific tools to access what should be extremely easy to access data. Keeping logging all in the same format (text) is what I want, it's not that I don't know how to use journalctl; I don't want to use journalctl, it's a pig in lipstick. > (Not to mention UNIX has log files that have been in a binary format since time immemorial, like utmp and wtmp.) I agree with this, they belong over in /var/lib/ somewhere and those files bug me, always have. "Just because these other guys did it" is however a logical fallacy, they are (IMO) just as wrong because they're more like database files than they are logs in my opinion. (the secure/auth.log is more a "log")
- zxzax 5y ago
  If you want to use journald for those programs you could just configure them to pipe the logs there, or you could just disable journald logging and have it pipe its logs to the syslog. If your distro didn't configure all those programs to log to the same place, that's more of a distro configuration problem than a problem with any specific syslogger. I personally dislike having a bunch of services that try to implement their own log rotation, I would rather have that handled by the system.
dec0dedab0de 5y ago
Say the computer writing the log was destroyed, you were able to recover the storage, but the only computer available did not have journalctl. Or if you're writing to network storage and would like to analyze the logs from your haiku box. Text is not perfect, but it's the one thing that is always available. Also I'm not sure about the journalctl format, but in general binary formats don't handle partial corruption well. Which is something kind of important for logs.
- geofft 5y ago
  > the only computer available did not have journalctl This is exactly what I don't understand. This is a world where no other computer exists? I have bigger things to worry about (even if you scope the problem down to to "no other computer with journalctl installed exists on my network"). > Or if you're writing to network storage and would like to analyze the logs from your haiku box. I don't have any Haiku boxes, but I do have Windows boxes. At my day job, where I'm a Linux sysadmin for a large finance company, my workstation is Windows, and I don't have admin on it. So even with conventional UNIX logs, a much more common case is - as I mentioned - that I'd want to read them from a computer that doesn't have gzip installed. But we're fine with gzip logs, because the way I'd actually do this is to get a Linux computer running. (Also, keep in mind that the filesystem itself is a binary format. If you're really worried about reading logs from Haiku, you wouldn't put /var/log on NFS because that sounds like a terrible idea, you'd log to a FAT filesystem. But nobody actually does that. Everyone's logs are on ext4 or XFS or btrfs or whatever, and nobody says those formats are a bad idea.)
  - gwillen 5y ago
    Note that not every journald/journalctl is created equal. It's easy to end up with logs written on one computer that journalctl on another computer can't read, even if the latter is a newer version, depending on which settings each one was compiled with (which is mostly a distro question.)
    - geofft 5y ago
      All right, then that seems like a reasonable objection to the format, not simply "It's a binary format." gzip files created anywhere can be read by any version of gzip. A standalone (and cross-platform) journalctl file reader seems like a useful thing to have around and not a terribly difficult thing for someone to build.
rsync 5y ago
"I really don't understand the scenario where binary logging is a problem." It's not a problem - it's just not UNIX.
gwillen 5y ago
Binary logging would be fine if it Just Worked, but my experience is that journald/journalctl does not just work, and when it breaks I don't have logs. For example, if the system clock is not monotonic, my text logs are still written sequentially in order, and it's easy to figure out what happens. It is the stated view of the journald maintainers that, if your clock is not monotonic, you get to keep both pieces. https://github.com/systemd/systemd/issues/662 https://github.com/systemd/systemd/issues/662 (The specific issue described there sounds complicated, but the fundamental problem behind it, which I have hit multiple times, is "if your system can't maintain a monotonic clock at all times, including early boot, then your logs will be mangled.") Similarly, a text log line is complete the moment it's written, even if the process that wrote it goes away immediately afterwards. Journald, by contrast, performs asynchronous metadata lookups for _each_ log line at some later time, which it apparently does not cache. This means that, when a process dies suddenly, the final messages will not appear in "journalctl -u", because they failed to get tagged with metadata. (I can't find the bug for this, so I haven't verified that it's still open and unfixed at this time. But it has certainly existed for years at this point.) So no, in principle I have no objection to _competent_ binary logs. Journald does not meet that bar.
- saurik 5y ago
  > (I can't find the bug for this, so I haven't verified that it's still open and unfixed at this time. But it has certainly existed for years at this point.) https://github.com/systemd/systemd/issues/2913 https://github.com/systemd/systemd/issues/2913 I am so happy you mentioned this, as I am in a position of developing a daemon used by people who insist on using systemd, and I keep asking them for logs when things fail and, in fact, the most critical logs -- arguably the only ones that ever truly matter: the ones that come immediately before the daemon terminates for some reason -- are often missing. Now I know why :/. https://unix.stackexchange.com/questions/291483/how-to-fix-unit-attribution-of-last-lines-of-systemd-logging-before-exit https://unix.stackexchange.com/questions/291483/how-to-fix-u... > So no, in principle I have no objection to _competent_ binary logs. Journald does not meet that bar. Yeah... I hadn't even gotten to the end of your message before immediately going "omg I have to look into this" and the most apt description that was coming to my mind for a logging system that fails to actually store logs -- and particularly the most critical death throws to the log before termination -- is 100% "incompetent" :/.
citrin_ru 5y ago
1. journalctl is painfully slow. Most frequent commands I use: journalctl -u myunit --since=today and journalctl -u myunit -n 1000 take minutes on loaded servers with large amount of logs. 1st command is instant for daily rotated plain text logs and 2nd is almost instant for any plain text logs. In theory it is possible to make a binary log database which will work fast for many queries including these, but journalctl is not fast at all. 2. People are fine with gzip because log compression is usually done in a way which makes log corruption/loss highly unlikely: gzip log.ystd, fsync log.ystd.gz, rm log.ystd At any time you can reset a server and you will either have uncompressed or compressed logs, not some partially written binary blob.

theranger 5y ago

Stagnation of the ecosystem should not be an argument here as systemd is trying to replace the sysvinit that has its roots in Unix System V released almost 40 years ago.

Nursie 5y ago

> 1. This can, and should, all be done without systemd, and not with idiosyncratic shell scripts and guesses. What other system is there, right now, that can do this so well? I'm definitely onboard with the issues around tight coupling, I'm really not a fan of binary logs etc. But the unit files are pretty awesome IMHO. So serious question - what else does those as well or better?

cpach 5y ago
s6 does it well https://skarnet.org/software/s6/ https://skarnet.org/software/s6/
- Nursie 5y ago
  Looks quite neat, though perhaps just a little more complex than systemd service files, I guess they essentially boil down to similar things. Where systemd has one file with a bunch of settings in, this is split into a directory of single-purpose files in s6. I'd hesitate to call it "better", but from a surface reading it seems roughly equivalent from a usability perspective.
  - cpach 5y ago
    According to the author, Laurent Bercot, the code for s6 is much cleaner than the systemd code. For anyone curious I guess the best way to find out is to compare and make a judgment of one’s own.
    - Nursie 5y ago
      While I appreciate clean code when I have to work on it, it's somewhat orthogonal to my requirements when we're talking about running systems rather than building them. I guess it could be preferred as a second-order factor.
    - lanstin 5y ago
      I would rather have complexity in the engine code than in the unit files. A clean implementation is less important than a clean interface.