4 ms·

Simply the fact that systemd turned logs into a binary format that can't be read with standard tools, and this was not (easily? at all?) possible ti cgange made

Simply the fact that systemd turned logs into a binary format that can't be read with standard tools, and this was not (easily? at all?) possible ti cgange made me really strongly dislike it. systemd has its own tool to read its binary log format, but I've already seen it corrupt its own logs and fail to read it. And did they do the binary format for efficiency? Get this: I've never seen anything be inefficient due to logging before systemd. Shortly after archlinux switched to it, something was being super slow. Sure enough, it was systemd not being able to handle the amount of ligs something produced. I think systemd is very opinionated, and something that opinionated should not be such basic piece of the linux landscape. There should be choices of individual components.

deleted 4y ago
[deleted]
blibble 4y ago
redhat seem to have tried their best to use systemd to turn Linux into Windows NT - over-complicated service manager - binary logs that are difficult to find - incomprehensible task scheduler - hidden caching dns resolution service - disk manager - network manager - login management - crappy ntp client all it needs is svchost.exe something that was 100% reliable is now about 98% reliable, and when it inevitably breaks it's completely un-introspectable by standard tooling I have yet to see a sysadmin write a 100% correct systemd service unit file
- haileys 4y ago
  Describing SysV init as 100% reliable is a very rose-tinted view of history, I think.
  - deleted 4y ago
    [deleted]
  - blueflow 4y ago
    At least you can throw strace or the -x flag onto it and look at it.
- nine_k 4y ago
  But look, having a stack that replaces many Unix services, and which you basically completely control, is very convenient. It allows, say, to drastically limit third-party userland in embedded systems, where Red Hat is big. Systemd is trying to replace Unix, a bit more successfully than GNU Hurd.
  - froh 4y ago
    this. the whole "sysv init is so slow because of shell scrips" was a scapegoat. yes bash is relatively slow. and yes, dash is the answer to that problem, not systemd. they who control systemd now control Linux as an OS. not as an API (that's kernel/libc) but as an OS. how you manage it, run it, suspend it, initialize it, turn it off, everything was RH, and now is MS.
    - nine_k 4y ago
      RH is s/MS/IBM/, quite literally.
      - lproven 4y ago
        IBM owns Red Hat. The systemd project came out of Red Hat. But Microsoft employs Lennart Pöttering, so that might be what the comment meant. Pöttering created systemd and is the main maintainer, even now.
    - pjmlp 4y ago
      Solaris: https://www.oracle.com/technical-resources/articles/solaris/intro-smf-basics-s11.html https://www.oracle.com/technical-resources/articles/solaris/... macOS: https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html https://developer.apple.com/library/archive/documentation/Ma...
- pjmlp 4y ago
  Only if we ignore the fact that commercial UNIXes already did this before systemd came to be. Also in the Cloud OS world, classical UNIX doesn't even matter that much, we only need something to run those containers or managed runtimes on top of.
dale_glass 4y ago
The binary log format has a huge advantage: it allows for arbitrary amounts of fields containing arbitrary content. Unlike syslog, journald is trivial to parse unambiguously, and you can even dump binary data like crash logs into it if you really want to. It also parses universally -- you always know what the timestamp is, you don't need to craft a per-service regex. I'm not sure what would be a better alternative. I guess it could have gone with dumping JSON, but then you have issues with newlines, or escaping, and a single bad character can break parsing. At that point you might as well go binary, IMO.
- deleted 4y ago
  [deleted]
- nine_k 4y ago
  A well-designed, well-implemented binary log format? Quite possibly so. Let's create a logging daemon that creates Apache Parquet files, for instance. Or at least uses a well-established, well-tested binary row format, with existing tools capable of easily unpacking it and working with it. Maybe journald has a good enough API to stick to it. I hope it's going to be replaced with a saner implementation of that API, like Pipewire did with Pulse Audio.
  - dale_glass 4y ago
    The format is documented: https://systemd.io/JOURNAL_FILE_FORMAT/ https://systemd.io/JOURNAL_FILE_FORMAT/ What issues do you have with it?
- throwawaylinux 4y ago
  If an application provides some standard command line tool that allows its internal data to be output as formatted text, then it in fact can be read with standard tools and manipulated with shell pipelines and scripts. Plain text files are nice where possible, but binary formats are not automatically going against the Unix Way.
  - dale_glass 4y ago
ilyt 4y ago
I like some parts systemd does but binary format is travesty. And I do not mean "the fact it is binary is bad" but format itself. Example: strace systemctl status haproxy 2>&1 |grep /var/log/journal |wc -l 356 356 files opened (whole logging dir is around 800MB) only to tell me CGroup: /system.slice/haproxy.service ├─2428 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock └─2434 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock Warning: journal has been rotated since unit was started, output may be incomplete. Yes, to tell me there are no logs for the app. And it takes multiple seconds (I tested it on NAS with spinning rust). because it opens ALL of the logfiles % /var/log/journal1 find . |wc -l 356 The bug has 6 years https://github.com/systemd/systemd/issues/2460 https://github.com/systemd/systemd/issues/2460 If it was just "a SQLite database on some sensible rotation" + maybe pointer-per-app showing where app last logged I'd actually be thrilled. Whole system logs instantly queryable by SQL queries ? Sign me in! Hell, while we're at it slap structured logging there (say if app opts in for it). But this abomination is an utter waste of time.
- nine_k 4y ago
  Fortunately binary logging can be switched off and redirected to syslog on hosts where have control but can't replace the init system. https://wiki.archlinux.org/title/Systemd/Journal#Journald_in_conjunction_with_syslog https://wiki.archlinux.org/title/Systemd/Journal#Journald_in...
  - throw0101c 4y ago
    > Fortunately binary logging can be switched off This is not switching them off. It is having the binary logs mirrored to text logs: > […] by letting systemd forward all messages via the socket /run/systemd/journal/syslog. So now you have two copies of the same information.
    - theamk 4y ago
      You use it with Storage=none, which turns off binary logging format
yarg 4y ago
That's fucking terrible. A compressed log file can be implemented with append only writes. A hash validating defragmenter can be implemented in case the compression tree becomes too lopsided. (It's a log file, if it's not small enough that you can store two copies, you're logging too much.) The reader itself has zero need to perform writes, letalone where it can corrupt the binary.

deleted 4y ago

[deleted]

blibble 4y ago

redhat seem to have tried their best to use systemd to turn Linux into Windows NT - over-complicated service manager - binary logs that are difficult to find - incomprehensible task scheduler - hidden caching dns resolution service - disk manager - network manager - login management - crappy ntp client all it needs is svchost.exe something that was 100% reliable is now about 98% reliable, and when it inevitably breaks it's completely un-introspectable by standard tooling I have yet to see a sysadmin write a 100% correct systemd service unit file

haileys 4y ago
Describing SysV init as 100% reliable is a very rose-tinted view of history, I think.
- deleted 4y ago
  [deleted]
- blueflow 4y ago
  At least you can throw strace or the -x flag onto it and look at it.
nine_k 4y ago
But look, having a stack that replaces many Unix services, and which you basically completely control, is very convenient. It allows, say, to drastically limit third-party userland in embedded systems, where Red Hat is big. Systemd is trying to replace Unix, a bit more successfully than GNU Hurd.
- froh 4y ago
  this. the whole "sysv init is so slow because of shell scrips" was a scapegoat. yes bash is relatively slow. and yes, dash is the answer to that problem, not systemd. they who control systemd now control Linux as an OS. not as an API (that's kernel/libc) but as an OS. how you manage it, run it, suspend it, initialize it, turn it off, everything was RH, and now is MS.
  - nine_k 4y ago
    RH is s/MS/IBM/, quite literally.
    - lproven 4y ago
      IBM owns Red Hat. The systemd project came out of Red Hat. But Microsoft employs Lennart Pöttering, so that might be what the comment meant. Pöttering created systemd and is the main maintainer, even now.
  - pjmlp 4y ago
    Solaris: https://www.oracle.com/technical-resources/articles/solaris/intro-smf-basics-s11.html https://www.oracle.com/technical-resources/articles/solaris/... macOS: https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html https://developer.apple.com/library/archive/documentation/Ma...
pjmlp 4y ago
Only if we ignore the fact that commercial UNIXes already did this before systemd came to be. Also in the Cloud OS world, classical UNIX doesn't even matter that much, we only need something to run those containers or managed runtimes on top of.

dale_glass 4y ago

The binary log format has a huge advantage: it allows for arbitrary amounts of fields containing arbitrary content. Unlike syslog, journald is trivial to parse unambiguously, and you can even dump binary data like crash logs into it if you really want to. It also parses universally -- you always know what the timestamp is, you don't need to craft a per-service regex. I'm not sure what would be a better alternative. I guess it could have gone with dumping JSON, but then you have issues with newlines, or escaping, and a single bad character can break parsing. At that point you might as well go binary, IMO.

deleted 4y ago
[deleted]
nine_k 4y ago
A well-designed, well-implemented binary log format? Quite possibly so. Let's create a logging daemon that creates Apache Parquet files, for instance. Or at least uses a well-established, well-tested binary row format, with existing tools capable of easily unpacking it and working with it. Maybe journald has a good enough API to stick to it. I hope it's going to be replaced with a saner implementation of that API, like Pipewire did with Pulse Audio.
- dale_glass 4y ago
  The format is documented: https://systemd.io/JOURNAL_FILE_FORMAT/ https://systemd.io/JOURNAL_FILE_FORMAT/ What issues do you have with it?
throwawaylinux 4y ago
If an application provides some standard command line tool that allows its internal data to be output as formatted text, then it in fact can be read with standard tools and manipulated with shell pipelines and scripts. Plain text files are nice where possible, but binary formats are not automatically going against the Unix Way.
- dale_glass 4y ago

ilyt 4y ago

I like some parts systemd does but binary format is travesty. And I do not mean "the fact it is binary is bad" but format itself. Example: strace systemctl status haproxy 2>&1 |grep /var/log/journal |wc -l 356 356 files opened (whole logging dir is around 800MB) only to tell me CGroup: /system.slice/haproxy.service ├─2428 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock └─2434 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock Warning: journal has been rotated since unit was started, output may be incomplete. Yes, to tell me there are no logs for the app. And it takes multiple seconds (I tested it on NAS with spinning rust). because it opens ALL of the logfiles % /var/log/journal1 find . |wc -l 356 The bug has 6 years https://github.com/systemd/systemd/issues/2460 https://github.com/systemd/systemd/issues/2460 If it was just "a SQLite database on some sensible rotation" + maybe pointer-per-app showing where app last logged I'd actually be thrilled. Whole system logs instantly queryable by SQL queries ? Sign me in! Hell, while we're at it slap structured logging there (say if app opts in for it). But this abomination is an utter waste of time.

nine_k 4y ago
Fortunately binary logging can be switched off and redirected to syslog on hosts where have control but can't replace the init system. https://wiki.archlinux.org/title/Systemd/Journal#Journald_in_conjunction_with_syslog https://wiki.archlinux.org/title/Systemd/Journal#Journald_in...
- throw0101c 4y ago
  > Fortunately binary logging can be switched off This is not switching them off. It is having the binary logs mirrored to text logs: > […] by letting systemd forward all messages via the socket /run/systemd/journal/syslog. So now you have two copies of the same information.
  - theamk 4y ago
    You use it with Storage=none, which turns off binary logging format

yarg 4y ago

That's fucking terrible. A compressed log file can be implemented with append only writes. A hash validating defragmenter can be implemented in case the compression tree becomes too lopsided. (It's a log file, if it's not small enough that you can store two copies, you're logging too much.) The reader itself has zero need to perform writes, letalone where it can corrupt the binary.