9 ms·

For me, systemd is the best thing since sliced bread. As a programmer, I now don't need to care about dropping privileges, managing logging, daemonization (the

For me, systemd is the best thing since sliced bread. As a programmer, I now don't need to care about dropping privileges, managing logging, daemonization (the moment I need to do the double-fork dance again, chairs will be flying, I swear), dropping into a chroot, and do half-arsed guesses "is it up and running yet?" from a convoluted mess of shell code that looks like a bunch of hair stuck down a drain for a month. I just write an easy-to-debug program which I can launch from command line and see it run, and when I'm satisfied, a systemd unit from a cookie-cutter template is going to make it run. Service dependencies are now a breeze, too. If I need to limit resources, I can just declare the limits in the unit. If I want a custom networking/mount namespace, it's taken care of.

einpoklum 5y ago
1. This can, and should, all be done without systemd, and not with idiosyncratic shell scripts and guesses. 2. Some of the systemd logging is binary, so good luck with that if there's a problem. 3. Have you tried non-systemd init systems other than sysvinit? 4. Yes, it is convenient when everything below your development is cenrtalized by a single entity. It can easily provide a consistently useful underpinning. But there's a price - overly strong coupling of the init system, the kernel and part of the user-space, centralized control of, well, almost all of how things work on the system, and stagnation of the ecosystem due to there being only one game in town.
- geofft 5y ago
  I really don't understand the scenario where binary logging is a problem. journalctl is a command just like, I dunno, gzip, and people are fine with gzipped logs. If something goes horribly wrong with your system, you're not looking at logs with an oscilloscope, you're looking at logs by mounting the disk on some other working OS - whether it's the initramfs, or a live CD, or whatever. You can run gunzip < /mnt/brokensystem/var/log/messages.1.gz from that other working OS to read text logs; you can also run journalctl --root /mnt/brokensystem to read journald logs. (Not to mention UNIX has log files that have been in a binary format since time immemorial, like utmp and wtmp.)
  - 5e92cb50239222b 5y ago
    A thoughtful comment downvoted into oblivion without any explanation. Classic HN. I too don't get the hate that binary logs receive. They aren't any harder to work with than text logs. journalctl makes it easy to read logs from an offline system, too. The compression it automatically provides saves a lot of space across our machines. I know that it can be achieved without binary logs, but does any popular distribution implement compression out of the box? I know only of the reverse — Ubuntu, which is still shipping rsyslogd alongside journald, forces me to wipe rsyslogd from every machine to save space.
    - sseagull 5y ago
      I think the argument is really about transferablilty of skills. I already know how to manipulate compressed files because I have to do that in other places. And once you realize logs are just text files, I can immediately transfer all my skills of dealing with text files to dealing with logs. But now I have to learn another set of tools (or at least another command to convert it to text files). It's not really a huge issue (I'm not really a systemd hater), but there can be a bit of dread when all the tools move off of standard formats, like simple text files, to custom formats, requiring you to learn idiosyncrasies of lots of different packages. (And the reverse is true. Knowing how to use journalctl only helps me with systemd, and nowhere else. It's a piece of knowledge helpful in only one area that I cannot transfer anywhere else)
      - lanstin 5y ago
        I think you can pump all the journal log files to a text file and then open them in emacs and start writing Python to analyze ;)
      - cout 5y ago
        I think this is exactly it. Journalctl doesn't resemble any tools I already know, so I have to look up how to use it each time. Binary logs aren't the problem; we have had those for ages (e.g lastlog). I just wish journalctl were a little more familiar.
- theranger 5y ago
  Stagnation of the ecosystem should not be an argument here as systemd is trying to replace the sysvinit that has its roots in Unix System V released almost 40 years ago.
- Nursie 5y ago
  > 1. This can, and should, all be done without systemd, and not with idiosyncratic shell scripts and guesses. What other system is there, right now, that can do this so well? I'm definitely onboard with the issues around tight coupling, I'm really not a fan of binary logs etc. But the unit files are pretty awesome IMHO. So serious question - what else does those as well or better?
  - cpach 5y ago
    s6 does it well https://skarnet.org/software/s6/ https://skarnet.org/software/s6/
    - Nursie 5y ago
      Looks quite neat, though perhaps just a little more complex than systemd service files, I guess they essentially boil down to similar things. Where systemd has one file with a bunch of settings in, this is split into a directory of single-purpose files in s6. I'd hesitate to call it "better", but from a surface reading it seems roughly equivalent from a usability perspective.
      - cpach 5y ago
        According to the author, Laurent Bercot, the code for s6 is much cleaner than the systemd code. For anyone curious I guess the best way to find out is to compare and make a judgment of one’s own.
        Nursie 5y ago
        While I appreciate clean code when I have to work on it, it's somewhat orthogonal to my requirements when we're talking about running systems rather than building them. I guess it could be preferred as a second-order factor.
        lanstin 5y ago
        I would rather have complexity in the engine code than in the unit files. A clean implementation is less important than a clean interface.
kevincox 5y ago
I agree. It puts the system administrator in control of a lot of these things. Sometimes that can be annoying (the developer knows that systems calls they need) but often it is a huge benefit. I think socket-passing especially is a huge win, it shifts a huge amount of complexity out of each application and gives huge benefits as the administrator can configure the sockets however they want without needing each application to support each feature independently. Furthermore it removes one of the most common reasons why applications need to be started as root. I wrote about this previously here: https://kevincox.ca/2021/04/15/my-ideal-service/#socket-passing https://kevincox.ca/2021/04/15/my-ideal-service/#socket-pass...
spinax 5y ago
As a systems guy with a focus more on ops, I agree. It's not all roses - journald/journalctl and binary logging can go die in a pit of fire for example - however setting LimitNOFile= in a unit is just really, really nice (as well as CPU limits and all sorts of other cgroup/namespace needs). But let me just mention again that journald/journalctl can go die in a pit of fire - if it wasn't for everyone adding rsyslog to create regular text files we would be a world of hurt. But in return we get almost complete, painless cgroup level handling right in the unit file with a simple key=value structure (so really, you don't have to know anything at all about cgroups or namespaces to be very effective). Most options have a doc for them - it's usually easier for me to find an obscure systemd setting with a nice blurb about what it does (in non-programmer speak) than it is to dig up an obscure sysctl setting, e.g. I can/could do without timesyncd and resolved (it's easy - just use chrony e.g.) but I like udevd being now part of systemd. It would be nice to not write /etc/udev/rules.d/ and instead have a foo.udev unit type, perhaps that is in our future (we do have .device units, but it's not the same - yet? the future). In this ballpark I think it's more on each distro picking and choosing - Ubuntu for example drank the kool-aide much deeper than RHEL - RHEL for example uses chrony out of the box, not timesyncd. However udevd and logind seem to be common across all distros now, as another user commented the KillBackground=yes setting in logind is just horrible to have as a default. The whole "homed" thing makes me sad that it's even being coded, I hope nobody adopts that (I dislike it for the same reason I dislike automount); someone out there wants it though. The ability to dynamically edit a unit (systemctl edit) and to dynamically alter the running service constraints (systemctl set-property), all PID file type needs are handled in /run (getting rid of the nasty SysV stale unexpected crash reboot pid problem which many scripts failed to handle properly). Users having the ability to use their own private init items (systemctl --user) is great - timers, socket activation, custom login units, all very well extended down into the user's control to leverage. I'm sort of 50/50 on cron vs. timers, that's more of a use case by use case decision (example: tossing a https://healthchecks.io https://healthchecks.io "&& curl ..." is just a lot quicker and easier in cron, but running a dyndns script on my laptop with a timer is nicer). Touching on systemctl edit, it's really easy now to show folks (think a DBA team who only has the fundamental ops skill) how to quickly chain their After= and Before= needs for start/stop of their (whatever) without having to go down a rabbit hole - it's simple to use, the words and design are accessible and familiar, the method by which it works is a little obtuse (it's rooted in understanding the "dot-d" sub-include design pattern). On RHEL at least it uses nano as the default editor, annoying to me but good for casual non-vim users and easy enough to override using $EDITOR. I used SysVinit for all the same years as everyone else (Solaris to Debian to Red Hat, ops touches it all) and wrote many my fair share of complex init units to start DB2, Oracle, java appservers (anyone remember ATG Dynamo?); systemd handles natively what 75% of that work was/is (managing PID files, watching/restarting failures, implementing namespaces/cgroups, handling dependency chains, etc.); for those complex scenarios (looking at you, Tomcat) you can still just have a unit launch a very complex shellscript "like in the old days". I haven't looked in awhile, but last time I knew in RHEL7, Red Hat did exactly that with Tomcat - just had the systemd unit launch a script. It is, however, a real bear to debug sometimes - it's far easier to "bash -x /etc/init.d/..." and figure out what in the world is going wrong than it is to debug systemd unit failures. But, the same holds true for trying to debug DBus (if you've never tried / had to, it's not fun at all without deep dbus knowledge). I would like to see the future add more ops-oriented debugging methodology - if you've every used "pcs" (the commandline tooling for Pacemaker offered by RHEL), we could really use "systemctl debug-start" type of interfaces to the commandline offer the same experience as "bash -x" days of old. There are debug settings, they're just not ergonomically dialed in for the ops user, IMHO - systemctl debug-start would save people a lot of headaches.
zh3 5y ago
Wait until it automagically fails. It may be great for less-skilled people, but for anyone running anything where it's too critical to outsource support it's then necessary to have a systemd expert inhouse (and such a person has proven extremely hard to find).
- WesolyKubeczek 5y ago
  Waiting for the last 8 years, and?
- gmmeyer 5y ago
  I have never encountered the need for this and I honestly doubt you have either. Systemd fails mysteriously far less often than poorly written init files I have seen and it is honestly not that hard to debug almost every failure I've ever encountered. And I have worked on some very large scale systems with systemd. A systemd file I helped write has run many millions of times over without issue. I honestly find this comment impossible to believe
  - mcspiff 5y ago
    I’m a big fan of systemd, but we’ve definitely run into actual bugs in it. Especially the early RHEL7 days (7.0-7.3). Some were fixed upstream, some we’ve worked around, etc.
    - WesolyKubeczek 5y ago
      In the beginning, the road was quite bumpy, and my irk was that people's use cases were being dismissed in a very cavalier fashion: like, if I didn't think of a workflow, or a use case, or a piece of software that doesn't play nice which you cannot change, it's not valid and let it burn. Oh, and whoever points these things out, or ridicules me for this attitude, is a troll or a hater. In 10 years, though, reading recent bug reports, I can see that the project's leads have grown out of it, largely. Case in point, they had recently put a bug into a 24x release that made a lot of machines, including mine, unbootable, but the fix was just as quick.
strictfp 5y ago
That's true, but you can get all that without taking over the entire system. Upstart was a more lightweigh contender to systemd which would give you all that but none of the "Enterprise Linux Userspace Daemon" crap.
tsujp 5y ago
The choice isn't systemd or roll-your-own-init-system. There are alternatives like runit, openrc etc. You can do the same painless setup (arguably even easier) with runit as the base template requirement is literally just #!/bin/sh the_executeable & Granted, logs in runit are optional but there are problems with default logging too, e.g. Docker will keep filling logs until the disk is full unless you explicitly tell it not to in either its configuration or your own custom log rotation rules. Neither of which are default.
bcrl 5y ago
It's great... until something unexpected happens. Like your NIC doesn't have link on the ethernet cable and systemd waits for minutes without allowing you to abort waiting on the network because other units depend on it. Or how about adding new buggy DNS code that doesn't work in common scenarios? Oh, sorry, here's another CVE because we didn't create enough test cases for the corner cases that are actually important. Or oops, "nobody uses ntsysv", right? Or "We don't need to implement chkconfig even though we broke it". systemd is a monolithic beast that is absorbing everything else in the system without considering that some of its decisions should be able to be disabled, and a lot of the design decisions are half baked. I don't believe in the philosophy of design its author has embraced. Progress is good, but please, stop breaking shit that has worked for decades. Anyone can write new code that partially implements a feature, but it takes real effort to responsibly migrate users from tools that worked to your new shiny half assed kitchen sink.

einpoklum 5y ago

1. This can, and should, all be done without systemd, and not with idiosyncratic shell scripts and guesses. 2. Some of the systemd logging is binary, so good luck with that if there's a problem. 3. Have you tried non-systemd init systems other than sysvinit? 4. Yes, it is convenient when everything below your development is cenrtalized by a single entity. It can easily provide a consistently useful underpinning. But there's a price - overly strong coupling of the init system, the kernel and part of the user-space, centralized control of, well, almost all of how things work on the system, and stagnation of the ecosystem due to there being only one game in town.

geofft 5y ago
I really don't understand the scenario where binary logging is a problem. journalctl is a command just like, I dunno, gzip, and people are fine with gzipped logs. If something goes horribly wrong with your system, you're not looking at logs with an oscilloscope, you're looking at logs by mounting the disk on some other working OS - whether it's the initramfs, or a live CD, or whatever. You can run gunzip < /mnt/brokensystem/var/log/messages.1.gz from that other working OS to read text logs; you can also run journalctl --root /mnt/brokensystem to read journald logs. (Not to mention UNIX has log files that have been in a binary format since time immemorial, like utmp and wtmp.)
- 5e92cb50239222b 5y ago
  A thoughtful comment downvoted into oblivion without any explanation. Classic HN. I too don't get the hate that binary logs receive. They aren't any harder to work with than text logs. journalctl makes it easy to read logs from an offline system, too. The compression it automatically provides saves a lot of space across our machines. I know that it can be achieved without binary logs, but does any popular distribution implement compression out of the box? I know only of the reverse — Ubuntu, which is still shipping rsyslogd alongside journald, forces me to wipe rsyslogd from every machine to save space.
  - sseagull 5y ago
    I think the argument is really about transferablilty of skills. I already know how to manipulate compressed files because I have to do that in other places. And once you realize logs are just text files, I can immediately transfer all my skills of dealing with text files to dealing with logs. But now I have to learn another set of tools (or at least another command to convert it to text files). It's not really a huge issue (I'm not really a systemd hater), but there can be a bit of dread when all the tools move off of standard formats, like simple text files, to custom formats, requiring you to learn idiosyncrasies of lots of different packages. (And the reverse is true. Knowing how to use journalctl only helps me with systemd, and nowhere else. It's a piece of knowledge helpful in only one area that I cannot transfer anywhere else)
    - lanstin 5y ago
      I think you can pump all the journal log files to a text file and then open them in emacs and start writing Python to analyze ;)
    - cout 5y ago
      I think this is exactly it. Journalctl doesn't resemble any tools I already know, so I have to look up how to use it each time. Binary logs aren't the problem; we have had those for ages (e.g lastlog). I just wish journalctl were a little more familiar.
theranger 5y ago
Stagnation of the ecosystem should not be an argument here as systemd is trying to replace the sysvinit that has its roots in Unix System V released almost 40 years ago.
Nursie 5y ago
> 1. This can, and should, all be done without systemd, and not with idiosyncratic shell scripts and guesses. What other system is there, right now, that can do this so well? I'm definitely onboard with the issues around tight coupling, I'm really not a fan of binary logs etc. But the unit files are pretty awesome IMHO. So serious question - what else does those as well or better?
- cpach 5y ago
  s6 does it well https://skarnet.org/software/s6/ https://skarnet.org/software/s6/
  - Nursie 5y ago
    Looks quite neat, though perhaps just a little more complex than systemd service files, I guess they essentially boil down to similar things. Where systemd has one file with a bunch of settings in, this is split into a directory of single-purpose files in s6. I'd hesitate to call it "better", but from a surface reading it seems roughly equivalent from a usability perspective.
    - cpach 5y ago
      According to the author, Laurent Bercot, the code for s6 is much cleaner than the systemd code. For anyone curious I guess the best way to find out is to compare and make a judgment of one’s own.
      - Nursie 5y ago
        While I appreciate clean code when I have to work on it, it's somewhat orthogonal to my requirements when we're talking about running systems rather than building them. I guess it could be preferred as a second-order factor.
      - lanstin 5y ago
        I would rather have complexity in the engine code than in the unit files. A clean implementation is less important than a clean interface.

kevincox 5y ago

I agree. It puts the system administrator in control of a lot of these things. Sometimes that can be annoying (the developer knows that systems calls they need) but often it is a huge benefit. I think socket-passing especially is a huge win, it shifts a huge amount of complexity out of each application and gives huge benefits as the administrator can configure the sockets however they want without needing each application to support each feature independently. Furthermore it removes one of the most common reasons why applications need to be started as root. I wrote about this previously here: https://kevincox.ca/2021/04/15/my-ideal-service/#socket-passing https://kevincox.ca/2021/04/15/my-ideal-service/#socket-pass...

spinax 5y ago

As a systems guy with a focus more on ops, I agree. It's not all roses - journald/journalctl and binary logging can go die in a pit of fire for example - however setting LimitNOFile= in a unit is just really, really nice (as well as CPU limits and all sorts of other cgroup/namespace needs). But let me just mention again that journald/journalctl can go die in a pit of fire - if it wasn't for everyone adding rsyslog to create regular text files we would be a world of hurt. But in return we get almost complete, painless cgroup level handling right in the unit file with a simple key=value structure (so really, you don't have to know anything at all about cgroups or namespaces to be very effective). Most options have a doc for them - it's usually easier for me to find an obscure systemd setting with a nice blurb about what it does (in non-programmer speak) than it is to dig up an obscure sysctl setting, e.g. I can/could do without timesyncd and resolved (it's easy - just use chrony e.g.) but I like udevd being now part of systemd. It would be nice to not write /etc/udev/rules.d/ and instead have a foo.udev unit type, perhaps that is in our future (we do have .device units, but it's not the same - yet? the future). In this ballpark I think it's more on each distro picking and choosing - Ubuntu for example drank the kool-aide much deeper than RHEL - RHEL for example uses chrony out of the box, not timesyncd. However udevd and logind seem to be common across all distros now, as another user commented the KillBackground=yes setting in logind is just horrible to have as a default. The whole "homed" thing makes me sad that it's even being coded, I hope nobody adopts that (I dislike it for the same reason I dislike automount); someone out there wants it though. The ability to dynamically edit a unit (systemctl edit) and to dynamically alter the running service constraints (systemctl set-property), all PID file type needs are handled in /run (getting rid of the nasty SysV stale unexpected crash reboot pid problem which many scripts failed to handle properly). Users having the ability to use their own private init items (systemctl --user) is great - timers, socket activation, custom login units, all very well extended down into the user's control to leverage. I'm sort of 50/50 on cron vs. timers, that's more of a use case by use case decision (example: tossing a https://healthchecks.io https://healthchecks.io "&& curl ..." is just a lot quicker and easier in cron, but running a dyndns script on my laptop with a timer is nicer). Touching on systemctl edit, it's really easy now to show folks (think a DBA team who only has the fundamental ops skill) how to quickly chain their After= and Before= needs for start/stop of their (whatever) without having to go down a rabbit hole - it's simple to use, the words and design are accessible and familiar, the method by which it works is a little obtuse (it's rooted in understanding the "dot-d" sub-include design pattern). On RHEL at least it uses nano as the default editor, annoying to me but good for casual non-vim users and easy enough to override using $EDITOR. I used SysVinit for all the same years as everyone else (Solaris to Debian to Red Hat, ops touches it all) and wrote many my fair share of complex init units to start DB2, Oracle, java appservers (anyone remember ATG Dynamo?); systemd handles natively what 75% of that work was/is (managing PID files, watching/restarting failures, implementing namespaces/cgroups, handling dependency chains, etc.); for those complex scenarios (looking at you, Tomcat) you can still just have a unit launch a very complex shellscript "like in the old days". I haven't looked in awhile, but last time I knew in RHEL7, Red Hat did exactly that with Tomcat - just had the systemd unit launch a script. It is, however, a real bear to debug sometimes - it's far easier to "bash -x /etc/init.d/..." and figure out what in the world is going wrong than it is to debug systemd unit failures. But, the same holds true for trying to debug DBus (if you've never tried / had to, it's not fun at all without deep dbus knowledge). I would like to see the future add more ops-oriented debugging methodology - if you've every used "pcs" (the commandline tooling for Pacemaker offered by RHEL), we could really use "systemctl debug-start" type of interfaces to the commandline offer the same experience as "bash -x" days of old. There are debug settings, they're just not ergonomically dialed in for the ops user, IMHO - systemctl debug-start would save people a lot of headaches.

zh3 5y ago

Wait until it automagically fails. It may be great for less-skilled people, but for anyone running anything where it's too critical to outsource support it's then necessary to have a systemd expert inhouse (and such a person has proven extremely hard to find).

WesolyKubeczek 5y ago
Waiting for the last 8 years, and?
gmmeyer 5y ago
I have never encountered the need for this and I honestly doubt you have either. Systemd fails mysteriously far less often than poorly written init files I have seen and it is honestly not that hard to debug almost every failure I've ever encountered. And I have worked on some very large scale systems with systemd. A systemd file I helped write has run many millions of times over without issue. I honestly find this comment impossible to believe
- mcspiff 5y ago
  I’m a big fan of systemd, but we’ve definitely run into actual bugs in it. Especially the early RHEL7 days (7.0-7.3). Some were fixed upstream, some we’ve worked around, etc.
  - WesolyKubeczek 5y ago
    In the beginning, the road was quite bumpy, and my irk was that people's use cases were being dismissed in a very cavalier fashion: like, if I didn't think of a workflow, or a use case, or a piece of software that doesn't play nice which you cannot change, it's not valid and let it burn. Oh, and whoever points these things out, or ridicules me for this attitude, is a troll or a hater. In 10 years, though, reading recent bug reports, I can see that the project's leads have grown out of it, largely. Case in point, they had recently put a bug into a 24x release that made a lot of machines, including mine, unbootable, but the fix was just as quick.

strictfp 5y ago

That's true, but you can get all that without taking over the entire system. Upstart was a more lightweigh contender to systemd which would give you all that but none of the "Enterprise Linux Userspace Daemon" crap.

tsujp 5y ago

The choice isn't systemd or roll-your-own-init-system. There are alternatives like runit, openrc etc. You can do the same painless setup (arguably even easier) with runit as the base template requirement is literally just #!/bin/sh the_executeable & Granted, logs in runit are optional but there are problems with default logging too, e.g. Docker will keep filling logs until the disk is full unless you explicitly tell it not to in either its configuration or your own custom log rotation rules. Neither of which are default.

bcrl 5y ago

It's great... until something unexpected happens. Like your NIC doesn't have link on the ethernet cable and systemd waits for minutes without allowing you to abort waiting on the network because other units depend on it. Or how about adding new buggy DNS code that doesn't work in common scenarios? Oh, sorry, here's another CVE because we didn't create enough test cases for the corner cases that are actually important. Or oops, "nobody uses ntsysv", right? Or "We don't need to implement chkconfig even though we broke it". systemd is a monolithic beast that is absorbing everything else in the system without considering that some of its decisions should be able to be disabled, and a lot of the design decisions are half baked. I don't believe in the philosophy of design its author has embraced. Progress is good, but please, stop breaking shit that has worked for decades. Anyone can write new code that partially implements a feature, but it takes real effort to responsibly migrate users from tools that worked to your new shiny half assed kitchen sink.