8 ms·

I'm a sysadmin myself and ever since I first worked with systemd when it was introduced in Arch, I fell in love with it and I've never understood why others of

I'm a sysadmin myself and ever since I first worked with systemd when it was introduced in Arch, I fell in love with it and I've never understood why others of my profession hated it for replacing a cumbersome collection of shell scripts that previously were used for init/service management.

saurik 6y ago
FWIW, I always found the "cumbersome collection of shell scripts" extremely transparent and easily hackable... I find systems that involve lots of native code that isn't like, clearly right there for me to read as I set things up, extremely opaque and difficult to debug.
- mekster 6y ago
  The point is, did you ever have to debug systemd because it was systemd that was broken and not other parts?
  - saurik 6y ago
    Well, I would argue "the point is" the ability to easily understand when and how something works is useful; and, to that end, having the behaviors of something directly in front of you is thereby valuable in a way that shouldn't be dismissed, particularly when the language it is all written in is the "lineup franca" of system administration: shell. I have often wondered when something happens and why it happens at some point and simply grepped my way through /etc seeing exactly how the extremely simple startup behavior of those "cumbersome shell scripts" worked, and taking that away and hiding those behaviors into some complex C program to save a tiny bit of startup performance seems like a sad tradeoff, particularly for people just beginning their journey of discovery (like, I ostensibly don't need it anymore, as I can now read complex C programs and have in fact carefully scrutinized the code for the various eras of Apple's launchd, and so have a pretty solid understanding of exactly how everything works... but 25 years ago, being able to do that on Slackware because it effectively just had this beautifully simple shell script that just ran stuff as if it wasn't hard... that showed me the beauty of this system and how easy it is to bootstrap these things: the world didn't actually get more complex since then in a way where any of this cruft is actually necessary).
    - kaba0 6y ago
      > to save a tiny bit of startup performance That is absolutely not the reason for systemd to exist, it is an additional benefit. How was early-boot logging, error handling, dependency handling between services implemented before? At best only in some ad-hoc way, different on each distro. The domain has an inherent complexity to it, init systems are not easy, there is no saving that. But I think systemd exposing the configurable parts in a declarative way is a net win over the previous solutions.
      - guenthert 6y ago
        >> to save a tiny bit of startup performance > That is absolutely not the reason for systemd to exist If you read the original web document announcing systemd, one could come away with the impression that it was though (even if substantiated with quite questionable performance tests using virtual machines).
        kaba0 6y ago
        According to a core developer, it was a side effect of doing things the right way: http://0pointer.de/blog/projects/the-biggest-myths.html http://0pointer.de/blog/projects/the-biggest-myths.html
    - saurik 6y ago
      lineup -> lingua :(
  - lmm 6y ago
    When systemd came out I started experiencing "I upgraded and now my system won't boot". When I got fed up enough and removed systemd (and eventually had to switch distributions as non-systemd became unsupported) that stopped happening. I'm sure there will be some sophisticated argument about how this wasn't really systemd's fault and was someone else's fault, but the correlation is pretty damn strong.
  - ragnese 6y ago
    That's a fair point, but where is the line? Do sysadmins worry about debugging Linux? Should it be written in Bash so that they can (obviously the kernel can't be written in Bash, but for the sake of argument)? As a non-sysadmin, I just feel like it's totally fair to have PID 1 be about as trusted and uniform as the kernel. If there's a bug in systemd, that's bad. If there's a bug in Linux, that's bad. But can't we at least aim for an init that is not just a bunch of Bash scripts running as root?
    - drran 6y ago
      I have more trust in bash than in C. I can enable tracing in bash script and then see how it executed exactly. IMHO, we should aim for an open standard for the init subsystem instead. For example, I want to implement my init system in Rust, but systemd is blob, which is hard to replace.
      - ragnese 6y ago
        > I have more trust in bash than in C. I can enable tracing in bash script and then see how it executed exactly. If I had to work in either day-to-day, I'd chew my hands off. :p > IMHO, we should aim for an open standard for the init subsystem instead. For example, I want to implement my init system in Rust, but systemd is blob, which is hard to replace. Well, hold on. The person I replied to was talking about being able to debug systemd. So this is very tangential to that. If we had an standard for inits, with different implementations, it would be hard to debug all of the ones that are compiled binaries. My point is still that a sysadmin should not be expected to debug PID 1 any more than they should be expected to debug a driver. Also, you can just treat systemd's APIs as the standard. GNOME, for example, depends on launchd. And, IIRC, there was some shim created so that GNOME could run on a BSD, which obviously doesn't have systemd. So, go ahead and write your init in Rust- Rust has great C API/ABI compatibility!
        drran 6y ago
        Minimal initd is very short: https://github.com/mesalock-linux/minit/blob/master/src/main.rs https://github.com/mesalock-linux/minit/blob/master/src/main... http://git.suckless.org/sinit/file/sinit.c.html http://git.suckless.org/sinit/file/sinit.c.html However, systemd is huge: 1.3 million lines of code. https://www.theregister.com/2020/01/06/linux_2020_kernel_systemd_code/ https://www.theregister.com/2020/01/06/linux_2020_kernel_sys...
  - guenthert 6y ago
    Not the application itself, no, but its configuration. Once I managed to screw up things by modifying the fstab: I forgot the (never used) fs_freq and fs_passno fields in a new entry for /tmp. mountall et. al. are fault tolerant and mount the fs regardless, but not so systemd -- it decided best course of action would be to not mount any local fs. You can imagine debugging that was a wee bit hairy, particularly since it was a head-less system.
- the_duke 6y ago
  But systemd service definitions are just as easy to copy and customise. Easier, actually, because they follow a convention and you don't have to be paranoid about screwing up a fiddly bash script.
  - drran 6y ago
    HTML is easy to write, but Firefox is hard to debug.
throw0101a 6y ago
> replacing a cumbersome collection of shell scripts that previously were used for init/service management. systemd-as-init was fine. systemd replacing udevd, syslog†, mount, ntpd, DNS, dhcpd, and probably a bunch more stuff I've forgotten or don't know about, is something else. If I wanted a tightly-coupled opaque binary blob of a system I'd run Windows. † Of course its log file is not ACID, so when it poops its pants and corrupts the file there's no way to recover it besides moving it out of the way. Though the log entries on why this happened are also corrupted so there's no root cause analysis.‡ They could have just used SQLite as the logging format and it would have probably been awesome with regards to tooling and doing queries (instead of grep-foo). But no. ‡ And journald also cannot send logs off-host in any useful format, so I end up having to run rsyslogd anyway to get things into my SIEM through something useful like RFC 3164 or 5424. But yeah: unit files are (non-sarcastically) pretty good.
- ahartmetz 6y ago
  In addition to the gratuitous tight coupling, the quality of implementation of most of these system service replacements is pretty bad! NTP and DNS clients are clearly worse (more limited and more error-prone and buggy than the separate packages they replace). The syslog replacement is only not a major issue for me because my distribution (K)Ubuntu has it configured to produce plaintext log files in addition to the binary files. I also have a problem with socket activation. There is a reason why (x)inetd fell out of favor - its whole purpose was to provide socket activation. It's a hack that can produce new error states and it can actually slow down startup compared to starting services proactively. That said, the OOM killer daemon seems to be a good thing.
  - vetinari 6y ago
    systemd-resolved and dnsmasq are the only two clients that can properly route DNS queries via the proper interface to their respective authoritative server based on the resolved hostname itself. It would be hard to argue, that systemd-resolved is worse/more-limited/more error prone/buggy than dnsmasq. The socket activation is intended for services, that are not normally needed, but occasionally they might be. It doesn't apply much to servers serving well-defined services, but for desktop, it is a godsend (together with dbus-activation). Just because you don't have use for it for your use cases doesn't mean it is not useful for other people.
yellowapple 6y ago
> cumbersome collection of shell scripts Those shell scripts never had to be cumbersome. The BSDs, for example, do away with a lot of that complexity and boilerplate quite nicely via e.g. rc.subr and rc.conf.
coldtea 6y ago
>I've never understood why others of my profession hated it Basically change aversion, and a misguided idea about the "unix way".
- tyingq 6y ago
  That may be true for the init system replacement part. But I think asking (for example) "why is systemd killing things I started with nohup and also my tmux sessions?" or "why does systemd need to replace ntpd?" are valid questions.
- drran 6y ago
  "UNIX way" is not an idea or a religion. It's a natural selection of small, understandable tools, which can be supported by any professional programmer.
  - pjmlp 6y ago
    The UNIX way is a cargo cult. I have done most commercial UNIXes starting with Xenix in 1993, none of them was (or are, regarding the surviving ones) following it. In fact some of them have introduced systemd like ideas way longer.
    - drran 6y ago
      What you think about the idea of "function", a small piece of code which does one job, but does it well? Do you think we need one big function, which will unifies interfaces of multiple different functions, lets call it "system-function"?
      - pjmlp 6y ago
        Depends, functions that take booleans and enumeration flags to change their behaviour come to mind. Also modules, objects and components packing functions together as means to make sense of disjointed functions.
    - tyingq 6y ago
api 6y ago
I never liked the cumbersome mess of shell scripts it replaced. Here's what I really don't like about systemd: (1) It's a "god object" at the system level, rolling massive amounts of functionality into a single monolithic "system manager." There are advantages to that, but it also makes it impossible to strip away complexity and leads to the core system becoming really bloated with functionality. That's the road Windows took that made Windows such a mess of cruft. (2) Poor UX... it's obtuse and counter-intuitive. This is my biggest gripe. It replaced a hard to use ball of shell scripts with... something even more confusing! Every command seems designed to be hard to remember and clunky to type. Common tasks are "undiscoverable" in that you have to search docs to figure out how to do the simplest things. Git has this same problem by the way, but to a somewhat lesser degree.

saurik 6y ago

FWIW, I always found the "cumbersome collection of shell scripts" extremely transparent and easily hackable... I find systems that involve lots of native code that isn't like, clearly right there for me to read as I set things up, extremely opaque and difficult to debug.

mekster 6y ago
The point is, did you ever have to debug systemd because it was systemd that was broken and not other parts?
- saurik 6y ago
  Well, I would argue "the point is" the ability to easily understand when and how something works is useful; and, to that end, having the behaviors of something directly in front of you is thereby valuable in a way that shouldn't be dismissed, particularly when the language it is all written in is the "lineup franca" of system administration: shell. I have often wondered when something happens and why it happens at some point and simply grepped my way through /etc seeing exactly how the extremely simple startup behavior of those "cumbersome shell scripts" worked, and taking that away and hiding those behaviors into some complex C program to save a tiny bit of startup performance seems like a sad tradeoff, particularly for people just beginning their journey of discovery (like, I ostensibly don't need it anymore, as I can now read complex C programs and have in fact carefully scrutinized the code for the various eras of Apple's launchd, and so have a pretty solid understanding of exactly how everything works... but 25 years ago, being able to do that on Slackware because it effectively just had this beautifully simple shell script that just ran stuff as if it wasn't hard... that showed me the beauty of this system and how easy it is to bootstrap these things: the world didn't actually get more complex since then in a way where any of this cruft is actually necessary).
  - kaba0 6y ago
    > to save a tiny bit of startup performance That is absolutely not the reason for systemd to exist, it is an additional benefit. How was early-boot logging, error handling, dependency handling between services implemented before? At best only in some ad-hoc way, different on each distro. The domain has an inherent complexity to it, init systems are not easy, there is no saving that. But I think systemd exposing the configurable parts in a declarative way is a net win over the previous solutions.
    - guenthert 6y ago
      >> to save a tiny bit of startup performance > That is absolutely not the reason for systemd to exist If you read the original web document announcing systemd, one could come away with the impression that it was though (even if substantiated with quite questionable performance tests using virtual machines).
      - kaba0 6y ago
        According to a core developer, it was a side effect of doing things the right way: http://0pointer.de/blog/projects/the-biggest-myths.html http://0pointer.de/blog/projects/the-biggest-myths.html
  - saurik 6y ago
    lineup -> lingua :(
- lmm 6y ago
  When systemd came out I started experiencing "I upgraded and now my system won't boot". When I got fed up enough and removed systemd (and eventually had to switch distributions as non-systemd became unsupported) that stopped happening. I'm sure there will be some sophisticated argument about how this wasn't really systemd's fault and was someone else's fault, but the correlation is pretty damn strong.
- ragnese 6y ago
  That's a fair point, but where is the line? Do sysadmins worry about debugging Linux? Should it be written in Bash so that they can (obviously the kernel can't be written in Bash, but for the sake of argument)? As a non-sysadmin, I just feel like it's totally fair to have PID 1 be about as trusted and uniform as the kernel. If there's a bug in systemd, that's bad. If there's a bug in Linux, that's bad. But can't we at least aim for an init that is not just a bunch of Bash scripts running as root?
  - drran 6y ago
    I have more trust in bash than in C. I can enable tracing in bash script and then see how it executed exactly. IMHO, we should aim for an open standard for the init subsystem instead. For example, I want to implement my init system in Rust, but systemd is blob, which is hard to replace.
    - ragnese 6y ago
      > I have more trust in bash than in C. I can enable tracing in bash script and then see how it executed exactly. If I had to work in either day-to-day, I'd chew my hands off. :p > IMHO, we should aim for an open standard for the init subsystem instead. For example, I want to implement my init system in Rust, but systemd is blob, which is hard to replace. Well, hold on. The person I replied to was talking about being able to debug systemd. So this is very tangential to that. If we had an standard for inits, with different implementations, it would be hard to debug all of the ones that are compiled binaries. My point is still that a sysadmin should not be expected to debug PID 1 any more than they should be expected to debug a driver. Also, you can just treat systemd's APIs as the standard. GNOME, for example, depends on launchd. And, IIRC, there was some shim created so that GNOME could run on a BSD, which obviously doesn't have systemd. So, go ahead and write your init in Rust- Rust has great C API/ABI compatibility!
      - drran 6y ago
        Minimal initd is very short: https://github.com/mesalock-linux/minit/blob/master/src/main.rs https://github.com/mesalock-linux/minit/blob/master/src/main... http://git.suckless.org/sinit/file/sinit.c.html http://git.suckless.org/sinit/file/sinit.c.html However, systemd is huge: 1.3 million lines of code. https://www.theregister.com/2020/01/06/linux_2020_kernel_systemd_code/ https://www.theregister.com/2020/01/06/linux_2020_kernel_sys...
- guenthert 6y ago
  Not the application itself, no, but its configuration. Once I managed to screw up things by modifying the fstab: I forgot the (never used) fs_freq and fs_passno fields in a new entry for /tmp. mountall et. al. are fault tolerant and mount the fs regardless, but not so systemd -- it decided best course of action would be to not mount any local fs. You can imagine debugging that was a wee bit hairy, particularly since it was a head-less system.
the_duke 6y ago
But systemd service definitions are just as easy to copy and customise. Easier, actually, because they follow a convention and you don't have to be paranoid about screwing up a fiddly bash script.
- drran 6y ago
  HTML is easy to write, but Firefox is hard to debug.

throw0101a 6y ago

> replacing a cumbersome collection of shell scripts that previously were used for init/service management. systemd-as-init was fine. systemd replacing udevd, syslog†, mount, ntpd, DNS, dhcpd, and probably a bunch more stuff I've forgotten or don't know about, is something else. If I wanted a tightly-coupled opaque binary blob of a system I'd run Windows. † Of course its log file is not ACID, so when it poops its pants and corrupts the file there's no way to recover it besides moving it out of the way. Though the log entries on why this happened are also corrupted so there's no root cause analysis.‡ They could have just used SQLite as the logging format and it would have probably been awesome with regards to tooling and doing queries (instead of grep-foo). But no. ‡ And journald also cannot send logs off-host in any useful format, so I end up having to run rsyslogd anyway to get things into my SIEM through something useful like RFC 3164 or 5424. But yeah: unit files are (non-sarcastically) pretty good.

ahartmetz 6y ago
In addition to the gratuitous tight coupling, the quality of implementation of most of these system service replacements is pretty bad! NTP and DNS clients are clearly worse (more limited and more error-prone and buggy than the separate packages they replace). The syslog replacement is only not a major issue for me because my distribution (K)Ubuntu has it configured to produce plaintext log files in addition to the binary files. I also have a problem with socket activation. There is a reason why (x)inetd fell out of favor - its whole purpose was to provide socket activation. It's a hack that can produce new error states and it can actually slow down startup compared to starting services proactively. That said, the OOM killer daemon seems to be a good thing.
- vetinari 6y ago
  systemd-resolved and dnsmasq are the only two clients that can properly route DNS queries via the proper interface to their respective authoritative server based on the resolved hostname itself. It would be hard to argue, that systemd-resolved is worse/more-limited/more error prone/buggy than dnsmasq. The socket activation is intended for services, that are not normally needed, but occasionally they might be. It doesn't apply much to servers serving well-defined services, but for desktop, it is a godsend (together with dbus-activation). Just because you don't have use for it for your use cases doesn't mean it is not useful for other people.

yellowapple 6y ago

> cumbersome collection of shell scripts Those shell scripts never had to be cumbersome. The BSDs, for example, do away with a lot of that complexity and boilerplate quite nicely via e.g. rc.subr and rc.conf.

coldtea 6y ago

>I've never understood why others of my profession hated it Basically change aversion, and a misguided idea about the "unix way".

tyingq 6y ago
That may be true for the init system replacement part. But I think asking (for example) "why is systemd killing things I started with nohup and also my tmux sessions?" or "why does systemd need to replace ntpd?" are valid questions.
drran 6y ago
"UNIX way" is not an idea or a religion. It's a natural selection of small, understandable tools, which can be supported by any professional programmer.
- pjmlp 6y ago
  The UNIX way is a cargo cult. I have done most commercial UNIXes starting with Xenix in 1993, none of them was (or are, regarding the surviving ones) following it. In fact some of them have introduced systemd like ideas way longer.
  - drran 6y ago
    What you think about the idea of "function", a small piece of code which does one job, but does it well? Do you think we need one big function, which will unifies interfaces of multiple different functions, lets call it "system-function"?
    - pjmlp 6y ago
      Depends, functions that take booleans and enumeration flags to change their behaviour come to mind. Also modules, objects and components packing functions together as means to make sense of disjointed functions.
  - tyingq 6y ago

api 6y ago

I never liked the cumbersome mess of shell scripts it replaced. Here's what I really don't like about systemd: (1) It's a "god object" at the system level, rolling massive amounts of functionality into a single monolithic "system manager." There are advantages to that, but it also makes it impossible to strip away complexity and leads to the core system becoming really bloated with functionality. That's the road Windows took that made Windows such a mess of cruft. (2) Poor UX... it's obtuse and counter-intuitive. This is my biggest gripe. It replaced a hard to use ball of shell scripts with... something even more confusing! Every command seems designed to be hard to remember and clunky to type. Common tasks are "undiscoverable" in that you have to search docs to figure out how to do the simplest things. Git has this same problem by the way, but to a somewhat lesser degree.