7 ms·

> I've grown to love systemd. I think I’ve grown to appreciate it, but not love it. It feels like both systems are at opposite ends of a pendulum. The RC/unit

> I've grown to love systemd. I think I’ve grown to appreciate it, but not love it. It feels like both systems are at opposite ends of a pendulum. The RC/unit system was very approachable for me. You could explain it easily, and you could get inside of it and mess around very easily. That affordance/discoverability was awesome. With systemd, simple things are nicely templated. I can find an example and tweak it to achieve what I want. Complicated things get complicated real fast.

fullstop 5y ago
The RC system was simple, I'll give you that. I love not having to track down pid files. I love not having to check if the pid file contents match a valid instance of the expected binary. The RC system was simple and also full of exceptions and corner cases.
- tluyben2 5y ago
  What is the use case for that? I never had these needs on servers. Maybe you are doing something very different from me, so I am curious. I use RC on servers and systemd on clients; things run robustly for 15+ years on many servers for me (with security updates included).
  - fullstop 5y ago
    Let's say that you're starting Apache Tomcat. You have to dig into the tomcat startup scripts and figure out where, if anywhere, it writes the pid file to disk so that you can later use it to determine if the process is running or not. If java happened to crash, though, this pid file is stale and might not point to a valid process. There's a chance that the pid has been reused, and it could have been reused by anything -- even another java process! This is important, because this pid file is used to determine which process receives the kill signal. If you get it wrong, and have the right permissions, you can accidentally kill something that you did not intend to kill. This is further complicated if you want to run multiple instances of Tomcat because now you need to have a unique path for this pid file per tomcat instance. If the thing that you're trying to run doesn't fork, you then have to execute it in the background and then store the result of $! somewhere so that you know how to kill the process later on. It's all very error prone and the process for each daemon is often different.
    - herpderperator 5y ago
      > Let's say that you're starting Apache Tomcat. You have to dig into the tomcat startup scripts and figure out where, if anywhere, it writes the pid file to disk so that you can later use it to determine if the process is running or not. https://tomcat.apache.org/tomcat-8.5-doc/windows-service-howto.html https://tomcat.apache.org/tomcat-8.5-doc/windows-service-how... The commandline argument is --PidFile and --LogPath. Most, if not all, programs allow you to customise this. It should never be a guessing game, especially when you are the one creating the init file, therefore you are the one in control of the running program.
      - fullstop 5y ago
        Those arguments are for the Windows service and don't appear to have a corresponding Linux option. On the Linux side, if things are still done the same way as they were in the past, you had to set CATALINA_PID before starting the java process. It's still a guessing game, though, even with CATALINA_PID. It is entirely possible for Java to crash (something which RC scripts do not handle, at all) and another java process starting up which happens to be assigned the same process id as the dead java process. This can not happen with systemd units because each service unit is its own Linux cgroup and it can tell which processes belong to the service.
        toast0 5y ago
        You could just run every daemon in its own jail. You don't need to chroot (unless you want to), and don't need to do any jail specific network config (unless you want to), but you could use the jail name to ensure no more than one tomcat (or tomcat-config) process, and you could use the jail name/id to capture process tree to kill it without ambiguity. With respect to servers that crash, approaches differ, but you could use an off the shelf, single purpose daemon respawner, or you could change the whole system, or you could endeavor to not have crashing servers, such that if it crashes, it's worth a human taking a look.
        throwawaysysd 5y ago
  - Spivak 5y ago
    You have to do it for correctness. Every use case needs this handled. Okay so you have a PID file somewhere /var/myservice/myservice.pid. The contents of that file is a number which is supposed to correspond to a process you can find in proc. But PIDs are recycled or more likely your PID files didn't get cleaned up on reboot. So you look at your file and it says 2567, you look up 2567 and see a running process! Done right? Well it just so happenes that a random other process was assigned that PID and your service isn't actually running. pidfd's are the real real solution to this but the short and long tail of software uses pidfiles.
    - capitainenemo 5y ago
      How about keeping pidfiles on tmpfs mounts so they do get cleaned up? I guess that'd be an organisational thing to get all the apps to change to a consistent /var/tmp so you could mount it..
      - fullstop 5y ago
        That's still not good enough. pids wrap and eventually it could point to something valid, especially on a busy system. systemd uses linux cgroups so that it knows exactly which pids belong to the group. The defaults have surely changed over the years, but pid_max used to be ~32K by default. On the system I'm typing this comment on, /proc/sys/kernel/pid_max is set to 4194304.
        cesarb 5y ago
        > The defaults have surely changed over the years, but pid_max used to be ~32K by default. On the system I'm typing this comment on, /proc/sys/kernel/pid_max is set to 4194304. The commit which changed the defaults is this one: https://github.com/systemd/systemd/commit/45497f4d3b212307569fc06bdab6a35a3e2dab07 https://github.com/systemd/systemd/commit/45497f4d3b21230756...
        lmm 5y ago
        How does systemd make sure it never reuses a cgroup, OOI? If you're worried about PIDs wrapping, surely that applies to anything randomly-generated as well.
  - znpy 5y ago
    Your nice shell-scripts in /etc/rc.d just won't handle a service crashing for any reason, at all (systemd does that) Your nice shell-scripts in /etc/rc.d will start EVERYTHING and ANYTHING just in, case, even if you don't always need it (systemd does support socket activation) Your nice shell-scripts can't handle parallelism (systemd can) Your nice shell-scripts can't reorder stuff at boot, you have to specify it by hand (systemd can via Needs/RequiredBy etc) Your nice shell-scripts are worthless if you need to run more than one instance of a service per server (with systemd having many instances of the same service is a feature, not an after-thought) Your nice shell-scripts won't help you troubleshoot a failing service (systemd will, via systemctl status AND journalctl).
    - folmar 5y ago
      SysV-RC handles all that apart from socket activation, which is the job of inetd. This is all before systemd. I must say that now it has more features and the configuration format, while could be simpler, is way better than sysvrc magic comments. I'd suggest it should be XML with a simple config file equivalent for simple cases, as the subtree update in YAML-like is a hell anyway.
    - inkyoto 5y ago
      Service restart has been a solved problem in UNIX for over 40 years. It is called inetd. It is not perfect, and it was originally created for a slightly different operating model, but it does restart crashed «services». They are, in fact, called daemons. Shell scripts are for manual service starts, stops and restarts (when the daemon does not support a restart on SIGHUP), they are not meant for the automated service restarts. Different UNIX systems have gone on to replace inetd to address specific shortcomings of the inetd model with launchd, service management facility, systemd, SAM, but the service restart is not an innovation that systemd has brought along. It is a further, fine-grained, improvement over a decade old concept and a tool + ecosystem that has implemented the concept (inetd + rc scripts).
    - anthk 5y ago
      If your service crashes, either your service or your build sucks. OpenBSD does this stuff right. You keep pushing faulty and buggy crap over and over.
- ybloviator 5y ago
  This speaks more to the issues regarding the "made of many little pieces without coordination" spelled out in the linked post than to using simple scripts for service management. Also the idea that you would have pid files scattered all over instead of /var/run seems like the sort of chaos linux spawns. The *BSD rc systems are robust and rely on a framework that abstracts stuff out, you're never going to be digging in there for a pid file's location.
  - fullstop 5y ago
    I looked into how the RC system works on one of my FreeBSD servers. You might be surprised to see how rc.subr does this -- it parses the output of 'ps' to look for the process name and matches it to what's listed in the script. This only works in the simplest of cases. If the script lists a pidfile, it finds that pid and makes sure that the process name of that pid matches what is in the script. You don't need to dig for the pid file's location, I'll give you that, but it's also possible for this robust rc subsystem to kill the wrong process under the right circumstances. My system runs unbound (dns cache), so I looked for its pidfile. It's not in /var/run, but rather wherever it happens to be specified in the config. The default is /usr/local/etc/unbound/unbound.pid. In mainstream Linux distributions, things in /var/run are likely sane if you live in the packaged world. My tomcat example was about adding software outside of the packaged world / ports tree, which is why you would have to track down the pid file to make a service. Given that unbound places its pid file outside of /var/run, which is more chaotic? I like FreeBSD, and I use it daily! I also very much like the features and organization that systemd brings to the table. 1. https://cgit.freebsd.org/ports/tree/dns/unbound/files/unbound.in?id=6eb036d0d6656a72d27b34557ed4bf1feb1cd4f0
- rbanffy 5y ago
  > The RC system was simple, I'll give you that. A lot of the staying power of Unix is that it's based on a very small set of simple, yet powerful, concepts. RC is one of them.
gerdesj 5y ago
Complicated? https://lwn.net/Articles/701549/ https://lwn.net/Articles/701549/ If you have time, read the first two articles mentioned early on.
- _0w8t 5y ago
  All the problems with NFS dependencies described in the article existed before. The system administrator was supposed to address them manually by ordering things in rc files adding sometimes various sleep pauses to wait for things to start. Systemd allowed to solve them in a reliable way at the distribution level with no need for the system administrator to do anything beyond providing the exports and fstab entries.
Matl 5y ago
I think RC was only simple if you stuck to a single distro and had a neatly predefined pattern of work. It's true that systemd is nowhere near as easy to hack/get into but at the same time I find that the largely consistent definition language/CLI give me a lot less reason to want to do so. When the need does arise, there's usually a line or two I can add to my service definition to get it to do what I want. I couldn't say the same about init scripts.

fullstop 5y ago

The RC system was simple, I'll give you that. I love not having to track down pid files. I love not having to check if the pid file contents match a valid instance of the expected binary. The RC system was simple and also full of exceptions and corner cases.

tluyben2 5y ago
What is the use case for that? I never had these needs on servers. Maybe you are doing something very different from me, so I am curious. I use RC on servers and systemd on clients; things run robustly for 15+ years on many servers for me (with security updates included).
- fullstop 5y ago
  Let's say that you're starting Apache Tomcat. You have to dig into the tomcat startup scripts and figure out where, if anywhere, it writes the pid file to disk so that you can later use it to determine if the process is running or not. If java happened to crash, though, this pid file is stale and might not point to a valid process. There's a chance that the pid has been reused, and it could have been reused by anything -- even another java process! This is important, because this pid file is used to determine which process receives the kill signal. If you get it wrong, and have the right permissions, you can accidentally kill something that you did not intend to kill. This is further complicated if you want to run multiple instances of Tomcat because now you need to have a unique path for this pid file per tomcat instance. If the thing that you're trying to run doesn't fork, you then have to execute it in the background and then store the result of $! somewhere so that you know how to kill the process later on. It's all very error prone and the process for each daemon is often different.
  - herpderperator 5y ago
    > Let's say that you're starting Apache Tomcat. You have to dig into the tomcat startup scripts and figure out where, if anywhere, it writes the pid file to disk so that you can later use it to determine if the process is running or not. https://tomcat.apache.org/tomcat-8.5-doc/windows-service-howto.html https://tomcat.apache.org/tomcat-8.5-doc/windows-service-how... The commandline argument is --PidFile and --LogPath. Most, if not all, programs allow you to customise this. It should never be a guessing game, especially when you are the one creating the init file, therefore you are the one in control of the running program.
    - fullstop 5y ago
      Those arguments are for the Windows service and don't appear to have a corresponding Linux option. On the Linux side, if things are still done the same way as they were in the past, you had to set CATALINA_PID before starting the java process. It's still a guessing game, though, even with CATALINA_PID. It is entirely possible for Java to crash (something which RC scripts do not handle, at all) and another java process starting up which happens to be assigned the same process id as the dead java process. This can not happen with systemd units because each service unit is its own Linux cgroup and it can tell which processes belong to the service.
      - toast0 5y ago
        You could just run every daemon in its own jail. You don't need to chroot (unless you want to), and don't need to do any jail specific network config (unless you want to), but you could use the jail name to ensure no more than one tomcat (or tomcat-config) process, and you could use the jail name/id to capture process tree to kill it without ambiguity. With respect to servers that crash, approaches differ, but you could use an off the shelf, single purpose daemon respawner, or you could change the whole system, or you could endeavor to not have crashing servers, such that if it crashes, it's worth a human taking a look.
        throwawaysysd 5y ago
- Spivak 5y ago
  You have to do it for correctness. Every use case needs this handled. Okay so you have a PID file somewhere /var/myservice/myservice.pid. The contents of that file is a number which is supposed to correspond to a process you can find in proc. But PIDs are recycled or more likely your PID files didn't get cleaned up on reboot. So you look at your file and it says 2567, you look up 2567 and see a running process! Done right? Well it just so happenes that a random other process was assigned that PID and your service isn't actually running. pidfd's are the real real solution to this but the short and long tail of software uses pidfiles.
  - capitainenemo 5y ago
    How about keeping pidfiles on tmpfs mounts so they do get cleaned up? I guess that'd be an organisational thing to get all the apps to change to a consistent /var/tmp so you could mount it..
    - fullstop 5y ago
      That's still not good enough. pids wrap and eventually it could point to something valid, especially on a busy system. systemd uses linux cgroups so that it knows exactly which pids belong to the group. The defaults have surely changed over the years, but pid_max used to be ~32K by default. On the system I'm typing this comment on, /proc/sys/kernel/pid_max is set to 4194304.
      - cesarb 5y ago
        > The defaults have surely changed over the years, but pid_max used to be ~32K by default. On the system I'm typing this comment on, /proc/sys/kernel/pid_max is set to 4194304. The commit which changed the defaults is this one: https://github.com/systemd/systemd/commit/45497f4d3b212307569fc06bdab6a35a3e2dab07 https://github.com/systemd/systemd/commit/45497f4d3b21230756...
      - lmm 5y ago
        How does systemd make sure it never reuses a cgroup, OOI? If you're worried about PIDs wrapping, surely that applies to anything randomly-generated as well.
- znpy 5y ago
  Your nice shell-scripts in /etc/rc.d just won't handle a service crashing for any reason, at all (systemd does that) Your nice shell-scripts in /etc/rc.d will start EVERYTHING and ANYTHING just in, case, even if you don't always need it (systemd does support socket activation) Your nice shell-scripts can't handle parallelism (systemd can) Your nice shell-scripts can't reorder stuff at boot, you have to specify it by hand (systemd can via Needs/RequiredBy etc) Your nice shell-scripts are worthless if you need to run more than one instance of a service per server (with systemd having many instances of the same service is a feature, not an after-thought) Your nice shell-scripts won't help you troubleshoot a failing service (systemd will, via systemctl status AND journalctl).
  - folmar 5y ago
    SysV-RC handles all that apart from socket activation, which is the job of inetd. This is all before systemd. I must say that now it has more features and the configuration format, while could be simpler, is way better than sysvrc magic comments. I'd suggest it should be XML with a simple config file equivalent for simple cases, as the subtree update in YAML-like is a hell anyway.
  - inkyoto 5y ago
    Service restart has been a solved problem in UNIX for over 40 years. It is called inetd. It is not perfect, and it was originally created for a slightly different operating model, but it does restart crashed «services». They are, in fact, called daemons. Shell scripts are for manual service starts, stops and restarts (when the daemon does not support a restart on SIGHUP), they are not meant for the automated service restarts. Different UNIX systems have gone on to replace inetd to address specific shortcomings of the inetd model with launchd, service management facility, systemd, SAM, but the service restart is not an innovation that systemd has brought along. It is a further, fine-grained, improvement over a decade old concept and a tool + ecosystem that has implemented the concept (inetd + rc scripts).
  - anthk 5y ago
    If your service crashes, either your service or your build sucks. OpenBSD does this stuff right. You keep pushing faulty and buggy crap over and over.
ybloviator 5y ago
This speaks more to the issues regarding the "made of many little pieces without coordination" spelled out in the linked post than to using simple scripts for service management. Also the idea that you would have pid files scattered all over instead of /var/run seems like the sort of chaos linux spawns. The *BSD rc systems are robust and rely on a framework that abstracts stuff out, you're never going to be digging in there for a pid file's location.
- fullstop 5y ago
  I looked into how the RC system works on one of my FreeBSD servers. You might be surprised to see how rc.subr does this -- it parses the output of 'ps' to look for the process name and matches it to what's listed in the script. This only works in the simplest of cases. If the script lists a pidfile, it finds that pid and makes sure that the process name of that pid matches what is in the script. You don't need to dig for the pid file's location, I'll give you that, but it's also possible for this robust rc subsystem to kill the wrong process under the right circumstances. My system runs unbound (dns cache), so I looked for its pidfile. It's not in /var/run, but rather wherever it happens to be specified in the config. The default is /usr/local/etc/unbound/unbound.pid. In mainstream Linux distributions, things in /var/run are likely sane if you live in the packaged world. My tomcat example was about adding software outside of the packaged world / ports tree, which is why you would have to track down the pid file to make a service. Given that unbound places its pid file outside of /var/run, which is more chaotic? I like FreeBSD, and I use it daily! I also very much like the features and organization that systemd brings to the table. 1. https://cgit.freebsd.org/ports/tree/dns/unbound/files/unbound.in?id=6eb036d0d6656a72d27b34557ed4bf1feb1cd4f0
rbanffy 5y ago
> The RC system was simple, I'll give you that. A lot of the staying power of Unix is that it's based on a very small set of simple, yet powerful, concepts. RC is one of them.

gerdesj 5y ago

Complicated? https://lwn.net/Articles/701549/ https://lwn.net/Articles/701549/ If you have time, read the first two articles mentioned early on.

_0w8t 5y ago
All the problems with NFS dependencies described in the article existed before. The system administrator was supposed to address them manually by ordering things in rc files adding sometimes various sleep pauses to wait for things to start. Systemd allowed to solve them in a reliable way at the distribution level with no need for the system administrator to do anything beyond providing the exports and fstab entries.

Matl 5y ago

I think RC was only simple if you stuck to a single distro and had a neatly predefined pattern of work. It's true that systemd is nowhere near as easy to hack/get into but at the same time I find that the largely consistent definition language/CLI give me a lot less reason to want to do so. When the need does arise, there's usually a line or two I can add to my service definition to get it to do what I want. I couldn't say the same about init scripts.