4 ms·

Problem with systemd is, startup is not deterministic. systemd starts everything it can start at the same time, leading to subtle race conditions between servic

by corty 5y ago

Problem with systemd is, startup is not deterministic. systemd starts everything it can start at the same time, leading to subtle race conditions between services and the network. If you have any dependency that isn't explicitly spelled out or any service that is falsely reported as ready, you will get funny system states on some boots, whereas it will work on others. systemd-udev does its part to make things even worse there, because it also introduces more races against kernel devices. So with systemd, even when you tested the system, you can not always be sure that it will start everything properly. Compared to sysv-init this is a regression, because init scripts usually ran sequentially and also didn't involve unreliable systemd-udev events. Compared to Solaris SMF this is pitiful. SMF takes into account service dependencies from its declarative service descriptions and computes a deterministic sequential startup order. If it has booted once, it will always boot again, in the same order, reliably. Best of all worlds.

viraptor 5y ago
> Compared to sysv-init this is a regression, because init scripts usually ran sequentially and also didn't involve unreliable systemd-udev events. This goes both ways - compared to sysv-init we finally get a good way to manage services in response to device changes.
- corty 5y ago
  It depends on the application. For laptops, sure, hotplugging stuff is important. For servers, not so much, reliability is far more important than getting your USB-stick automatically recognized. systemd is heavily skewed towards the unreliable but fast and dynamic laptop usecase.
  - viraptor 5y ago
    A server won't have many udev messages happening after boot, (Past the drive and network cards) so it shouldn't hold up and services either. Have you got specific situations in mind that are not reliable?
    - corty 5y ago
      Network cards aren't simply just cards anymore. Often you get multiple interfaces depending on each other, e.g. bridges depending on multiple ethernet interfaces which wait for STP delay to be over until they are really forwarding packets. VLAN interfaces that are dependent on their parent ethernet interface. VPN and other tunnels. So you get stuff like "start wpa_suplicant for 802.1x, take the ethernet interfaces up. start the bridge interface. check if DNS works for the VPN. then start the VPN daemon and the VPN interface. then start network services listening on the VPN interface". That would be a typical setup for a server in a remote office somewhere. Another fun situation is a server with LVM, RAID and encrypted disks (also typical for a remote office somewhere). SAS disks will come online (after spinup time), generate an event. LVM and raid detection will run, generating more events, usually dependent on one another. Device naming is in some cases dependent on the event order, so you also have to rely on systemd-udev not fucking up by-id or by-name links. Then you have cryptsetup, which decrypts the disk, generating another event, but only upon an automatic or manual all-clear and password from the (hopefully working) network. Then you arrive at the actual filesystems in fstab, after a chain of udev events, each of which (and especially the LVM and cryptsetup ones) sometimes fails. Same "fun" for SAN devices. Oh, and another fun problem is systemd exhausting randomness for too many UUIDs, so cryptsetup hangs for swap devices because those get a key from /dev/random. Of course those bugs get often fixed over time, but only until the next RedHat release arrives with yet another systemd-somethingd, networkmanager, systemd-udev-bug that introduces another new problem in the above. Usability has gone up since RH7, but reliability is definitely down.
      - viraptor 5y ago
        This is a long list of complexities, but I'm not sure how they relate to udev/systemd. Yes, setting up aggregate interfaces, VPNs and fancy drive configurations is complex and can fail due to various delays. That's not a problem with udev reliability if an underlying system fails. We had the same issues before systemd/udev was a thing. > cryptsetup hangs for swap devices because those get a key from /dev/random You can choose urandom (third column of crypttab). Systemd doesn't enforce anything here.
klodolph 5y ago
At best, ordered startup should be optional, since boot otherwise takes too long on workstations. It would be nice to have hard errors when dependencies are not explicitly declared. For example, if you don’t declare dependency on the network, your service gets put in a sandbox where it can’t access the network at all. In practice, that kind of sandboxing sounds difficult. My personal experience is that it is fairly straightforward to debug races in startup. Some service fails, you look in the log and see that it has a dependency that hasn’t started, and then add that missing dependency to the service’s declaration. Just speaking from personal experience—this doesn’t come up especially often and doesn’t require a large amount of time to fix, and the benefits of fast startup save you much more time than the extra debugging costs you.
- iso1210 5y ago
  I rarely boot my workstation, certainly no more than once a month (I don't bother with every new kernel update)
- drewg123 5y ago
  I reboot my workstation about once a month, and I couldn't care less how long it takes. What I do care about is that my workstation actually reboots predictably. If it needs manual intervention when I'm traveling, I'm screwed.
- aidenn0 5y ago
  My POST takes longer than my serialized init scripts on my workstation.
acdha 5y ago
> Compared to sysv-init this is a regression, because init scripts usually ran sequentially and also didn't involve unreliable systemd-udev events. I find this really odd because we spent decades with that exact class of problems on SysV init, in part due to the primitive dependency model and the lack of standard mechanisms to launch daemons or react to events meant so many different reinventions of that particular wheel (need Apache to wait for NFS mounts? Here’s another shell kludge to maintain!). Switching to systemd gave a single, simple answer for all of that: use the init system. I don’t know what services you manage but for the mostly Debian/RHEL environments I’ve worked on your description more fits the before state than after in my experience.
- corty 5y ago
  Apache waiting for NFS mounts is still not solved. Now Apache may depend on a file system, but it still cannot depend on "the LDAP is available so you can resolve the UIDs on that NFS". So we are a tiny step ahead maybe. But the problem isn't solved at all, and back in the day we just solved it by reordering Apache manually all to the back, maybe with a "check that UIDs are resolvable"-script in front. systemd can't do that (except if you allow "evil" shell-scripts in bootup again...).
  - zxzax 5y ago
    I don't see why you couldn't do that in systemd (or any other init), it's not hard to make a small unit that simply runs a shell script.
    - corty 5y ago
      But it is a mess. You'll need 2 files for that unit (itself and the shellscript) plus changes to all necessary dependencies to put it in the right place. A sysv-init script is just edited in the right place and you are done.
      - vetinari 5y ago
        > A sysv-init script is just edited in the right place and you are done. If you never intended to update the package, that owns the sysv-init script, that is. If you do, you will find out at update time, that you have to manually play with 3-way diff and hope, that the init script didn't change too much. You are never done. With systemd, you have an override, you don't change the original unit. It will update cleanly.
kaba0 5y ago
> If it has booted once, it will always boot again, in the same order, reliably Or not if one of them fails suddenly. That’s just a false sense of stability (not saying it could not be stable, but that is not due to having a sequential startup) Services should have proper dependency chains and that’s not that hard of a problem to overcome. Without them every other part of service management will be incorrect.

by corty 5y ago

viraptor 5y ago

> Compared to sysv-init this is a regression, because init scripts usually ran sequentially and also didn't involve unreliable systemd-udev events. This goes both ways - compared to sysv-init we finally get a good way to manage services in response to device changes.

corty 5y ago
It depends on the application. For laptops, sure, hotplugging stuff is important. For servers, not so much, reliability is far more important than getting your USB-stick automatically recognized. systemd is heavily skewed towards the unreliable but fast and dynamic laptop usecase.
- viraptor 5y ago
  A server won't have many udev messages happening after boot, (Past the drive and network cards) so it shouldn't hold up and services either. Have you got specific situations in mind that are not reliable?
  - corty 5y ago
    Network cards aren't simply just cards anymore. Often you get multiple interfaces depending on each other, e.g. bridges depending on multiple ethernet interfaces which wait for STP delay to be over until they are really forwarding packets. VLAN interfaces that are dependent on their parent ethernet interface. VPN and other tunnels. So you get stuff like "start wpa_suplicant for 802.1x, take the ethernet interfaces up. start the bridge interface. check if DNS works for the VPN. then start the VPN daemon and the VPN interface. then start network services listening on the VPN interface". That would be a typical setup for a server in a remote office somewhere. Another fun situation is a server with LVM, RAID and encrypted disks (also typical for a remote office somewhere). SAS disks will come online (after spinup time), generate an event. LVM and raid detection will run, generating more events, usually dependent on one another. Device naming is in some cases dependent on the event order, so you also have to rely on systemd-udev not fucking up by-id or by-name links. Then you have cryptsetup, which decrypts the disk, generating another event, but only upon an automatic or manual all-clear and password from the (hopefully working) network. Then you arrive at the actual filesystems in fstab, after a chain of udev events, each of which (and especially the LVM and cryptsetup ones) sometimes fails. Same "fun" for SAN devices. Oh, and another fun problem is systemd exhausting randomness for too many UUIDs, so cryptsetup hangs for swap devices because those get a key from /dev/random. Of course those bugs get often fixed over time, but only until the next RedHat release arrives with yet another systemd-somethingd, networkmanager, systemd-udev-bug that introduces another new problem in the above. Usability has gone up since RH7, but reliability is definitely down.
    - viraptor 5y ago
      This is a long list of complexities, but I'm not sure how they relate to udev/systemd. Yes, setting up aggregate interfaces, VPNs and fancy drive configurations is complex and can fail due to various delays. That's not a problem with udev reliability if an underlying system fails. We had the same issues before systemd/udev was a thing. > cryptsetup hangs for swap devices because those get a key from /dev/random You can choose urandom (third column of crypttab). Systemd doesn't enforce anything here.

klodolph 5y ago

At best, ordered startup should be optional, since boot otherwise takes too long on workstations. It would be nice to have hard errors when dependencies are not explicitly declared. For example, if you don’t declare dependency on the network, your service gets put in a sandbox where it can’t access the network at all. In practice, that kind of sandboxing sounds difficult. My personal experience is that it is fairly straightforward to debug races in startup. Some service fails, you look in the log and see that it has a dependency that hasn’t started, and then add that missing dependency to the service’s declaration. Just speaking from personal experience—this doesn’t come up especially often and doesn’t require a large amount of time to fix, and the benefits of fast startup save you much more time than the extra debugging costs you.

iso1210 5y ago
I rarely boot my workstation, certainly no more than once a month (I don't bother with every new kernel update)
drewg123 5y ago
I reboot my workstation about once a month, and I couldn't care less how long it takes. What I do care about is that my workstation actually reboots predictably. If it needs manual intervention when I'm traveling, I'm screwed.
aidenn0 5y ago
My POST takes longer than my serialized init scripts on my workstation.

acdha 5y ago

> Compared to sysv-init this is a regression, because init scripts usually ran sequentially and also didn't involve unreliable systemd-udev events. I find this really odd because we spent decades with that exact class of problems on SysV init, in part due to the primitive dependency model and the lack of standard mechanisms to launch daemons or react to events meant so many different reinventions of that particular wheel (need Apache to wait for NFS mounts? Here’s another shell kludge to maintain!). Switching to systemd gave a single, simple answer for all of that: use the init system. I don’t know what services you manage but for the mostly Debian/RHEL environments I’ve worked on your description more fits the before state than after in my experience.

corty 5y ago
Apache waiting for NFS mounts is still not solved. Now Apache may depend on a file system, but it still cannot depend on "the LDAP is available so you can resolve the UIDs on that NFS". So we are a tiny step ahead maybe. But the problem isn't solved at all, and back in the day we just solved it by reordering Apache manually all to the back, maybe with a "check that UIDs are resolvable"-script in front. systemd can't do that (except if you allow "evil" shell-scripts in bootup again...).
- zxzax 5y ago
  I don't see why you couldn't do that in systemd (or any other init), it's not hard to make a small unit that simply runs a shell script.
  - corty 5y ago
    But it is a mess. You'll need 2 files for that unit (itself and the shellscript) plus changes to all necessary dependencies to put it in the right place. A sysv-init script is just edited in the right place and you are done.
    - vetinari 5y ago
      > A sysv-init script is just edited in the right place and you are done. If you never intended to update the package, that owns the sysv-init script, that is. If you do, you will find out at update time, that you have to manually play with 3-way diff and hope, that the init script didn't change too much. You are never done. With systemd, you have an override, you don't change the original unit. It will update cleanly.

kaba0 5y ago

> If it has booted once, it will always boot again, in the same order, reliably Or not if one of them fails suddenly. That’s just a false sense of stability (not saying it could not be stable, but that is not due to having a sequential startup) Services should have proper dependency chains and that’s not that hard of a problem to overcome. Without them every other part of service management will be incorrect.