15 ms·

Systemd through the eyes of a musl distribution maintainer

167 points by awilfox 3y ago

dijit 3y ago
Before this topic boils down to: SystemD gave me cancer/SystemD cured my aids as it always seems to. Please permit me get my opinion across (as it is more nuanced) before I continue. I sincerely believe that systemd solved a problem that nobody was willing to solve, and it has every right to solve those problems any way it wants (when you ask for help you don't get to choose how you are helped after all) - my concerns boil down to the fact that the adoption it has seen has lead indirectly (or, directly) into a monoculture; and a monoculture that almost certainly will stifle innovation since a replacement will need to be bug-for-bug compatible. Prior init's were actually quite easy to replace and alternatives were often used, but these days most software assumes systemd and this situation gets worse every year. That said; and with the knowledge that while I am afraid of systemd as monoculture (and a relatively opaque one); this line "systemd, as a service manager, is not actually a bad piece of software by itself. The fact it can act as both a service manager and an inetd(8) replacement is really cool." Is something I vehemently disagree with. For starters, (x)inetd is an anti-pattern, everything I understand about systems development indicates we should be seperating concerns as much as possible, having one super-server that launches everything under one daemon is directly opposed to this. "But", I hear you thinking already: "systemd runs services as independent users, it solves that!", and I would agree with you, except now pid 0 is listening to the network instead.. That doesn't strike me as much better. If there's a bug/backdoor in your binary distributed version of systemd then you are SOL and your whole system is owned as root, but at least a bug in your application might not expose your entire inetd user. :\ It's also a common issue that inetd's architecture can lend itself to getting DoS'd harder than other more-standard daemons, except now it's your pid 0 being DoS'd; not sure how you recover from that honestly. EDIT: if you are going to downvote, please provide reasons. Sick of this holy war, lets just end it reasonably please.
- awilfox 3y ago
  The monoculture problem is one that I really tried to cover in the article, indeed. While I do agree that a "super server" listening to the network could be a new (or old, with (x)inetd) concern, a well-written and well-designed pid1 doing this isn't much more risky than having the services manage themselves. The listener could/should be just as unprivileged as the target daemon, nominally using the same uid/groups and root directory as the target as well. And indeed, systemd's .socket files support the same environment control variables (User=, Group=, SupplementaryGroups=, RootDirectory=, WorkingDirectory=, et al) that services do. So that boils down to "are people writing socket activation units correctly", which is probably "no", but could be "yes".
  - znpy 3y ago
    > The monoculture problem is one that I really tried to cover in the article, indeed. The issue is that the previous "multi-culture" really sucked real bad. It was a shit show, really. systemd is rising the bar by A LOT in terms of systems management, and most alternatives are simply not keeping up. But can you really blame it on systemd making a stellar job on its own? If anything, we could blame it on the "alternative projects" doing a fairly poor job and delivering very little. edit: systemd is so good that FreeBSD people have already started pondering if they should build something similar for themselves: https://www.youtube.com/watch?v=o_AIw9bGogo https://www.youtube.com/watch?v=o_AIw9bGogo
    - dijit 3y ago
      Fuuuuuck I hate that talk by Benno. Mostly it strawmans the opposition, says its because they dont like change primarily. Then he indicates that macos’s launchd is an improvement, despite it being one of the most painful parts of advanced macos administration. Its a talk that comes up often in these threads but it completely betrays the idea of having a fruitful dialog about the pros and cons of this design and the situations where it can be beneficial.
      - acdha 3y ago
        > Then he indicates that macos’s launchd is an improvement, despite it being one of the most painful parts of advanced macos administration. This seems like a rather hot take. Most of the Mac admins I knew when I worked in the field were quite happy to have a single standard way to solve that class of problems.
- sevagh 3y ago
  >"systemd runs services as independent users, it solves that!" It's in a terrible way, too. Not sure it improved but I recall somebody once, at a company I worked at, trying to introduce per-unix-user systemd services (e.g. `graphite` user for running Graphite), and there were some atrocious steps required like `loginctl --enable-linger` and God knows what. We moved to running everything systemd as root, it's easier (and you can specify which unixuser the actual systemd unit runs at, which is "close enough").
  - Nullabillity 3y ago
    Loginctl/per-user systemds are for managing interactive sessions. System services with User= is exactly the way you'd go for isolating background daemons.
    - ptx 3y ago
      What if you want to let a non-root user manage the systemd unit for a particular service? Is there a better way than per-user systemd instances?
      - chuckadams 3y ago
        chgrp, setfacl, sudo.conf ... lots of choices.
      - acdha 3y ago
        Check example 3.4 here for a polkit policy allowing an arbitrary user to restart a single unit: https://wiki.archlinux.org/title/Polkit https://wiki.archlinux.org/title/Polkit
        twic 3y ago
        Does this let them modify the unit file, create units, create timers, etc? User systemd allows that kind of complete self-service, and so lets you do application deployment and management without touching the root account, which is rather nice.
- ptx 3y ago
  Why is inetd an anti-pattern? Isn't separating all the work of opening ports and launching daemons into a separate component (so that it's not mixed into every other component) exactly what separation of concerns is all about?
  - pi-e-sigma 3y ago
    Inetd approach is just fine. Don't listen to to the worshippers of systemd complexity. It's like listening again the guys who insisted that JEE is the future and you are stupid because you prefer simple solutions to simple problems.
- Karellen 3y ago
  > my concerns boil down to the fact that the adoption it has seen has lead indirectly (or, directly) into a monoculture; Were there multiple implementations of sysv-init being used by different distros on Linux before systemd came along?
  - dijit 3y ago
    yes, at least 5. sysvinit, s6, openrc, runit and Solaris SMF
    - Karellen 3y ago
      Were they alternate implementations of sysv-init? Or did they do their own thing, and also happen to run sysv-init scripts for back-compat? Because systemd also runs sysv-init scripts for back-compat. Edit: Also, were any distros actually shipping those as supported init systems? I was under the impression that most of them were still in the "experimental" stage and not viable replacements (yet).
      - liveoneggs 3y ago
        OpenRC and Solaris SMF definitely did their own thing. SystemV-init and the "run levels" concept was flawed from the very start when it replaced the previous BSD-style "single user" vs "multi user" boot up. Linux only adopted SystemV style init because it was the norm in Solaris at the time. It is not a linux-ism. As this article notes - systemd is actually fine as an init system and hardly anyone denies it. It's all of the other stuff (journald, resolved, timers, etc) + tight coupling + environmental assumptions that is the problem.
      - awilfox 3y ago
        I'm unaware of any Linux distribution using SMF. If there was, I would have been running it. I didn't really like pfexec (it felt bolted on and not quite there yet), and I really really hated ZFS (and still do), but SMF made Solaris administration a joy. I don't think any distribution was using runit before systemd, but it was available in Gentoo as a sysvinit replacement and ISTR it was used in the Rails community for supervising Unicorn.
- rst 3y ago
  "Nobody was willing to solve" is a bit of a stretch when upstart got shipped at pretty much the same time. (Sponsored by the commercial home of a particular Linux distribution -- but so was systemd.)
  - acdha 3y ago
    I used Upstart a lot but it did less and had various unfixed bugs whose failure mode was becoming unmanageable. I give Ubuntu credit for starting it but was reminded of jwz’s CADT rant periodically until we upgraded to a release which used systemd instead.
- Nullabillity 3y ago
  > Prior init's were actually quite easy to replace and alternatives were often used, but these days most software assumes systemd and this situation gets worse every year. Not really. Sysvinit scripts are... scripts full of sysvinitisms (double forking and PID files, anyone?). Idiomatic systemd daemons are, comparatively speaking, very straightforward. Of course, there are a lot of nice-to-have features, but to get a running system, you should largely be able to get away with parsing Wants=, After=, and ExecStart=.
  - einpoklum 3y ago
    > Not really. Sysvinit scripts are... scripts full of sysvinitisms (double forking and PID files, anyone?). ... well, a bunch of scripts are something that's relatively easy to replace. It's not as though other system components have reliance of these scripts and their sysvinit'isms baked in. From what I've heard (though not verified) - non-systemd distributions seem to manage to work with upstart, or openrc, instead of sysvinit, without much hassle.
    - dijit 3y ago
      Previous inits were glorified loop { fork() } programs. There were many scripts, many implementations, but interestingly most were compatible with each other. The largest issue inits had was that the flexibility they provided gave distro maintainers a lot of choice in how those scripts should operate to be consistent with the rest of the system. Things like log locations. The issue to be solved is: how to determine parallelism in boot, how to supervise a process with minimum complexity, and how do we do structured logging. Interestingly: SMF solved all of these a decade before.
      - einpoklum 3y ago
        > Previous inits were glorified loop { fork() } programs. Maybe (I'm not an expert on old init systems), but current inits aren't that. > The issue to be solved is: how to determine parallelism in boot I think you're replying to another comment of mine. At any rate, there were and are alternative solutions rising to meet this challenge - which involve nothing like the behemoth which is systemd. Examples: OpenRC, runit, GNU Shepherd (sort of). https://en.wikipedia.org/wiki/OpenRC https://en.wikipedia.org/wiki/OpenRC
- einpoklum 3y ago
  > I sincerely believe that systemd solved a problem that nobody was willing to solve What problem was there which nobody was willing to solve? Also, "solving" a problem by creating a mechanism that in itself highly problematic in other ways does not necessarily count as a solution; it is a shifting-around of problems. ----- The systemd "holy war", such as it is (mostly complaints on forums and in blog posts and in personal chats; don't remember any violence in this war) - is due to four reasons: * The significant problems which systemd introduces. * The way systemd has been developed and managed as project (including some grievances with individuals). * The fact that distributions have not only adopted systemd, but made it effectively impossible to opt out of. * Faults with the process in which systemd was adopted as a required default by all of the main distributions - bypassing wide opposition without addressing its criticism. So, the adoption of systemd exposed technical-governance/power-dynamics problems in the Linux distro world, with systemd serving as the symbol for those.
  - awilfox 3y ago
    I like the off-by-one there, and choose to interpret it as intentional. The fourth item is what I find to be the most interesting. I really hadn't thought about it that way before. I remember the discussion in Fedora mailing lists around systemd and iirc the criticisms were mostly actually answered and handled. But in Debian it really wasn't and it fractured the community a lot. It is perhaps the case that the "anti-systemd" crowd wouldn't be "anti-systemd" if they felt like they had been heard. Perhaps that's the bigger lesson we should all be learning: to listen closely and respond respectfully.
    - einpoklum 3y ago
      > I like the off-by-one there, and choose to interpret it as intentional. Amongst our weaponry are such elements as... I'll come in again. https://www.goodreads.com/quotes/1495-nobody-expects-the-spanish-inquisition-our-chief-weapon-is-surprise https://www.goodreads.com/quotes/1495-nobody-expects-the-spa... > It is perhaps the case that the "anti-systemd" crowd wouldn't be "anti-systemd" if they felt like they had been heard. I would say it is the other way around. systemd would not have been adopted as the default init system, had the anti-systemd crowed not been ignored. But that's not even the main point. The grievance is not about how some individuals did not "hear". It is about the _possibility_ of such a decision being taken with that level of technical-community resistance; i.e. the expectation is that decent process would not depend only on the benevolence of those in charge. That's why it's a structural rather than a personal failure IMNSHO. And again, there's the conflation, or bundling, of the multiple decisions: 1. Offer systemd in the distribution 2. Make systemd the default option for the distribution 3. Necessitate installation & use of systemd with the distribution I am specifically pretty certain that Devuan would never have been forked if systemd were merely a configurable installation option in Debian.
- teddyh 3y ago
  > now pid 0 is listening to the network Pid 1, surely.
- acdha 3y ago
  > For starters, (x)inetd is an anti-pattern, everything I understand about systems development indicates we should be seperating concerns as much as possible, having one super-server that launches everything under one daemon is directly opposed to this. It means that this server had to be carefully secured but the benefit is that you have exactly one bit of heavily-audited code listening to the network, logging activity & problems, starting processes, changing users / dropping privileges, setting up namespaces, etc. I’ve seen a lot of code get various combinations of those wrong so I think that’s a far more nuanced problem than in your portrayal. Those are all things people should know how to do but I’ve seen Java or PHP running as root in production because someone couldn’t figure out how to drop privileges needed only at startup enough times to appreciate the benefits from systemd making it so much easier to do things right.
INTPenis 3y ago
Nice article, after years of emotional flaming finally something fair and balanced about systemd. The point about there existing no competition and no respect for glibc alternatives is very valid and to that I can only say; if you build it, they will come. Systemd had the advantage of Red Hat backing so they got a great headstart. But in general people will use whatever works. So roll your sleeves up and get cracking.
- nerdponx 3y ago
  I always appreciated this presentation for being a reasonable and balanced take, if you're interested in other examples: https://youtu.be/o_AIw9bGogo https://youtu.be/o_AIw9bGogo
- plagiarist 3y ago
  If someone built the vision in this article I would use it. I love declarative config. The article is such a good take it makes me want to try out the author's Adélie project.
- throwawaaarrgh 3y ago
  People will use what works. But there is a limit. If you have two things, one of which does everything under the sun, and the other only does mostly everything, people tend to choose the first. The more people do this, the more "weight" the first carries. Until one day, it makes no sense at all to use anything but the first. Even if it causes you tons of headaches. The more different the two are, the less and less likely that you could or would ever switch. It's too big not to use. And so, building an alternative becomes a fool's errand. Nobody would use it, because there's just too many reasons why they shouldn't or can't. (In case you don't believe me: it is now impossible to run a desktop without a half dozen Systemd shims, because the entire ecosystem is now hard coded to use Systemd. I know because my desktop is Alpine Linux, a musl distro)
  - traverseda 3y ago
    No kde?
ofrzeta 3y ago
I've found systemd-nspawn to be a great alternative to the olde chroot method for fixing systems in a rescue shell. systemd-nspawn -D /mnt mounts the system like chroot but the real value comes with systemd-nspawn -b -D /mnt/ that actually boots the system in a container.
- q2dg 3y ago
  I use systemd-nspawn as a built-in alternative to LXC/LXD.
- awilfox 3y ago
  Thanks! I hadn't considered that use case.
habitue 3y ago
Maybe a good way to get competition is to give someone a possibly dubious (but fun!) goal of writing a systemd compatible alternative in rust. Reimplementing such a massive piece of infrastructure is pretty daunting. I don't think many people who are not being paid to would embark on it and then keep up the momentum to actually cover all those edge cases and maintain a community around it. Rewriting things in rust seems to be pretty motivating though, so maybe it could be a force for good here
- nerdponx 3y ago
  If you're just looking for init and service management, the S6 system (and its wrapper 66suite) are pretty good. It's the only alternative that seems to match Systemd for feeling like a modern tool in my opinion, rather than a relic of the old days.
  - generalizations 3y ago
    Been watching that for a couple years now. At one point there were plans to integrate with alpine Linux, but it wasn’t ready yet. Has there been progress towards that?
    - awilfox 3y ago
      Yes, there is some progress it seems: https://skarnet.com/projects/service-manager.html https://skarnet.com/projects/service-manager.html
  - throwaway7356 3y ago
    Don't want s6 as it brings with it a binary configuration database (Windows registry). systemd at least has text files for config.
    - egberts1 3y ago
      My sole resistance to s6, binary config AND default-all privilege for spawned daemon. Better to stick with INI v1.4 format. https://cloanto.com/specs/ini/#escapesequences https://cloanto.com/specs/ini/#escapesequences
sevagh 3y ago
Systemd is Godsent for people who have to administer diverse Linux boxes. I can see how people who want to run their own tight ship on their machine would balk at it. I also sort of hate the systemd-resolv and the fact that it instantly rendered worthless 99% of online guides of how to unfuck your local DNS resolution.
- forinti 3y ago
  I had no opinion on systemd until I found out I could configure the mount points in fstab to magically mount themselves whenever they were needed. I have a lot of network shares and some of them caused real trouble when they failed for whatever reason. Now I really appreciate it.
  - pi-e-sigma 3y ago
    Various automount tools have been available since the 90s.
    - mariusor 3y ago
      Indeed, why use Dropbox, when FTP existed for decades?
      - pi-e-sigma 3y ago
        Except these automount tools provided exactly the same functionality that is currently provided by systemd. And did it better
        bravetraveler 3y ago
        I contest 'better'. What's the difference? It's mounted... no? Allowing 'systemd' to be aware of mounts lets you create dependencies between your mounts and services. For example: 'Requires=mnt-myfs.mount' (and 'After'). Use 'PartOf' if you want the service to get restarted during an unmount/re-mount. This integration is mint! Anything not systemd [when it's already managing your services] trying to orchestrate is, by nature, second-rate One may note the removal of the fstab generator. It's overblown. Any distribution including systemd vendors it.
t43562 3y ago
I'm enjoying artix linux with dinit. I tried a couple of other alternatives - s6 and openrc - although I wouldn't say I was that fair on openrc. dinit may lack many things - I don't know - I haven't had any reason to be unhappy with it and it seems to work in a fairly understandable way. I defined a new service (for minidlnad) and that appears to be straightforward.
Philpax 3y ago
This is quite a balanced take on the matter. Appreciated!
macNchz 3y ago
I was a big skeptic earlier on, but I have come to appreciate systemd in recent years. That said, the section on resolved is a big one–it has been one of the only things that consistently causes me annoyance over the three years since I started primarily using Linux on the desktop. After one particularly deep rabbit hole where I never actually solved a problem where local network mDNS lookups always took exactly 6 seconds, I wound up disabling it entirely in favor of unbound, which works great but revealed that a handful of other things seem to expect resolved to be present and won’t work properly without it.
kemotep 3y ago
> I prefer my init scripts to handcrafted by local artisans. Each time the computer boots up is an objet d’art. In all seriousness this is a wonderful article. Has the author/op seen this talk[0] by a FreeBSD developer on systemd? What I like about the talk is asking about what is going to happen when we need to move on to newer ways of doing things that will have advantages and disadvantages over the old way. We can see a similar story with X11 and Wayland. Systemd does do some things better than the old way but does have flaws. [0]:https://youtu.be/o_AIw9bGogo https://youtu.be/o_AIw9bGogo
chasil 3y ago
I use much functionality in systemd that was not present in SysV init, and I really appreciate it. It has never crashed any OS that I have run. However, there are a few aspects of it that are inconvenient. Automount units use an unintuitive naming scheme, and you are not free to name them as you wish (as you might for a socket unit). If you are mounting an NFS volume immediately below the root directory, you don't see the problem, but if the mount is several directories deep and/or uses ASCII symbols (non-alphanumeric), it is not pretty. Socket units require two files per port. When I am moving complex inetd.conf setups to Linux, it's far easier to implement them with busybox inetd than convert dozens/hundreds of services to unit files, despite the increased functionality. Somebody has probably written some scripting to do this. I am not aware of any include directive for my own directories, so I don't have to place everything in /etc/systemd/system. There probably is a way to do this, and I am betraying my ignorance. And my, things can get messy in a hurry in /etc/systemd/system. I don't know how to configure users to be able to maintain their own (personal) units. And lastly, it's so seductive that I have no idea how to do many things in other operating systems that I easily do in Linux. I wish this itself was not a walled garden (but I'm not leaving).
- assbuttbuttass 3y ago
  > I don't know how to configure users to be able to maintain their own (personal) units. This is something I use all the time! Just put unit files in ~/.config/systemd/user and use systemctl --user to start/enable them https://wiki.archlinux.org/title/systemd/User https://wiki.archlinux.org/title/systemd/User
  - isignal 3y ago
    Just a note that this requires the admin to enable linger if the units need to be started before the user logs in and run after they logout.
- frankjr 3y ago
  > I am not aware of any include directive for my own directories, so I don't have to place everything in /etc/systemd/system. There probably is a way to do this, and I am betraying my ignorance. Systemd by default looks at a bunch of directories depending on the context. $ systemd-analyze unit-paths (or --user / --global) https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#Unit%20File%20Load%20Path https://www.freedesktop.org/software/systemd/man/latest/syst...
nijave 3y ago
More structured metadata in journald is nice and I don't have any qualms with binary logging formats, but the author does have a point on log shipping. Additionally, reading journal files has always seemed painfully slow to me (30 seconds of 100% CPU usage) I'm not overly found of timer units, either. They seem feature rich but much more complicated to setup than a single line in a crontab. On the other hand, real dependency management between daemons, mounts, and sockets is a huge win. I can't quite remember the right terminology but afaik systemd-resolvd supports routing different domains to different DNS servers on different network interfaces which can be auto configured via DHCP. The practical implication being, you can connect to a split tunnel VPN and domains accessed over the VPN get routed to the VPN DNS server.
- akritid 3y ago
  It’s called split DNS. Can be implemented using dnsmasq
  - nijave 3y ago
    It can be implemented with dnsmasq but dnsmasq doesn't integrate with DHCP (edit, client) out of the box and requires a restart to apply configuration updates which invalidates query cache. Edit- I think split dns more commonly refers to serving different internal/private and external/public zones. In this case, the DNS servers may return the same results but if I'm connecting to a VPN running some distance away, I don't want to route all DNS requests to that server--only the ones for domains on that network. The Domains directive allows configuring this https://man.archlinux.org/man/systemd.network.5#%5BNETWORK%5D_SECTION_OPTIONS https://man.archlinux.org/man/systemd.network.5#%5BNETWORK%5... You can add a dhclient hook to reconfigure dnsmasq with `server=` directives to achieve the same thing, but, as mentioned, that still requires restarting the dnsmasq daemon. That usually manifests as: the VPN connection times out, the hook runs, dnsmasq is restarted, but a random web page or request will fail while the daemon is restarting. Sure, you could do SO_REUSEADDR or something to prevent the disruption, but, once again, this all just works out of the box with systemd-*
    - 3y ago
greyw 3y ago
Reading the section about resolved reminds me of musl having a "broken" stub resolver implementation for years (whenever the dns payload was larger than 512 bytes). "Broken" because the TCP fallback was supposedly intentionally not added for complexity reasons. DNS and all the additions seem to be really not easy to handle.
- acdha 3y ago
  Yes - that lower level networking stuff is just a minefield of edge cases which most people never need but are really annoying if you’re unlucky enough to hit them, and Linux has always had this problem where if something works for Windows the people who run those broken networks and many users will say it’s a Linux problem. A lot of the systemd decisions are probably explained by someone getting a ticket saying their Red Hat laptop takes much longer than Mac to get on the corporate network or was unable to deal with the captive portal on some hotel Wi-Fi, and they realized fixing it was going to require work across 5 different code bases.
  - 20after4 3y ago
    I think this is the most likely explanation for why systemd includes everything but the kitchen sink. For consistency, and ease of maintenance, having it all under one umbrella is certainly a benefit (for the developers of systemd.) Also, NIH is a hell of a drug.
    - deleted 3y ago
      [deleted]
    - yencabulator 3y ago
      Systemd's catalogue of kitchen sinks is getting to be truly impressive. Here's an NVMe-over-TCP server, for some reason: https://github.com/systemd/systemd/blob/main/src/storagetm/storagetm.c https://github.com/systemd/systemd/blob/main/src/storagetm/s...
      - znpy 3y ago
frankjr 3y ago
I generally embrace systemd and have been pretty happy with it but there's one component which simply doesn't work correctly and that's systemd-resolved in combination with DNSSEC. I eventually had to replace it with Knot Resolver which works flawlessly on the same machine / network. https://github.com/systemd/systemd/issues/9867 https://github.com/systemd/systemd/issues/9867 https://www.knot-resolver.cz/ https://www.knot-resolver.cz/
remram 3y ago
I think the problem of systemd is that it is complex. It is fine when it works correctly, but when you eventually have to dig into any sort of problem, it is hell. For example, all of my boxes spew many logs per minute about "Failed to set up mount unit". This is apparently a bug with the generated name of some internal unit related to mountpoints (too long). This one is not as bad as others, because it does have a bug reported for it, but is still something I'll have to deal with until I upgrade the distro on all the servers. Many similar bugs I can't track down at all. Systemd makes a lot of things easy, but not simple, and that is a big problem in practice.
- twic 3y ago
  This is it. For example, systemd (a) allows you to run services as a normal user, not as root, (b) treats child processes of a service process as part of that service, and (c) collects logs from services. Except if you try to do all three, that just doesn't work, because they've half-arsed the log collection process: https://serverfault.com/a/1143851/92104 https://serverfault.com/a/1143851/92104 There's no reason for this not to work! Taking a very straightforward and traditional approach to running processes (as in my demo code in that question), you don't have this problem. But rather than that, they did something clever which doesn't work properly.
  - thwarted 3y ago
    I can't make heads or tails of either of those answers, and they seem to contradict each other. The output capturing is all to the same file descriptors, this is the way resources are inherited on fork (and there's no redirection here). That it works with unbuffer is weird because the subshell and subprocess both end up flushing their output when they exit, so there should be no need to flush before exiting. If the first answer is correct, then the unbuffer works because the process introspection as the source of the output is unified to be the unbuffer process. But why bother introspecting the exact process that originated a write to the output? Did we have a problem with file descriptors being highjacked? And even if we did, losing the output (or not associating it with the parent process) is worse.
ivolimmen 3y ago
I like the honesty of the article. I never got the grunge others have against it. I am a software engineer and work in office automation (a.k.a. the boring stuff). On Linux I am a poweruser. If Ubuntu used initd I use it. They switched to systemd so I use that. I liked it. Making an application start on reboot was easy to do. And easier than with the rc files 20 years back.
traverseda 3y ago
I originally wrote this comment for reddit, but I feel like hackernews might appreciate it. If you think it's too long or detracting from the article or something feel free to downvote, I won't be offended. Here are some of my criticisms, although I do still use systemd daily on my personal devices and servers. * Bad security Systemd is architected in a way that has a lot of code running as root, it's also written in a language that isn't memory safe. This means it has a large attack surface (a lot of code you need to make sure is bug free to be secure) and it's harder to make sure your code doesn't have really severe security related bugs (The [NSA recommends](https://www.nsa.gov/Press-Room/Press-Releases-Statements/Press-Release-View/Article/3608324/us-and-international-partners-issue-recommendations-to-secure-software-products/ https://www.nsa.gov/Press-Room/Press-Releases-Statements/Pre...) using memory safe languages for critical stuff like this). There are certainly other critical projects (like this linux kernel) that are similarly important and written in memory unsafe languages, but systemd has had some [pretty critical vulnerabilities](https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=systemd https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=systemd) discovered that do not inspire confidence in their ability to use these kinds of dangerous languages safely, and they don't have nearly the same budget as the linux kernel for detecting and preventing these kinds of issues. This is less of a problem if you're a large enterprise customer running up to date SELinux, but it should still have been possible to write systemd in a way that limited the pid1 attack surface while retaining all current functionality. * Journalctl is a pain for desktop users and smaller teams Journalctl is how systemd manages logs. By default it saves logs in a binary format with additional metadata. This makes it easier for large teams to ingest the logs into a centralized log-collection daemon, like the ones offered by redhat for enterprise deployments, but it breaks a lot of workflows that older sysadmins probably used. Things like just rsync-ing a bunch of logs to one place, or using tools like grep and find to inspect logs. Systemd does of course provide replacements for those tools, instead of using grep you can use journalctl to search through your logs, but you could use grep to search through any text file. Config files, source code, or logs. Now I need to memorize all the flags for one more tool, and change a bunch of stuff about how I collect logs. This also presents a challenge for people doing embedded work, as you can't just grab the SD card out of a system can look at it's logs. You need a working journalctl CLI on your host machine. They've add in a --directory flag, but in the past this was much harder, requiring you to actually chroot into the embedded system (that may be broken in strange ways) in order to read logs. Journalctl has advantages, but they're mostly enjoyed by large enterprises. Yes I know it's actually systemd-journal or just "journal" or whatever they call it. Journalctl is the command most users will be familiar with though. * Poor [Locality of behavior](https://htmx.org/essays/locality-of-behaviour/ https://htmx.org/essays/locality-of-behaviour/) makes it harder to reason about When you're trying to understand how a system works it's nice to be able to see everything in one places. There are like 9 different places a systemd unit file can live, you can apply an over-ride to a unit file, unit files can depend on other files like socket files. This is good for large teams as it makes it easier for specific groups in a company to claim ownership over parts of the system, but it means as a desktop user or sysadmin working with a small business you've added a lot of complexity. You can't just type `ls /etc/init.d` to get a rough overview of what services exist, you need to memorize more systemd-specific commands. If you want to edit a service you can't just edit a service, you need to create an over-ride using another systemd specific command, make sure you have the EDITOR environment variable set up, and then open the original service in another editor so you can compare the two. It creates some more work and complexity and encourages you to use a bunch of systemd-specific tools (and presumably get red-hat certified training). * People use systemd stuff before it's ready I'm not sure this is something I can blame systemd or redhat for, but the official stance of redhat is that systemd-resolved is not ready for production, and yet it's used all over the place. That can give people a poor impression of systemd after the 9th time they try to do something even slightly different with their networking setups and systemd-resolvd breaks, not to mention the numerous security issues in systemd-resolvd. * Unix philosophy A lot of people say that either unix philosophy doesn't matter, or that systemd does embrace unix philosophy. That stuff about logging and systemd-specific tools I mentioned above? That's what people actually mean when they talk about unix philosophy, they mean being able to grep through their logs and rsync logs to a remote server. They mean using standard text files and not needing to have a special command that wraps your text editor to "properly" edit a unit file. * Doesn't run in a chroot As someone who splits my time between embedded linux and server linux this is just a personal pet peeve of mine. It makes it very hard to debug embedded systems that use systemd if you're not using systemd on your workstation. It does feel like I'm being forced to use systemd sometimes, and while I've largely gotten over it I'm still a bit bitter. It's also made some small personal projects, like getting a full linux distro running on a KoBo e-reader, much much more difficult than they had to be. It's a mess under docker, and why I need to use alternative OCI-runtimes like nestybox to do a bunch of testing for embedded systems. Thankfully I wasn't an early adopter to docker and didn't have those problems until there were already mature solutions. But there's really no reason why it should have to run as pid1, other than them wanting you to use docker-alternatives that are deeply integrated with systemd, like their podman tool or systemd-nspawn. This was just such a blatant attempt to abuse their near-monopoly position that it bears some extra whining. * OpenRc does everything systemd does, but better Unfortunately other red-hat influenced projects like Gnome won't support or test on non-systemd init systems, let alone providing default services files for them, meaning that any distro that wants to be compatible with gnome will need to do a bunch of extra work to write and test service files. For one project that's potentially reasonable, and certainly there are distros that do that extra work, but for projects like Arch who have the explicit goal of sticking as close to upstream sources as possible it makes it more or less impossible. Systemd survives not because it's a good solution to the problem, but because it has a large corporate backer, is widely deployed, and is a safe thing to code against. There's a very old IT saying, "No one ever got fired for buying IBM". If you pick a safe industry-standard options no one can blame you if it goes wrong, even if it's the technologically inferior option. As much as I'm a systemd-hater I still do use it on my personal devices and servers, because it's by far the path of least resistance. I hope we see a similar situation like with pulseaudio and pipewire, where the pulseaudio rewrite was much much nicer than the original. I don't think that's going to happen until systemd slows down though, right now if you tried to re-implement systemd I suspect you'd get the rug pulled out from under you as they changed standards and behaviors (I've seriously thought about doing it myself, at least for relatively simple unit files). I'd still prefer to be using OpenRc, as I don't know how a rewrite would deal with the locality-of-behavior issues, but systemd has been getting better and more reliable over time.
deleted 3y ago
[deleted]
gavinhoward 3y ago
I think the author is right on just about everything. But the author is especially right on the need for competition. So I am working on it. I am building a build system, and I am making it usable as a library. When I have made it so, I will implement an near drop-in [1] replacement for systemd. You'll be able to use systemd unit files, and there will be an option to have binaries with the same name (off by default to not interfere, though) so that users don't have to learn new stuff right away. [1]: I am not going to implement journald for example; logs will be text, even if they are structured internally. Besides, implementing things to be perfectly compatible would only strengthen the monoculture, not weaken it. The plan is to be compatible on the things that matter for distro integration, so that distros can ship both with little work, but have other things different, so that users can choose what works best for them between the two.
- traverseda 3y ago
  Have you considered integrating this with OpenRC?
  - gavinhoward 3y ago
    I am not sure what you mean.
- twic 3y ago
  In a similar vein, but already existing, nosh lets you use systemd unit files and commands via a shim layer: https://jdebp.uk/Softwares/nosh/ https://jdebp.uk/Softwares/nosh/
  - throwaway7356 3y ago
    Does anyone use it? It there a distribution (BSD or Linux) using nosh?
- vetinari 3y ago
  > I think the author is right on just about everything. The author is very wrong on the systemd-resolved bit. Not understanding the ability to have per-interface specific zones is ok; it is a thing for desktops with multiple interfaces that come and go (like VPNs, for example). There is no other resolver on Linux capable of doing it and integrating with NetworkManager. You can kinda-sorta make dnsmasq do it, but with some limitations. But the crown is taken by the "mDNS is better taken care of by a dedicated package like Avahi." Uh, oh. It is like saying, that you don't need Chrome, it's duties are better taken care by nginx. Similarly, you can use Avahi for _advertising_ mDNS services, but not for _resolving_. Which, as an user, you probably are interested in.
smitty1e 3y ago
The Famous Article (TFA) is an exemplar of dispassionate substantive, liberal criticism. The shortcomings of systemd seem likely the overall shortcomings of open source: none of the $PROJECT maintainers experience $PAIN_POINT, so it is simply not a priority for $PROJECT. Somehow this seems a variation on the tragedy of the commons => https://en.m.wikipedia.org/wiki/Tragedy_of_the_commons https://en.m.wikipedia.org/wiki/Tragedy_of_the_commons Without some capitalist skin in the game, the $PAIN_POINTS become difficult to prioritize.
WesolyKubeczek 3y ago
I would in fact love a portable user-level process manager that can speak systemd unit files and is portable. Something like supervisord, but you use systemd unit files, and it tries to do as much as it can within limitations (process tracking sure can be wonky, no dbus everywhere, no cgroups everywhere, no absolute freedom in resource limitations if you’re not PID 1).
- bmicraft 3y ago
  Process tracking without cgroups seems like an absolute nightmare w.r.t. race conditions
peter_d_sherman 3y ago
Most people (including most Unix greybeards!) really don't understand Unix's 'init' (AKA "the init daemon", "the init process", etc., etc.) -- much less any of its massively-increasing-in-LOC (and complexity!) successor programs... So we need to start with 'init'. 'init' -- even in its absolute first, simplest incarnation -- is still too complex to understand correctly! You see, we need to shift perspectives! We need to shift perspectives from a longtime System Administrator -- to that of a new barebone OS programmer. What is 'init'? Is 'init' a program that handles runlevels, starts and stops services, that mounts filesystems, that processes messages, that captures dead processes, that waits for hardware to become available, that logs and maintains informational/database/etc files, that starts audio, that starts X11, that stars the GUI, that acts as a proxy for sockets, or does anything else with the system? No! From the point of view of a new barebone OS programmer (as Dennis Richie and Ken Thompson were when they invented Unix and invented 'init') -- 'init' is NONE of these things! 'init' is only THE FIRST PROGRAM, THE FIRST COMPUTER CODE THAT RUNS IN USER SPACE. And that's it! That is all that 'init' ever is, or ever was! (User space, to recap, is the unprotected AKA "unprivileged" AKA "non-supervisor" memory running unprotected (AKA "user-land") code: https://en.wikipedia.org/wiki/User_space_and_kernel_space https://en.wikipedia.org/wiki/User_space_and_kernel_space) 'init' (and every single 'init' successor program, i.e., OpenRC, systemd, etc.) -- are the first program, the first set of computer code OUTSIDE OF KERNEL CODE (which has been running and is currently still running) to be run by the system. Now, what should that first program do? See, that's the magic question -- which gives rise to all that is to follow! In theory you could have an OS where the 'init' program, or its equivalent -- did absolutely nothing! But that wouldn't be very productive! If the 'init' program isn't itself a shell program (i.e., sh, bash, etc.) -- then (because there's no GUI at this point) the computer will not be able to accept typed command-line commands -- which is the first thing that you want a new OS to do! So now our 'init' expands in scope (and lines of code)! Our 'init' could be hardcoded to launch 'sh' or 'bash' (or whatever shell program exists) -- but what if the user wants to change that? OK, so now we need our first configuration file. Where to put that exactly? Oh, it's on a filesystem that hasn't been mounted yet? Well, maybe init should mount that filesystem! Point is, there's a set of problems (and sub-problems!) -- which give rise to increasing and increasing init's functionality over time! init, as the first user-space program for an OS to run, on whatever OS it is ran on, in whatever form it is in -- could simply be written to run and 'outsource' all of its functionality to other programs... But init (as it evolved into its very large LOC complex descendants) -- became a "dumping ground" -- for functionality that was inconvenient to go in other places and/or to be outsourced to other programs. See, all of the code in all userland Linux utilities -- could in theory be grafted together into one big super program in userspace. It would have the same functionality as all of the individual Unix/Linux command-line programs put together (and maybe that would be desirable to some people). But from a Software Engineering "separation of concerns" AKA dependency reduction AKA modularity AKA "do one thing and do it right" AKA loose-coupling perspective -- doing that might not be so desirable! And yet, with the complexity brought about by 'init' descendants -- it seems like we're going down that exact route! Which leads us full circle (because history always repeats itself!) -- back to the reason why Unix was created -- because of the complexity and problems brought about by the complexity of its predecessor, Multics! https://en.wikipedia.org/wiki/Multics https://en.wikipedia.org/wiki/Multics Point is -- 'init' in whatever form it takes -- is by no means obligated to do anything -- although if it is to do nothing, then it should at least launch one other program which will do something! If that's the case, then why not put that under user control? But wait, if we're doing that, why not make it launch multiple other programs! OK, now we need a file to tell init where that should be! But what if the filesystem for that file is not mounted? Anyway, you see how the "rabbit hole" of problems (and increasing LOC complexity) forms! Related: https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...
t43562 3y ago
Linux isn't what it was - it's contributed to via a lot of companies with commercial interests. The open source developers who started doing it all for love are often doing it for money now. Certain features are desired, there's a way to do it, they do it. The actual "customers" of linux now - the ones providing the money that employs people - aren't Linux enthusiasts but big companies. So it's absolutely obvious that their needs are "what matter". It's ok. The rest of us can fork and "be irrelevant" in the same way Linux was irrelevant before it became popular.

167 points by awilfox 3y ago

Before this topic boils down to: SystemD gave me cancer/SystemD cured my aids as it always seems to. Please permit me get my opinion across (as it is more nuanced) before I continue. I sincerely believe that systemd solved a problem that nobody was willing to solve, and it has every right to solve those problems any way it wants (when you ask for help you don't get to choose how you are helped after all) - my concerns boil down to the fact that the adoption it has seen has lead indirectly (or, directly) into a monoculture; and a monoculture that almost certainly will stifle innovation since a replacement will need to be bug-for-bug compatible. Prior init's were actually quite easy to replace and alternatives were often used, but these days most software assumes systemd and this situation gets worse every year. That said; and with the knowledge that while I am afraid of systemd as monoculture (and a relatively opaque one); this line "systemd, as a service manager, is not actually a bad piece of software by itself. The fact it can act as both a service manager and an inetd(8) replacement is really cool." Is something I vehemently disagree with. For starters, (x)inetd is an anti-pattern, everything I understand about systems development indicates we should be seperating concerns as much as possible, having one super-server that launches everything under one daemon is directly opposed to this. "But", I hear you thinking already: "systemd runs services as independent users, it solves that!", and I would agree with you, except now pid 0 is listening to the network instead.. That doesn't strike me as much better. If there's a bug/backdoor in your binary distributed version of systemd then you are SOL and your whole system is owned as root, but at least a bug in your application might not expose your entire inetd user. :\ It's also a common issue that inetd's architecture can lend itself to getting DoS'd harder than other more-standard daemons, except now it's your pid 0 being DoS'd; not sure how you recover from that honestly. EDIT: if you are going to downvote, please provide reasons. Sick of this holy war, lets just end it reasonably please.

awilfox 3y ago
The monoculture problem is one that I really tried to cover in the article, indeed. While I do agree that a "super server" listening to the network could be a new (or old, with (x)inetd) concern, a well-written and well-designed pid1 doing this isn't much more risky than having the services manage themselves. The listener could/should be just as unprivileged as the target daemon, nominally using the same uid/groups and root directory as the target as well. And indeed, systemd's .socket files support the same environment control variables (User=, Group=, SupplementaryGroups=, RootDirectory=, WorkingDirectory=, et al) that services do. So that boils down to "are people writing socket activation units correctly", which is probably "no", but could be "yes".
- znpy 3y ago
  > The monoculture problem is one that I really tried to cover in the article, indeed. The issue is that the previous "multi-culture" really sucked real bad. It was a shit show, really. systemd is rising the bar by A LOT in terms of systems management, and most alternatives are simply not keeping up. But can you really blame it on systemd making a stellar job on its own? If anything, we could blame it on the "alternative projects" doing a fairly poor job and delivering very little. edit: systemd is so good that FreeBSD people have already started pondering if they should build something similar for themselves: https://www.youtube.com/watch?v=o_AIw9bGogo https://www.youtube.com/watch?v=o_AIw9bGogo
  - dijit 3y ago
    Fuuuuuck I hate that talk by Benno. Mostly it strawmans the opposition, says its because they dont like change primarily. Then he indicates that macos’s launchd is an improvement, despite it being one of the most painful parts of advanced macos administration. Its a talk that comes up often in these threads but it completely betrays the idea of having a fruitful dialog about the pros and cons of this design and the situations where it can be beneficial.
    - acdha 3y ago
      > Then he indicates that macos’s launchd is an improvement, despite it being one of the most painful parts of advanced macos administration. This seems like a rather hot take. Most of the Mac admins I knew when I worked in the field were quite happy to have a single standard way to solve that class of problems.
sevagh 3y ago
>"systemd runs services as independent users, it solves that!" It's in a terrible way, too. Not sure it improved but I recall somebody once, at a company I worked at, trying to introduce per-unix-user systemd services (e.g. `graphite` user for running Graphite), and there were some atrocious steps required like `loginctl --enable-linger` and God knows what. We moved to running everything systemd as root, it's easier (and you can specify which unixuser the actual systemd unit runs at, which is "close enough").
- Nullabillity 3y ago
  Loginctl/per-user systemds are for managing interactive sessions. System services with User= is exactly the way you'd go for isolating background daemons.
  - ptx 3y ago
    What if you want to let a non-root user manage the systemd unit for a particular service? Is there a better way than per-user systemd instances?
    - chuckadams 3y ago
      chgrp, setfacl, sudo.conf ... lots of choices.
    - acdha 3y ago
      Check example 3.4 here for a polkit policy allowing an arbitrary user to restart a single unit: https://wiki.archlinux.org/title/Polkit https://wiki.archlinux.org/title/Polkit
      - twic 3y ago
        Does this let them modify the unit file, create units, create timers, etc? User systemd allows that kind of complete self-service, and so lets you do application deployment and management without touching the root account, which is rather nice.
ptx 3y ago
Why is inetd an anti-pattern? Isn't separating all the work of opening ports and launching daemons into a separate component (so that it's not mixed into every other component) exactly what separation of concerns is all about?
- pi-e-sigma 3y ago
  Inetd approach is just fine. Don't listen to to the worshippers of systemd complexity. It's like listening again the guys who insisted that JEE is the future and you are stupid because you prefer simple solutions to simple problems.
Karellen 3y ago
> my concerns boil down to the fact that the adoption it has seen has lead indirectly (or, directly) into a monoculture; Were there multiple implementations of sysv-init being used by different distros on Linux before systemd came along?
- dijit 3y ago
  yes, at least 5. sysvinit, s6, openrc, runit and Solaris SMF
  - Karellen 3y ago
    Were they alternate implementations of sysv-init? Or did they do their own thing, and also happen to run sysv-init scripts for back-compat? Because systemd also runs sysv-init scripts for back-compat. Edit: Also, were any distros actually shipping those as supported init systems? I was under the impression that most of them were still in the "experimental" stage and not viable replacements (yet).
    - liveoneggs 3y ago
      OpenRC and Solaris SMF definitely did their own thing. SystemV-init and the "run levels" concept was flawed from the very start when it replaced the previous BSD-style "single user" vs "multi user" boot up. Linux only adopted SystemV style init because it was the norm in Solaris at the time. It is not a linux-ism. As this article notes - systemd is actually fine as an init system and hardly anyone denies it. It's all of the other stuff (journald, resolved, timers, etc) + tight coupling + environmental assumptions that is the problem.
    - awilfox 3y ago
      I'm unaware of any Linux distribution using SMF. If there was, I would have been running it. I didn't really like pfexec (it felt bolted on and not quite there yet), and I really really hated ZFS (and still do), but SMF made Solaris administration a joy. I don't think any distribution was using runit before systemd, but it was available in Gentoo as a sysvinit replacement and ISTR it was used in the Rails community for supervising Unicorn.
rst 3y ago
"Nobody was willing to solve" is a bit of a stretch when upstart got shipped at pretty much the same time. (Sponsored by the commercial home of a particular Linux distribution -- but so was systemd.)
- acdha 3y ago
  I used Upstart a lot but it did less and had various unfixed bugs whose failure mode was becoming unmanageable. I give Ubuntu credit for starting it but was reminded of jwz’s CADT rant periodically until we upgraded to a release which used systemd instead.
Nullabillity 3y ago
> Prior init's were actually quite easy to replace and alternatives were often used, but these days most software assumes systemd and this situation gets worse every year. Not really. Sysvinit scripts are... scripts full of sysvinitisms (double forking and PID files, anyone?). Idiomatic systemd daemons are, comparatively speaking, very straightforward. Of course, there are a lot of nice-to-have features, but to get a running system, you should largely be able to get away with parsing Wants=, After=, and ExecStart=.
- einpoklum 3y ago
  > Not really. Sysvinit scripts are... scripts full of sysvinitisms (double forking and PID files, anyone?). ... well, a bunch of scripts are something that's relatively easy to replace. It's not as though other system components have reliance of these scripts and their sysvinit'isms baked in. From what I've heard (though not verified) - non-systemd distributions seem to manage to work with upstart, or openrc, instead of sysvinit, without much hassle.
  - dijit 3y ago
    Previous inits were glorified loop { fork() } programs. There were many scripts, many implementations, but interestingly most were compatible with each other. The largest issue inits had was that the flexibility they provided gave distro maintainers a lot of choice in how those scripts should operate to be consistent with the rest of the system. Things like log locations. The issue to be solved is: how to determine parallelism in boot, how to supervise a process with minimum complexity, and how do we do structured logging. Interestingly: SMF solved all of these a decade before.
    - einpoklum 3y ago
      > Previous inits were glorified loop { fork() } programs. Maybe (I'm not an expert on old init systems), but current inits aren't that. > The issue to be solved is: how to determine parallelism in boot I think you're replying to another comment of mine. At any rate, there were and are alternative solutions rising to meet this challenge - which involve nothing like the behemoth which is systemd. Examples: OpenRC, runit, GNU Shepherd (sort of). https://en.wikipedia.org/wiki/OpenRC https://en.wikipedia.org/wiki/OpenRC
einpoklum 3y ago
> I sincerely believe that systemd solved a problem that nobody was willing to solve What problem was there which nobody was willing to solve? Also, "solving" a problem by creating a mechanism that in itself highly problematic in other ways does not necessarily count as a solution; it is a shifting-around of problems. ----- The systemd "holy war", such as it is (mostly complaints on forums and in blog posts and in personal chats; don't remember any violence in this war) - is due to four reasons: * The significant problems which systemd introduces. * The way systemd has been developed and managed as project (including some grievances with individuals). * The fact that distributions have not only adopted systemd, but made it effectively impossible to opt out of. * Faults with the process in which systemd was adopted as a required default by all of the main distributions - bypassing wide opposition without addressing its criticism. So, the adoption of systemd exposed technical-governance/power-dynamics problems in the Linux distro world, with systemd serving as the symbol for those.
- awilfox 3y ago
  I like the off-by-one there, and choose to interpret it as intentional. The fourth item is what I find to be the most interesting. I really hadn't thought about it that way before. I remember the discussion in Fedora mailing lists around systemd and iirc the criticisms were mostly actually answered and handled. But in Debian it really wasn't and it fractured the community a lot. It is perhaps the case that the "anti-systemd" crowd wouldn't be "anti-systemd" if they felt like they had been heard. Perhaps that's the bigger lesson we should all be learning: to listen closely and respond respectfully.
  - einpoklum 3y ago
    > I like the off-by-one there, and choose to interpret it as intentional. Amongst our weaponry are such elements as... I'll come in again. https://www.goodreads.com/quotes/1495-nobody-expects-the-spanish-inquisition-our-chief-weapon-is-surprise https://www.goodreads.com/quotes/1495-nobody-expects-the-spa... > It is perhaps the case that the "anti-systemd" crowd wouldn't be "anti-systemd" if they felt like they had been heard. I would say it is the other way around. systemd would not have been adopted as the default init system, had the anti-systemd crowed not been ignored. But that's not even the main point. The grievance is not about how some individuals did not "hear". It is about the _possibility_ of such a decision being taken with that level of technical-community resistance; i.e. the expectation is that decent process would not depend only on the benevolence of those in charge. That's why it's a structural rather than a personal failure IMNSHO. And again, there's the conflation, or bundling, of the multiple decisions: 1. Offer systemd in the distribution 2. Make systemd the default option for the distribution 3. Necessitate installation & use of systemd with the distribution I am specifically pretty certain that Devuan would never have been forked if systemd were merely a configurable installation option in Debian.
teddyh 3y ago
> now pid 0 is listening to the network Pid 1, surely.
acdha 3y ago
> For starters, (x)inetd is an anti-pattern, everything I understand about systems development indicates we should be seperating concerns as much as possible, having one super-server that launches everything under one daemon is directly opposed to this. It means that this server had to be carefully secured but the benefit is that you have exactly one bit of heavily-audited code listening to the network, logging activity & problems, starting processes, changing users / dropping privileges, setting up namespaces, etc. I’ve seen a lot of code get various combinations of those wrong so I think that’s a far more nuanced problem than in your portrayal. Those are all things people should know how to do but I’ve seen Java or PHP running as root in production because someone couldn’t figure out how to drop privileges needed only at startup enough times to appreciate the benefits from systemd making it so much easier to do things right.

INTPenis 3y ago

Nice article, after years of emotional flaming finally something fair and balanced about systemd. The point about there existing no competition and no respect for glibc alternatives is very valid and to that I can only say; if you build it, they will come. Systemd had the advantage of Red Hat backing so they got a great headstart. But in general people will use whatever works. So roll your sleeves up and get cracking.

nerdponx 3y ago
I always appreciated this presentation for being a reasonable and balanced take, if you're interested in other examples: https://youtu.be/o_AIw9bGogo https://youtu.be/o_AIw9bGogo
plagiarist 3y ago
If someone built the vision in this article I would use it. I love declarative config. The article is such a good take it makes me want to try out the author's Adélie project.
throwawaaarrgh 3y ago
People will use what works. But there is a limit. If you have two things, one of which does everything under the sun, and the other only does mostly everything, people tend to choose the first. The more people do this, the more "weight" the first carries. Until one day, it makes no sense at all to use anything but the first. Even if it causes you tons of headaches. The more different the two are, the less and less likely that you could or would ever switch. It's too big not to use. And so, building an alternative becomes a fool's errand. Nobody would use it, because there's just too many reasons why they shouldn't or can't. (In case you don't believe me: it is now impossible to run a desktop without a half dozen Systemd shims, because the entire ecosystem is now hard coded to use Systemd. I know because my desktop is Alpine Linux, a musl distro)
- traverseda 3y ago
  No kde?

ofrzeta 3y ago

I've found systemd-nspawn to be a great alternative to the olde chroot method for fixing systems in a rescue shell. systemd-nspawn -D /mnt mounts the system like chroot but the real value comes with systemd-nspawn -b -D /mnt/ that actually boots the system in a container.

q2dg 3y ago
I use systemd-nspawn as a built-in alternative to LXC/LXD.
awilfox 3y ago
Thanks! I hadn't considered that use case.

habitue 3y ago

Maybe a good way to get competition is to give someone a possibly dubious (but fun!) goal of writing a systemd compatible alternative in rust. Reimplementing such a massive piece of infrastructure is pretty daunting. I don't think many people who are not being paid to would embark on it and then keep up the momentum to actually cover all those edge cases and maintain a community around it. Rewriting things in rust seems to be pretty motivating though, so maybe it could be a force for good here

nerdponx 3y ago
If you're just looking for init and service management, the S6 system (and its wrapper 66suite) are pretty good. It's the only alternative that seems to match Systemd for feeling like a modern tool in my opinion, rather than a relic of the old days.
- generalizations 3y ago
  Been watching that for a couple years now. At one point there were plans to integrate with alpine Linux, but it wasn’t ready yet. Has there been progress towards that?
  - awilfox 3y ago
    Yes, there is some progress it seems: https://skarnet.com/projects/service-manager.html https://skarnet.com/projects/service-manager.html
- throwaway7356 3y ago
  Don't want s6 as it brings with it a binary configuration database (Windows registry). systemd at least has text files for config.
  - egberts1 3y ago
    My sole resistance to s6, binary config AND default-all privilege for spawned daemon. Better to stick with INI v1.4 format. https://cloanto.com/specs/ini/#escapesequences https://cloanto.com/specs/ini/#escapesequences

sevagh 3y ago

Systemd is Godsent for people who have to administer diverse Linux boxes. I can see how people who want to run their own tight ship on their machine would balk at it. I also sort of hate the systemd-resolv and the fact that it instantly rendered worthless 99% of online guides of how to unfuck your local DNS resolution.

forinti 3y ago
I had no opinion on systemd until I found out I could configure the mount points in fstab to magically mount themselves whenever they were needed. I have a lot of network shares and some of them caused real trouble when they failed for whatever reason. Now I really appreciate it.
- pi-e-sigma 3y ago
  Various automount tools have been available since the 90s.
  - mariusor 3y ago
    Indeed, why use Dropbox, when FTP existed for decades?
    - pi-e-sigma 3y ago
      Except these automount tools provided exactly the same functionality that is currently provided by systemd. And did it better
      - bravetraveler 3y ago
        I contest 'better'. What's the difference? It's mounted... no? Allowing 'systemd' to be aware of mounts lets you create dependencies between your mounts and services. For example: 'Requires=mnt-myfs.mount' (and 'After'). Use 'PartOf' if you want the service to get restarted during an unmount/re-mount. This integration is mint! Anything not systemd [when it's already managing your services] trying to orchestrate is, by nature, second-rate One may note the removal of the fstab generator. It's overblown. Any distribution including systemd vendors it.

t43562 3y ago

I'm enjoying artix linux with dinit. I tried a couple of other alternatives - s6 and openrc - although I wouldn't say I was that fair on openrc. dinit may lack many things - I don't know - I haven't had any reason to be unhappy with it and it seems to work in a fairly understandable way. I defined a new service (for minidlnad) and that appears to be straightforward.

Philpax 3y ago

This is quite a balanced take on the matter. Appreciated!

macNchz 3y ago

I was a big skeptic earlier on, but I have come to appreciate systemd in recent years. That said, the section on resolved is a big one–it has been one of the only things that consistently causes me annoyance over the three years since I started primarily using Linux on the desktop. After one particularly deep rabbit hole where I never actually solved a problem where local network mDNS lookups always took exactly 6 seconds, I wound up disabling it entirely in favor of unbound, which works great but revealed that a handful of other things seem to expect resolved to be present and won’t work properly without it.

kemotep 3y ago

> I prefer my init scripts to handcrafted by local artisans. Each time the computer boots up is an objet d’art. In all seriousness this is a wonderful article. Has the author/op seen this talk[0] by a FreeBSD developer on systemd? What I like about the talk is asking about what is going to happen when we need to move on to newer ways of doing things that will have advantages and disadvantages over the old way. We can see a similar story with X11 and Wayland. Systemd does do some things better than the old way but does have flaws. [0]:https://youtu.be/o_AIw9bGogo https://youtu.be/o_AIw9bGogo

chasil 3y ago

I use much functionality in systemd that was not present in SysV init, and I really appreciate it. It has never crashed any OS that I have run. However, there are a few aspects of it that are inconvenient. Automount units use an unintuitive naming scheme, and you are not free to name them as you wish (as you might for a socket unit). If you are mounting an NFS volume immediately below the root directory, you don't see the problem, but if the mount is several directories deep and/or uses ASCII symbols (non-alphanumeric), it is not pretty. Socket units require two files per port. When I am moving complex inetd.conf setups to Linux, it's far easier to implement them with busybox inetd than convert dozens/hundreds of services to unit files, despite the increased functionality. Somebody has probably written some scripting to do this. I am not aware of any include directive for my own directories, so I don't have to place everything in /etc/systemd/system. There probably is a way to do this, and I am betraying my ignorance. And my, things can get messy in a hurry in /etc/systemd/system. I don't know how to configure users to be able to maintain their own (personal) units. And lastly, it's so seductive that I have no idea how to do many things in other operating systems that I easily do in Linux. I wish this itself was not a walled garden (but I'm not leaving).

assbuttbuttass 3y ago
> I don't know how to configure users to be able to maintain their own (personal) units. This is something I use all the time! Just put unit files in ~/.config/systemd/user and use systemctl --user to start/enable them https://wiki.archlinux.org/title/systemd/User https://wiki.archlinux.org/title/systemd/User
- isignal 3y ago
  Just a note that this requires the admin to enable linger if the units need to be started before the user logs in and run after they logout.
frankjr 3y ago
> I am not aware of any include directive for my own directories, so I don't have to place everything in /etc/systemd/system. There probably is a way to do this, and I am betraying my ignorance. Systemd by default looks at a bunch of directories depending on the context. $ systemd-analyze unit-paths (or --user / --global) https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#Unit%20File%20Load%20Path https://www.freedesktop.org/software/systemd/man/latest/syst...

nijave 3y ago

More structured metadata in journald is nice and I don't have any qualms with binary logging formats, but the author does have a point on log shipping. Additionally, reading journal files has always seemed painfully slow to me (30 seconds of 100% CPU usage) I'm not overly found of timer units, either. They seem feature rich but much more complicated to setup than a single line in a crontab. On the other hand, real dependency management between daemons, mounts, and sockets is a huge win. I can't quite remember the right terminology but afaik systemd-resolvd supports routing different domains to different DNS servers on different network interfaces which can be auto configured via DHCP. The practical implication being, you can connect to a split tunnel VPN and domains accessed over the VPN get routed to the VPN DNS server.

akritid 3y ago
It’s called split DNS. Can be implemented using dnsmasq
- nijave 3y ago
  It can be implemented with dnsmasq but dnsmasq doesn't integrate with DHCP (edit, client) out of the box and requires a restart to apply configuration updates which invalidates query cache. Edit- I think split dns more commonly refers to serving different internal/private and external/public zones. In this case, the DNS servers may return the same results but if I'm connecting to a VPN running some distance away, I don't want to route all DNS requests to that server--only the ones for domains on that network. The Domains directive allows configuring this https://man.archlinux.org/man/systemd.network.5#%5BNETWORK%5D_SECTION_OPTIONS https://man.archlinux.org/man/systemd.network.5#%5BNETWORK%5... You can add a dhclient hook to reconfigure dnsmasq with `server=` directives to achieve the same thing, but, as mentioned, that still requires restarting the dnsmasq daemon. That usually manifests as: the VPN connection times out, the hook runs, dnsmasq is restarted, but a random web page or request will fail while the daemon is restarting. Sure, you could do SO_REUSEADDR or something to prevent the disruption, but, once again, this all just works out of the box with systemd-*
  - 3y ago

greyw 3y ago

Reading the section about resolved reminds me of musl having a "broken" stub resolver implementation for years (whenever the dns payload was larger than 512 bytes). "Broken" because the TCP fallback was supposedly intentionally not added for complexity reasons. DNS and all the additions seem to be really not easy to handle.

acdha 3y ago
Yes - that lower level networking stuff is just a minefield of edge cases which most people never need but are really annoying if you’re unlucky enough to hit them, and Linux has always had this problem where if something works for Windows the people who run those broken networks and many users will say it’s a Linux problem. A lot of the systemd decisions are probably explained by someone getting a ticket saying their Red Hat laptop takes much longer than Mac to get on the corporate network or was unable to deal with the captive portal on some hotel Wi-Fi, and they realized fixing it was going to require work across 5 different code bases.
- 20after4 3y ago
  I think this is the most likely explanation for why systemd includes everything but the kitchen sink. For consistency, and ease of maintenance, having it all under one umbrella is certainly a benefit (for the developers of systemd.) Also, NIH is a hell of a drug.
  - deleted 3y ago
    [deleted]
  - yencabulator 3y ago
    Systemd's catalogue of kitchen sinks is getting to be truly impressive. Here's an NVMe-over-TCP server, for some reason: https://github.com/systemd/systemd/blob/main/src/storagetm/storagetm.c https://github.com/systemd/systemd/blob/main/src/storagetm/s...
    - znpy 3y ago

frankjr 3y ago

I generally embrace systemd and have been pretty happy with it but there's one component which simply doesn't work correctly and that's systemd-resolved in combination with DNSSEC. I eventually had to replace it with Knot Resolver which works flawlessly on the same machine / network. https://github.com/systemd/systemd/issues/9867 https://github.com/systemd/systemd/issues/9867 https://www.knot-resolver.cz/ https://www.knot-resolver.cz/

remram 3y ago

I think the problem of systemd is that it is complex. It is fine when it works correctly, but when you eventually have to dig into any sort of problem, it is hell. For example, all of my boxes spew many logs per minute about "Failed to set up mount unit". This is apparently a bug with the generated name of some internal unit related to mountpoints (too long). This one is not as bad as others, because it does have a bug reported for it, but is still something I'll have to deal with until I upgrade the distro on all the servers. Many similar bugs I can't track down at all. Systemd makes a lot of things easy, but not simple, and that is a big problem in practice.

twic 3y ago
This is it. For example, systemd (a) allows you to run services as a normal user, not as root, (b) treats child processes of a service process as part of that service, and (c) collects logs from services. Except if you try to do all three, that just doesn't work, because they've half-arsed the log collection process: https://serverfault.com/a/1143851/92104 https://serverfault.com/a/1143851/92104 There's no reason for this not to work! Taking a very straightforward and traditional approach to running processes (as in my demo code in that question), you don't have this problem. But rather than that, they did something clever which doesn't work properly.
- thwarted 3y ago
  I can't make heads or tails of either of those answers, and they seem to contradict each other. The output capturing is all to the same file descriptors, this is the way resources are inherited on fork (and there's no redirection here). That it works with unbuffer is weird because the subshell and subprocess both end up flushing their output when they exit, so there should be no need to flush before exiting. If the first answer is correct, then the unbuffer works because the process introspection as the source of the output is unified to be the unbuffer process. But why bother introspecting the exact process that originated a write to the output? Did we have a problem with file descriptors being highjacked? And even if we did, losing the output (or not associating it with the parent process) is worse.

ivolimmen 3y ago

I like the honesty of the article. I never got the grunge others have against it. I am a software engineer and work in office automation (a.k.a. the boring stuff). On Linux I am a poweruser. If Ubuntu used initd I use it. They switched to systemd so I use that. I liked it. Making an application start on reboot was easy to do. And easier than with the rc files 20 years back.

traverseda 3y ago

I originally wrote this comment for reddit, but I feel like hackernews might appreciate it. If you think it's too long or detracting from the article or something feel free to downvote, I won't be offended. Here are some of my criticisms, although I do still use systemd daily on my personal devices and servers. * Bad security Systemd is architected in a way that has a lot of code running as root, it's also written in a language that isn't memory safe. This means it has a large attack surface (a lot of code you need to make sure is bug free to be secure) and it's harder to make sure your code doesn't have really severe security related bugs (The [NSA recommends](https://www.nsa.gov/Press-Room/Press-Releases-Statements/Press-Release-View/Article/3608324/us-and-international-partners-issue-recommendations-to-secure-software-products/ https://www.nsa.gov/Press-Room/Press-Releases-Statements/Pre...) using memory safe languages for critical stuff like this). There are certainly other critical projects (like this linux kernel) that are similarly important and written in memory unsafe languages, but systemd has had some [pretty critical vulnerabilities](https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=systemd https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=systemd) discovered that do not inspire confidence in their ability to use these kinds of dangerous languages safely, and they don't have nearly the same budget as the linux kernel for detecting and preventing these kinds of issues. This is less of a problem if you're a large enterprise customer running up to date SELinux, but it should still have been possible to write systemd in a way that limited the pid1 attack surface while retaining all current functionality. * Journalctl is a pain for desktop users and smaller teams Journalctl is how systemd manages logs. By default it saves logs in a binary format with additional metadata. This makes it easier for large teams to ingest the logs into a centralized log-collection daemon, like the ones offered by redhat for enterprise deployments, but it breaks a lot of workflows that older sysadmins probably used. Things like just rsync-ing a bunch of logs to one place, or using tools like grep and find to inspect logs. Systemd does of course provide replacements for those tools, instead of using grep you can use journalctl to search through your logs, but you could use grep to search through any text file. Config files, source code, or logs. Now I need to memorize all the flags for one more tool, and change a bunch of stuff about how I collect logs. This also presents a challenge for people doing embedded work, as you can't just grab the SD card out of a system can look at it's logs. You need a working journalctl CLI on your host machine. They've add in a --directory flag, but in the past this was much harder, requiring you to actually chroot into the embedded system (that may be broken in strange ways) in order to read logs. Journalctl has advantages, but they're mostly enjoyed by large enterprises. Yes I know it's actually systemd-journal or just "journal" or whatever they call it. Journalctl is the command most users will be familiar with though. * Poor [Locality of behavior](https://htmx.org/essays/locality-of-behaviour/ https://htmx.org/essays/locality-of-behaviour/) makes it harder to reason about When you're trying to understand how a system works it's nice to be able to see everything in one places. There are like 9 different places a systemd unit file can live, you can apply an over-ride to a unit file, unit files can depend on other files like socket files. This is good for large teams as it makes it easier for specific groups in a company to claim ownership over parts of the system, but it means as a desktop user or sysadmin working with a small business you've added a lot of complexity. You can't just type `ls /etc/init.d` to get a rough overview of what services exist, you need to memorize more systemd-specific commands. If you want to edit a service you can't just edit a service, you need to create an over-ride using another systemd specific command, make sure you have the EDITOR environment variable set up, and then open the original service in another editor so you can compare the two. It creates some more work and complexity and encourages you to use a bunch of systemd-specific tools (and presumably get red-hat certified training). * People use systemd stuff before it's ready I'm not sure this is something I can blame systemd or redhat for, but the official stance of redhat is that systemd-resolved is not ready for production, and yet it's used all over the place. That can give people a poor impression of systemd after the 9th time they try to do something even slightly different with their networking setups and systemd-resolvd breaks, not to mention the numerous security issues in systemd-resolvd. * Unix philosophy A lot of people say that either unix philosophy doesn't matter, or that systemd does embrace unix philosophy. That stuff about logging and systemd-specific tools I mentioned above? That's what people actually mean when they talk about unix philosophy, they mean being able to grep through their logs and rsync logs to a remote server. They mean using standard text files and not needing to have a special command that wraps your text editor to "properly" edit a unit file. * Doesn't run in a chroot As someone who splits my time between embedded linux and server linux this is just a personal pet peeve of mine. It makes it very hard to debug embedded systems that use systemd if you're not using systemd on your workstation. It does feel like I'm being forced to use systemd sometimes, and while I've largely gotten over it I'm still a bit bitter. It's also made some small personal projects, like getting a full linux distro running on a KoBo e-reader, much much more difficult than they had to be. It's a mess under docker, and why I need to use alternative OCI-runtimes like nestybox to do a bunch of testing for embedded systems. Thankfully I wasn't an early adopter to docker and didn't have those problems until there were already mature solutions. But there's really no reason why it should have to run as pid1, other than them wanting you to use docker-alternatives that are deeply integrated with systemd, like their podman tool or systemd-nspawn. This was just such a blatant attempt to abuse their near-monopoly position that it bears some extra whining. * OpenRc does everything systemd does, but better Unfortunately other red-hat influenced projects like Gnome won't support or test on non-systemd init systems, let alone providing default services files for them, meaning that any distro that wants to be compatible with gnome will need to do a bunch of extra work to write and test service files. For one project that's potentially reasonable, and certainly there are distros that do that extra work, but for projects like Arch who have the explicit goal of sticking as close to upstream sources as possible it makes it more or less impossible. Systemd survives not because it's a good solution to the problem, but because it has a large corporate backer, is widely deployed, and is a safe thing to code against. There's a very old IT saying, "No one ever got fired for buying IBM". If you pick a safe industry-standard options no one can blame you if it goes wrong, even if it's the technologically inferior option. As much as I'm a systemd-hater I still do use it on my personal devices and servers, because it's by far the path of least resistance. I hope we see a similar situation like with pulseaudio and pipewire, where the pulseaudio rewrite was much much nicer than the original. I don't think that's going to happen until systemd slows down though, right now if you tried to re-implement systemd I suspect you'd get the rug pulled out from under you as they changed standards and behaviors (I've seriously thought about doing it myself, at least for relatively simple unit files). I'd still prefer to be using OpenRc, as I don't know how a rewrite would deal with the locality-of-behavior issues, but systemd has been getting better and more reliable over time.

deleted 3y ago

[deleted]

gavinhoward 3y ago

I think the author is right on just about everything. But the author is especially right on the need for competition. So I am working on it. I am building a build system, and I am making it usable as a library. When I have made it so, I will implement an near drop-in [1] replacement for systemd. You'll be able to use systemd unit files, and there will be an option to have binaries with the same name (off by default to not interfere, though) so that users don't have to learn new stuff right away. [1]: I am not going to implement journald for example; logs will be text, even if they are structured internally. Besides, implementing things to be perfectly compatible would only strengthen the monoculture, not weaken it. The plan is to be compatible on the things that matter for distro integration, so that distros can ship both with little work, but have other things different, so that users can choose what works best for them between the two.

traverseda 3y ago
Have you considered integrating this with OpenRC?
- gavinhoward 3y ago
  I am not sure what you mean.
twic 3y ago
In a similar vein, but already existing, nosh lets you use systemd unit files and commands via a shim layer: https://jdebp.uk/Softwares/nosh/ https://jdebp.uk/Softwares/nosh/
- throwaway7356 3y ago
  Does anyone use it? It there a distribution (BSD or Linux) using nosh?
vetinari 3y ago
> I think the author is right on just about everything. The author is very wrong on the systemd-resolved bit. Not understanding the ability to have per-interface specific zones is ok; it is a thing for desktops with multiple interfaces that come and go (like VPNs, for example). There is no other resolver on Linux capable of doing it and integrating with NetworkManager. You can kinda-sorta make dnsmasq do it, but with some limitations. But the crown is taken by the "mDNS is better taken care of by a dedicated package like Avahi." Uh, oh. It is like saying, that you don't need Chrome, it's duties are better taken care by nginx. Similarly, you can use Avahi for _advertising_ mDNS services, but not for _resolving_. Which, as an user, you probably are interested in.

smitty1e 3y ago

The Famous Article (TFA) is an exemplar of dispassionate substantive, liberal criticism. The shortcomings of systemd seem likely the overall shortcomings of open source: none of the $PROJECT maintainers experience $PAIN_POINT, so it is simply not a priority for $PROJECT. Somehow this seems a variation on the tragedy of the commons => https://en.m.wikipedia.org/wiki/Tragedy_of_the_commons https://en.m.wikipedia.org/wiki/Tragedy_of_the_commons Without some capitalist skin in the game, the $PAIN_POINTS become difficult to prioritize.

WesolyKubeczek 3y ago

I would in fact love a portable user-level process manager that can speak systemd unit files and is portable. Something like supervisord, but you use systemd unit files, and it tries to do as much as it can within limitations (process tracking sure can be wonky, no dbus everywhere, no cgroups everywhere, no absolute freedom in resource limitations if you’re not PID 1).

bmicraft 3y ago
Process tracking without cgroups seems like an absolute nightmare w.r.t. race conditions

peter_d_sherman 3y ago

Most people (including most Unix greybeards!) really don't understand Unix's 'init' (AKA "the init daemon", "the init process", etc., etc.) -- much less any of its massively-increasing-in-LOC (and complexity!) successor programs... So we need to start with 'init'. 'init' -- even in its absolute first, simplest incarnation -- is still too complex to understand correctly! You see, we need to shift perspectives! We need to shift perspectives from a longtime System Administrator -- to that of a new barebone OS programmer. What is 'init'? Is 'init' a program that handles runlevels, starts and stops services, that mounts filesystems, that processes messages, that captures dead processes, that waits for hardware to become available, that logs and maintains informational/database/etc files, that starts audio, that starts X11, that stars the GUI, that acts as a proxy for sockets, or does anything else with the system? No! From the point of view of a new barebone OS programmer (as Dennis Richie and Ken Thompson were when they invented Unix and invented 'init') -- 'init' is NONE of these things! 'init' is only THE FIRST PROGRAM, THE FIRST COMPUTER CODE THAT RUNS IN USER SPACE. And that's it! That is all that 'init' ever is, or ever was! (User space, to recap, is the unprotected AKA "unprivileged" AKA "non-supervisor" memory running unprotected (AKA "user-land") code: https://en.wikipedia.org/wiki/User_space_and_kernel_space https://en.wikipedia.org/wiki/User_space_and_kernel_space) 'init' (and every single 'init' successor program, i.e., OpenRC, systemd, etc.) -- are the first program, the first set of computer code OUTSIDE OF KERNEL CODE (which has been running and is currently still running) to be run by the system. Now, what should that first program do? See, that's the magic question -- which gives rise to all that is to follow! In theory you could have an OS where the 'init' program, or its equivalent -- did absolutely nothing! But that wouldn't be very productive! If the 'init' program isn't itself a shell program (i.e., sh, bash, etc.) -- then (because there's no GUI at this point) the computer will not be able to accept typed command-line commands -- which is the first thing that you want a new OS to do! So now our 'init' expands in scope (and lines of code)! Our 'init' could be hardcoded to launch 'sh' or 'bash' (or whatever shell program exists) -- but what if the user wants to change that? OK, so now we need our first configuration file. Where to put that exactly? Oh, it's on a filesystem that hasn't been mounted yet? Well, maybe init should mount that filesystem! Point is, there's a set of problems (and sub-problems!) -- which give rise to increasing and increasing init's functionality over time! init, as the first user-space program for an OS to run, on whatever OS it is ran on, in whatever form it is in -- could simply be written to run and 'outsource' all of its functionality to other programs... But init (as it evolved into its very large LOC complex descendants) -- became a "dumping ground" -- for functionality that was inconvenient to go in other places and/or to be outsourced to other programs. See, all of the code in all userland Linux utilities -- could in theory be grafted together into one big super program in userspace. It would have the same functionality as all of the individual Unix/Linux command-line programs put together (and maybe that would be desirable to some people). But from a Software Engineering "separation of concerns" AKA dependency reduction AKA modularity AKA "do one thing and do it right" AKA loose-coupling perspective -- doing that might not be so desirable! And yet, with the complexity brought about by 'init' descendants -- it seems like we're going down that exact route! Which leads us full circle (because history always repeats itself!) -- back to the reason why Unix was created -- because of the complexity and problems brought about by the complexity of its predecessor, Multics! https://en.wikipedia.org/wiki/Multics https://en.wikipedia.org/wiki/Multics Point is -- 'init' in whatever form it takes -- is by no means obligated to do anything -- although if it is to do nothing, then it should at least launch one other program which will do something! If that's the case, then why not put that under user control? But wait, if we're doing that, why not make it launch multiple other programs! OK, now we need a file to tell init where that should be! But what if the filesystem for that file is not mounted? Anyway, you see how the "rabbit hole" of problems (and increasing LOC complexity) forms! Related: https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...

t43562 3y ago

Linux isn't what it was - it's contributed to via a lot of companies with commercial interests. The open source developers who started doing it all for love are often doing it for money now. Certain features are desired, there's a way to do it, they do it. The actual "customers" of linux now - the ones providing the money that employs people - aren't Linux enthusiasts but big companies. So it's absolutely obvious that their needs are "what matter". It's ok. The rest of us can fork and "be irrelevant" in the same way Linux was irrelevant before it became popular.