28 ms·

Systemd as tragedy

288 points by wyldfire 8y ago

ordu 8y ago
> "It's software" so of course it's buggy, he said. The notion that systemd has to be perfect, unlike any other system, raises the bar too high. systemd is a PID 1 program, it means it have to raise bar higher. When troubles begin, you would need tools to fix them, and if PID1 is crashed, you are out of luck. If system cannot boot into shell, you'd need to fix it from initrd shell. Or to boot other system, to fix this one. It sucks. Linux kernel chases very high standards of reliability, because when kernel panics it is even worse than PID1 crash. Init system should follow the same standards as linux.
- geofft 8y ago
  Have you ever had pid 1 (systemd or any other init) crash? For the last ~three years I've been paid to maintain high reliability algorithmic trading systems that ran systemd and a whole lot of other stuff, and systemd has never crashed on me. Lots of other stuff, including the kernel itself, has crashed. The bar is higher for pid 1 - if I were designing systemd I would have made a tiny pid 1 that just did message-passing to a more complex secondary process that could be restarted, or something, just to be safe - but I think systemd has empirically cleared the bar.
  - CBLT 8y ago
    I've never experienced an outright crash, but I've been bitten by [0] on some of my servers. [0] https://github.com/systemd/systemd/issues/719 https://github.com/systemd/systemd/issues/719
  - rachelbythebay 8y ago
    Yep. CentOS 6's upstart can be felled by generating a bunch of inotify events in /etc. http://rachelbythebay.com/w/2014/11/24/touch/ http://rachelbythebay.com/w/2014/11/24/touch/
  - rurban 8y ago
    The current debian testing version crashes with a NULL pointer segv in the kernel module. You need to downgrade to the previous version.
    - newnewpdro 8y ago
      In what kernel module? There is no "the kernel module" in a systemd context.
      - rurban 8y ago
        There is. udev loads kernel modules. See eg. http://www.linuxfromscratch.org/lfs/view/development/chapter07/udev.html http://www.linuxfromscratch.org/lfs/view/development/chapter...
        JdeBP 8y ago
        Waving in the direction of udev does not clarify what kernel module is supposedly the kernel module, which is what you were asked.
        geofft 8y ago
        'rurban is one of our resident trolls - see also https://news.ycombinator.com/item?id=13364173 https://news.ycombinator.com/item?id=13364173
  - ordu 8y ago
    > Have you ever had pid 1 (systemd or any other init) crash? No, I have not. But I have seen how systemd gracefully failed to boot system to login, with good looking colorful error message. Something that reminded me "Keyboard is not found. Press F1 to run setup."
    - geofft 8y ago
      Oh, to be clear I'm real mad about how systemd fails boot if (say) one of your filesystems is unavailable and makes you log in with a root password to fix it. But OP was asserting that systemd crashes under normal operation because its pid 1 is too fragile, which is very different. At scale I already expect that there's a chance a machine won't come back if I reboot it - it's annoying if I can't ssh in, but, well, I already lost a disk I care about and it won't return to service and I need to fix it anyway. (And it's an easy fix, just add "nofail" to fstab.) At scale I don't expect init to crash under normal operation.
  - ploxiln 8y ago
    I've had shutdown (for reboot) hang a few times after a systemd update, forcing me to cut the power. It's made me a bit paranoid, so I block the systemd package from having updates automatically installed, and every 6 months or so carefully manage update and reboot of each and every server ... EDITs: there's the classic case of the linux "debug" parameter: https://bugs.freedesktop.org/show_bug.cgi?id=76935 https://bugs.freedesktop.org/show_bug.cgi?id=76935 and the even more classic case of firmware loading events: https://lkml.org/lkml/2012/10/3/484 https://lkml.org/lkml/2012/10/3/484 and while "all software has bugs" systemd really has the most annoying bugs (by virtue of trying to do everything core to the system) and always insists that they are features and we are backwards whiny geeks for complaining.
  - 65a 8y ago
    There was that time it was bricking computers by erasing UEFI variables, but I'll allocate equal blame between systemd and UEFI
    - chris_wot 8y ago
      Actually, that was nothing to do with systemd. That was definitely a UEFI implementation issue. And systemd didn't delete anything, the user did - they ran: rm -rf --no-preserve-root / https://lwn.net/Articles/674940/ https://lwn.net/Articles/674940/
      - majewsky 8y ago
        The bug was in the kernel (it should not have allowed userspace to write arbitrary UEFI variables), but AFAIR it was exposed by systemd because it eagerly mounted the UEFI variable filesystem provided by the kernel into /sys/something/efivars.
        chris_wot 8y ago
        Indeed, but again that was a firmware issue. systemd didn't delete the variables. And systemd was setting EFI variables, so consequently it needed it to be mounted as read/write. The configuration files should have set that to read only after boot. The kernel patch where this was fixed can be found here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0389075ecfb6231818de9b0225d3a5a21a661171 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
    - voxadam 8y ago
      Personally, I lay the blame for this issue squarely at the feet of various UEFI implementations which fail to boot when the system's EFI variables are, for whatever reason, wiped clean. The UEFI spec explicitly states that clearing all of the variables on a system must not result in an unbootable system.
    - JdeBP 8y ago
      You shouldn't, because the maintainer of the kernel subsystem concerned told us all that systemd wasn't to blame for it. * https://news.ycombinator.com/item?id=15973577 https://news.ycombinator.com/item?id=15973577 * https://news.ycombinator.com/item?id=11152880 https://news.ycombinator.com/item?id=11152880
  - setquk 8y ago
    I’ve had Systemd completely stop responding before on numerous occasions on centos 7. As in can’t reboot or hangs rebooting or all commands hang. Only recourse has been to reboot the instance from AWS dashboard. I can’t get to the bottom of it because the tools don’t work when it’s down and there’s nothing there when it comes back up. I am not enjoying boiling to death in this pot of shit. And then there’s the situation where it just won’t boot. I just fire up a new instance then because it’s easier than debugging it.
  - dvfjsdhgfv 8y ago
    > Have you ever had pid 1 (systemd or any other init) crash? For the last ~three years I've been paid to maintain high reliability algorithmic trading systems that ran systemd and a whole lot of other stuff, and systemd has never crashed on me. You see, that's the argument I hear a lot from Systemd advocates. The problem with anecdotal evidence is obvious. When you hear people opposing Systemd, practically all of them have some real-life issues with it, often related to functionality that would otherwise be non-essential (i.e. doesn't really need to be handled by PID 1). Of course if you don't have a particular problem, you don't feel it's important. That's precisely the attitude people resent.
    - geofft 8y ago
      > When you hear people opposing Systemd, practically all of them have some real-life issues with it Yes, but a lot of people have real-life issues with it on their desktop of the form "It's too complicated." I'm asking specifically about real-life issues on production servers at scale. There will of course be tools that are poorly suited for a personal machine (even a personal server) but well suited for a team that wants to run a bunch of reliable servers. For instance I would never be happy running RHEL on my desktop, but that doesn't mean RHEL is useless.
      - dvfjsdhgfv 8y ago
        I can't quote any statistics but have the impression that a large part of non-Systemd crowd are old-time admins who maintain a large number of servers, myself included. When you break something on a desktop machine, that's easily fixable. When you need to deal with a large heterogeneous environment, you prefer to have things handled a bit more gracefully. Linus is a good example of a person who got this right.
  - nonbirithm 8y ago
    Very recently I had this issue[0] as the result of a systemd upgrade, requiring the use of a recovery disk to downgrade to the previous version as the keyboard input had failed to be initialized. [0] https://github.com/systemd/systemd/issues/11314 https://github.com/systemd/systemd/issues/11314
    - majewsky 8y ago
      If this bug hits you in staging, no big problem, just don't promote that particular update to production. If this bug hits you in production, your lack of a staging environment is the bigger concern IMO. (I have been hit by the same issue on my private notebook, but I have procedures in place to cleanly recover from failed upgrades on all systems, so it was not a big deal.)
      - yjftsjthsd-h 8y ago
        "Yeah the software completely broke, but that's fine because you should be able to deal with that" does not make me feel better about the software in question.
  - RX14 8y ago
    Using systemd automount and NFS you can easily get pid1 unresponsive, hung in uninteruptible sleep forever.
  - stewartm 8y ago
    Yep. 3AM, deep slumber, called out to look at a stricken server. Its problems included that systemd was frozen. Reluctantly I came to the conclusion that a restart was the only route forward. Cept, that is when you discover that the commands that have served you well for 2 decades don't work, as they are all wrappers for systemd, which has keeled over. To this day, the `shutdown` man page, which I was checking in, makes no mention of how to resolve, tho in fairness the other commands (poweroff, halt, init) do. I discovered this after stumbling across https://github.com/systemd/systemd/issues/3282 https://github.com/systemd/systemd/issues/3282 If you find yourself stuck in the middle of the night, reading through docs to try and figure out how recover a machine with a crashed systemd, then `systemctl reboot -ff` or equivalent is what you are now looking for, the `-ff` being the key to "JUST £&*(ing RESTART THE MACHINE!!!". Experiences like that, don't win you friends.
    - Twirrim 8y ago
      Did raising elephants not work (SysRq + R E I S U B)
      - auscompgeek 8y ago
        systemd disables the magic sysrq keys by default.
    - setquk 8y ago
      The worst thing about this is when stuff goes down, it does so at the least convenient time. Back in 2003 I was on a customer site who had a RH server and there was no internet connection available (as it was routed through the box) and my phone was a Treo 180G which had precisely fuck all useful internet on it. The company still exists and is in the middle of nowhere on the end of a shonky ADSL line and no mobile phone reception so the story hasn't improved. If this happened to me today with systemd I'd be up shit creek without a paddle.
  - JdeBP 8y ago
    No. Not mine. Not systemd. Not others. And I touched upon how rare this was in practice in my experience some years ago on Hacker News. * https://news.ycombinator.com/item?id=8384251 https://news.ycombinator.com/item?id=8384251 But it does happen to other people. * https://unix.stackexchange.com/questions/440229/ https://unix.stackexchange.com/questions/440229/ And there was one crash that made the headlines. * https://news.ycombinator.com/item?id=12600413 https://news.ycombinator.com/item?id=12600413
andrewstuart 8y ago
I really like systemd. It's got tons of really great functionality. systemd feels like it is well thought out and modern, like it's an integrated rethink and rebuild of lots of various software utilities that have evolved over many years. I'm investing all my learning in systemd where appropriate instead of things I used previously like cron and supervisor.
- CaliforniaKarl 8y ago
  I agree with that. But at the same time, systemd makes assumptions that I don’t think it should. Case in point: I deal with several clusters that use Stanford Central LDAP for account info, and our UID numbers have gotten pretty high. So much so that it overlaps the range systemd uses for dynamic service UIDs. The biggest annoyance is that the UID range is hard-coded, so I can’t provide an alternate range. More details: https://github.com/systemd/systemd/issues/9843 https://github.com/systemd/systemd/issues/9843 (but please don’t spam the GitHub issue!)
  - inferiorhuman 8y ago
    Do you remember when systemd would take usernames it couldn't parse (due to undocumented restrictions on valid usernames) in unit files and just translate them to uid 0?
    - CaliforniaKarl 8y ago
      It seems that was fixed in https://github.com/systemd/systemd/commit/bb28e68477a3a39796e4999a6cbc6ac6345a9159 https://github.com/systemd/systemd/commit/bb28e68477a3a39796...
  - tyingq 8y ago
    Wish my customers could file bugs and feature requests with that kind of thoughtful detail.
    - CaliforniaKarl 8y ago
darkpuma 8y ago
> "I'm not going to defend Lennart's approach to community interaction, but what I will say is you have to admire the willpower and determination of somebody who at work one day says 'I'm gonna write my own init system- actually no it's going to be a full on system layer for linux' [...] and then he gets it put into pretty much every major linux distribution." Yeah, I suppose I also admire the willpower and determination of Genghis Khan. Corbet is almost as dismissive as Lennart himself of the life-disruptive bugs real people encounter due to Lennart's approach to software engineering and project management. "It has bugs? It's software" This attitude is positively correlated with the amount of hate systemd (and pulseaudio) have gotten over the years.
- HarryHirsch 8y ago
  If planes fell out of the sky due to buggy software as often as systemd crashes due to yet another bug, air travel would get much hate, too.
  - freddie_mercury 8y ago
    If systemd cost as much as a plane, it would have fewer bugs. We can go back and forth with dumb comparisons all day but I don't really think we're advancing the conversation.
    - peterwwillis 8y ago
      I'm in the dumb comparisons camp, so here's mine: Systemd is communism, BSD init is capitalism.
    - HarryHirsch 8y ago
      I disagree about the value of the conversation. Airplane software is meticulously tested because of the safety implications. If we can't have proper testing for a widely used, sprawling piece of software that runs with root privileges and PID 1 we may be better off without that piece of software. Also this: the equally widespread SQLite seems to be programmed with a much better level of discipline and comes with much better testing. It isn't nearly as bug-ridden.
deleted 8y ago
[deleted]
shittyadmin 8y ago
This article is a bit of a joke "It's software so of course it's buggy" isn't a great argument when you're replacing something that didn't suffer the same issues. I just count my blessings that runit is widely packaged in every major distro because it can just happily sit on top of sysvinit, systemd, upstart, pretty much any init system and does things in a very simple shell script style, I really wasn't a fan of the weird ini-like format for systemd or several different tools I'm expected to learn just to read my (now binary) log files competently. If you're sick of switching init systems constantly or don't want to have to write separate scripts for your linux box and your freebsd box even, I highly recommend checking runit out. I'm sure I'll give it a serious shot eventually... in about 3 years once they work all the Poettering kinks out, just like PulseAudio. They're doing some cool things with cgroups and stuff, so I hope it gets there eventually.
- cetra3 8y ago
  I second using runit. We use runit to be able to use the same service definitions inside docker, on a VM or bare metal. If you've ever tried to use systemd inside docker to bring up a couple of services, you would know the hoops you have to jump through to get it working. (I understand that docker wasn't invented to run multiple services in the one container, but sometimes it can't be avoided and simplifies app deployment vastly I.e, using CI to test your service actually starts up as per its definition: just run up a quick docker image with runit and a service definition file)
  - pestaa 8y ago
    I've only seen supervisord as the root process in multi-purpose containers. Is there any significant gain to using systemd instead?
    - cetra3 8y ago
      If you use systemd, you can use standard packages from your distro to run up services inside a container. That's basically the only reason I considered it.
- 8y ago
Lazare 8y ago
> When Lennart Poettering started thinking about the problem, he first looked at Upstart, which was an event-based system that was still based around scripts, but he concluded that he could do a better job. I think I'd like systemd more if I had more confidence that he had done a better job. I never really understood why Upstart didn't get more traction; was it just the Canonical backing made other distros avoid it, or did it have drawbacks I never ran across?
- na85 8y ago
  Indeed. Early systemd was a fucking dumpster fire and part of the reason I no longer use Linux.
  - danbolt 8y ago
    What do you use nowadays? Do you feel it addresses things better?
    - dijit 8y ago
      Can’t speak to the parent but I have a similar feeling. Personally I’ve been using openbsd quite heavily for personal projects, for work my team transitioned from RHEL6 to FreeBSD- and while it has quirks (mostly on installation of software) it is incredibly stable. It’s a shame my chosen cloud provider doesn’t treat it like a first class citizen, but that’s fine since the community projects seem to work on making good images. I still use Linux on my desktop and laptop; since I feel like the design of systemd is more suited to those roles (systemd’s design in general feels like windows service activation) but I’ve been having issues that seem to be related to systemd. So it’s not winning me over.
      - busterarm 8y ago
        Ditto. I've switched to OpenBSD for nearly all personal work. We're still on RHEL/CentOS at work though. I haven't figured out how to get xmonad running on OpenBSD yet or my workstation at work would be on it too.
anfilt 8y ago
Systemd just makes things worse. I remeber seeing someone post one of the simplest init process one could write. It was only about 100 lines of C.
- geofft 8y ago
  It's much less than that, but it's sort of an uninteresting demo I think - you could pretty easily write an init process that just spawns systemd in a container, for instance, and run the whole system in a container. Or write an init process that spawns systemd as pid 2, runs prctl(PR_SET_CHILD_SUBREAPER) to make pid 2 responsible for reaping instead of pid 1, and then sits around carefully doing nothing. If systemd crashes, whatever, hopefully sshd etc. are still up. Of course, if you do that then the question is why even bother with an init process. Patch the kernel to not treat pid 1 specially. If a process gets orphaned, don't reparent it to init, just reparent it to nonexistent pid 0. Have the kernel deal with the thing that the 10 lines of C would be doing (waiting on processes to terminate). I think there are meaningful complaints about systemd but the fact that it's a complicated pid 1 is not one of those. If it were actually a problem someone would have written and productionalized one of the above two approaches.
  - anfilt 8y ago
    It was code that even let one make simple files to start up one shot process or daemons. If a long running daemon died it would restart it. Plus it handled the getting the status of orphaned processes, and few other things. Really the biggest issue with systemd is is scope creep scope creep means more and more packages will have systemd as a hard dependency. Like there some aspects of gnome that will not work without systemd, but at least it's not entirely a hard dependency just means you lose features unless you patch gnome.
    - geggam 8y ago
      Yet OpenBSD has Gnome without systemd. So it is possible.
      - 8y ago
djsumdog 8y ago
I hate how he talks about knee-jerk reactions to change. because I don't think that's what's going on here. I remember I first saw systemd on a release of opensuse. I didn't think anything of it at first, except that I didn't really like the command line interface (systemctl) and I found the flags and options cumbersome. I often see new software in new releases (including the old HAL/DBUS layer) and didn't have the same reactions (although HAL had a lot of issues and was later removed or merged into dbus). I've seen the BSD talk on this and I agree, having a system layer is helpful. It'd be nice if it was plugable, NetworkManager (or others that have some standard messages you can send/get via dbus), consolekit OR logind, etc. systemd does make it nice that I only have to write startup/shutdown scripts once for each distro, but I'm not happy with the layout of target files, the way mounts are handled, some of the weird race conditions I've found between systemd mount targets and fstab, etc. systemd is modular, but the modules are still all part of the whole and are not easily replaceable. The same can be said when Docker went to a modeler refactor, but there are alternative implementations of the entire docker engine. Every attempt to create alternative implementations of systemd have eventually gone unmaintained because systemd keeps getting more and more complex and engulfing more systems. If it wasn't for distros like Void, Gentoo, Alpine, Slackware, et. al, we'd no longer have a choice at all. There would be some things that simply couldn't be deployed on embedded systems because all of the dbus shims just wouldn't exist. It's not that people are opposed to change, it's that there are legit concerns about some of the ways systemd works and is implemented, and the way it's been ham-fisted as a political move in a lot of ways. Honestly, I don't think it will matter in a few years. I think the way things are going, eventually all services will be hosted via docker containers and it will be much easier to make Linux distros that have a tiny init layer that just launches a docker daemon and services. RacherOS already does this, with the init process being a container, which can be uses to start up shell environment containers and other service containers.
- lsc 8y ago
  >It's not that people are opposed to change, it's that there are legit concerns about some of the ways systemd works and is implemented, and the way it's been ham-fisted as a political move in a lot of ways. I personally think the industry needs a lot more resistance to change when it comes to interfaces and other things humans have to understand. I mean, I'm not talking about systemd in particular; I'm talking about in general about how interfaces change over time and people don't seem to take into account the cognitive costs of that change. Sure, ss is better than netstat and IP is better than ifconfig... but how much of that 'better' could you have done in a way that didn't toss away the historical knowledge so many people have of those tools? And really, sysadmin tools are the least of it; I mean, they are operated by professionals, so if you want to pay for retraining (or pay the costs associated with there being fewer of us) People change customer facing interfaces to no benefit all the time, forcing people who are trying to do other things to put effort into re-learning their interface. I mean, my point is that interface changes are expensive, and should not be undertaken without a really good argument that they bring more benefit than the cost of retraining.
mantlepro 8y ago
A well-written, thoughtful article, worth the read. > Nerds have a complicated relationship to change; it's awesome when we are the ones creating the change, but it's untrustworthy when it comes from outside.
- mantlepro 8y ago
  Yikes! How in the world did I lose 2 of my 3 karma points on this?! Somebody please explain and upvote? Linux user since the 90s but totally new to Hacker News. I'm good with constructive criticism, opinions, etc. but not a fan of downvotes without explanation.
stevefan1999 8y ago
I reckon there's a SystemD bug that didn't mount LVM volumes correctly, some has lost data that's real tragedy
- JoshTriplett 8y ago
  Surely if there's an issue that serious, you have a bug report number?
  - shakna 8y ago
    Here you go: [0], from 2017. [0] https://github.com/systemd/systemd/issues/6066 https://github.com/systemd/systemd/issues/6066
    - JoshTriplett 8y ago
      So, fstab says to mount a filesystem, and systemd mounts the filesystem. The reporter explicitly says they want the filesystem mounted at boot time. The response is correct: there's no such thing as "mounted at boot time but not 'after system boot process completed'", because there's no magic "after system boot process completed" point at which that behavior could change. When the device becomes available it gets mounted. sysvinit and typical distribution init scripts couldn't implement the behavior seemingly desired by the user, either.
      - shakna 8y ago
        ... All I did was find the issue number. The ticket follows the sensible approach: Document it, and close it as expected behaviour, and it did lead eventually to an LVM issue that did require a fix [0]. systemd did good here, above board, but there was a faff about it, because the expected behaviour of the audience differed from the reality of the situation. [0] https://github.com/systemd/systemd/pull/6174 https://github.com/systemd/systemd/pull/6174
    - stevefan1999 8y ago
      plus this: https://github.com/systemd/systemd/issues/11255 https://github.com/systemd/systemd/issues/11255,
voltagex_ 8y ago
I'd recommend everyone watch the video: https://www.youtube.com/watch?v=o_AIw9bGogo https://www.youtube.com/watch?v=o_AIw9bGogo
bitwize 8y ago
How about no? How about you not lecture to me that I only hate systemd because I'm just butthurt that Linux is changing, and I need to get with the program? Because that's how people who want to wrest control talk. It's like telling people that the only reason why they dislike Ghostbusters (2016) is because they're misogynists who feel threatened by strong women. Who are you to tell me how I feel and why I feel that way? I really don't like systemd, but I'm glad that it exists. Because more free software is better than less free software. I just don't want to use it on the systems I run. And I don't want the entire Linux ecosystem to hard-depend on it. Lennart can implement all the cockamamie ideas he wants; as a software developer that's his right, and as a software developer myself I would be remiss in defending software freedom if I did not acknowledge and support his exercise of that right. Where I draw the line is the subsequent push to ensure that the Linux ecosystem breaks without systemd.
dingo_bat 8y ago
My only issue is the binary log files. When you're writing out the logs to disk, just make sure it is in text! How hard can it be?? It creates unnecessary friction when dealing with logs from a crashed system.
- XorNot 8y ago
  Does this ever actually come up for all the people complaining about it? Where are these people working that they're not willing to install rsyslog which reads journald logs directly and will write plaintext, but who have this problem enough to warrant all the complaining? Like, previously to have any logs at all you had to have a syslog daemon. This is not a new situation.
  - inferiorhuman 8y ago
    > Like, previously to have any logs at all you had to have a syslog daemon. This is not a new situation. What's new is that now more and more-difficult-to-debug pieces are required. For what benefit?
  - yarrel 8y ago
    This is a "now you have two problems" solution.
Karrot_Kream 8y ago
We specifically use runit because we've had systemd issues in the past. Systemd may have matured now but at this point we're moving over to containers anyway, so it doesn't matter.
makz 8y ago
As a conspiracy theorist, I'm going to say that systemd is aimed towards destroying Linux and everything it represents. They want to push the cloud, to better control us, take away power from us, leave us at their mercy. The next step is, when everything that matters is on the cloud, they are going to replace Linux with something else that only they know how it works on the inside, and that's going to be the end of it, we'll be left only with paywalled APIs and services, but they'll say it's better for everyone and all of that. It's sad.
- newnewpdro 8y ago
  You're confusing systemd with docker and kubernetes.
  - busterarm 8y ago
    I hate how accurate this is. The more I work on Kubernetes implementations, the more I feel like it's a tool that 99% of companies can't effectively operate. The attitude is very reminiscent of the mainframe era where ops didn't want mere users touching their precious systems. I feel like as an industry we're wasting tons of cycles solving the wrong problem.
squirrelicus 8y ago
I hate how the Unix philosophy argument is boiled down to "one vs many" binaries. It's about independence to the degree of reusability. There's nothing in systemd suite that's useful unless you're two feet in. You can say things about streams and text and binary formats, but those are distractions. Systemd tooling is tightly coupled to each other. It's a huge codebase, a huge system that shoddily replaces that which was historically done with less. And it's the biggest reason I don't run a Linux workstation anymore.
- progman 8y ago
  Basically said, systemd follows quite the opposite of the KISS principle which made Unix and Linux so great.
jiqiren 8y ago
My impression of systemd: shoddy version of launchd.
myrandomcomment 8y ago
I hate systemd with a passion, however I understand the reason for it. I think that systemd is a bit Franklinstien by the nature of having to pull udev and d-bus into it. So I guess it is a victim of having to deal with the system it is on. Oh well. I think the only Unix system to get this right was likely Solaris via SMF. https://en.m.wikipedia.org/wiki/Service_Management_Facility https://en.m.wikipedia.org/wiki/Service_Management_Facility WindowsNT also did it correctly.
j16sdiz 8y ago
I hate parallel service start up on server. Yes, it is faster. But when it breaks because bad dependency, it is hard to reason the breakage on a initrd shell on a small remote console. I value debuggability, reliablity and manual fallback options on server.
- kakwa_ 8y ago
  I hate writing systemd .service files. I'm always confused with the Type (should it be simple? forking?) and my program ends-up starting but not stopping or things like that. I know I lack the knowledge to write a good .service file in one go, as I write one every other month at best and forget all the details in between. It would be the same with a traditional init scripts. However, with init scripts, the way to debug is obvious: just add a '-x' somewhere and here we go, you have a detailed execution. I don't know the equivalent with systemd and I end-up bashing my head against the wall trying everything blindly.
throw2016 8y ago
The author goes into 'contempt culture' which is valid in any group of people ie 'PHP haters' but then tries to slip through the old trope of 'resistance to change' which is a clever bait and switch that positions change as 'always good' and refashions dissenters as anti-change which is a time tested PR strategy to push through unpopular changes that benefit a few by positioning any 'dissent' as 'anti change' and thus avoiding the meat of the discussion - the change in question. We can agree on the problem and then on the solution, but any proposed solution can't just pass through 'without scrutiny' because a problem exists. Systemd proponents since the beginning have positioned any criticism as 'motivated', 'greybeards', 'haters' or anti-change which tells you that there is no space for discussion here, only 'acceptance'. Can there be any legitimate criticism of systemd without this kind of politics, hypersensitivity and now posing as victims?
- aidenn0 8y ago
  Also the whole "One solution for people who don't like it is to create their own alternative; that is a good way to find out just how much fun that task is." This seems to imply that those criticizing it didn't have alternatives, which is just not true. However it is so monolithic that as soon as a tiny part is depended on by something else (e.g. gnome) alternatives quickly become a non-starter. systemd seems to be getting rewarded by not playing well with others which is something that plays strongly into a lot of people's dislike with it.
jimrandomh 8y ago
Back in 2016, systemd started killing user processes on logout (rather than send them the SIGHUP signal, as POSIX says should happen). This caused problems for programs like nohup, screen and tmux, which deliberately keep running. Systemd's response was to say that they should incorporate systemd's library, and use systemd's new daemonization API. As far as I know, none of them did. Two years later, you can find hundreds of support requests across the internet, from frustrated users who are having their sessions killed by systemd. Bugs are annoying, but that's life. On the other hand, when you're an impacted user who's lost work, and researching the bug leads you to a years-old discussion in which someone is actively denying that the bug exists and refusing to fix it, that's infuriating. I don't think systemd's developers deserve the trust that maintaining a core piece of infrastructure requires; they don't seem to care enough about whether they've broken things.
- MereInterest 8y ago
  Absolutely. I can understand implementing this feature for some special cases, like containers that should clear all hint of a user away on log off. It should never have been the default, and breaks an entire category of software. In my standard .bashrc file, I have the following snippet to warn me if I am on a system with that stupid setting enabled. if which loginctl > /dev/null && loginctl >& /dev/null; then if loginctl show-user | grep KillUserProcesses | grep -q yes; then echo "systemd is set to kill user processes on logoff" echo "This will break screen, tmux, emacs --daemon, nohup, etc" echo "Tell the sysadmin to set KillUserProcesses=no in /etc/systemd/login.conf" fi fi
  - m45t3r 8y ago
    Thanks, now I know why Emacs daemon keeps delaying my restarts in the system (just discovered that NixOS defaults KillUserProcesses to false). Turning this on to true, for me it does no make sense to a user service (yeah, I run emacs as a user's systemd service) to keep running after I logout of my system. P.S.: And the fact that for some people this behavior makes sense is why I think Lenart decision to put this as an option makes sense.
m463 8y ago
I believe systemd solves a problem or two. However, I believe the implementation is not great. A couple specific observations: * lots of binaries. /lib/systemd, the logs. This more than anything else is a tragedy. Binaries get in the way of viewing, diagnosing, understanding and changing things. It seems like premature optimization to me - I don't think speed requires it. I think the majority are completely unnecessary. * copying from launchd? Commercial software vendors ship binaries. They are aligned with them. They resist disclosure, which protects from reverse engineering, lawsuits and user modifications. Does that align with linux? * the config files are a disorganized mess. It is like /etc/init.d but poorly understood and then /etc/rc.d was lumped into the same directory and organization became chaos. I looked at one system here and in /etc/systemd there are 16 directories, 23 symbolic links and just 11 files. Try unraveling the interdependencies. additionally the config files (which mimic windows config files) lack depth. Simple config files can have benefits, say if a GUI read and wrote them to change settings, but I don't think that applies here. What I noticed is that they cannot do anything, so additional logic requires an intermediate script or compiled binary elsewhere. * it is responsible for too many functions. I bet it didn't solve so many in its first iteration, but sucked in more over time. It reminds me of the accounting program that grew to rule everything in Tron. I wonder if these problems could be resolved. It's hard to remove complexity and add elegance later.
- atombender 8y ago
  Launchd has been open source (Apache license) pretty much from the beginning: https://wiki.freebsd.org/launchd https://wiki.freebsd.org/launchd
- mixmastamyk 8y ago
  Systemd as the new MCP, interesting. “Sark, prepare that daemon for immediate deresolution.”
- poettering 8y ago
  I kinda like how a good chunk of the community complains we are "too monolithic", and the other side of the community complains we have "too many binaries". We can't win with you guys, can we? Lennart
mirimir 8y ago
Damn. So I'm just a simple Debian user. Systemd showed up, so I learned as much as I needed. And in some ways, I found it easier to use than init scripts. Now I've been using Docker. And there is no systemd in containers. But of course, why would you want that? But the problem is that many packages now depend on systemd. Or at least, useful features do. So I'm back to fighting with scripts. It's funny, no?
- inferiorhuman 8y ago
  > And there is no systemd in containers. But of course, why would you want that? I knew a guy who invented an orchestration system that involved shoving an init process into docker containers. Much sadness ensued.
  - mirimir 8y ago
    Are you referring to dumb-init? Is there a better way to reap zombies?
    - inferiorhuman 8y ago
      > Are you referring to dumb-init? Nope.
- tomjakubowski 8y ago
  Maybe you wouldn't want systemd, but a basic init system as PID 1 is wise for a Docker container. See https://github.com/Yelp/dumb-init https://github.com/Yelp/dumb-init
  - mirimir 8y ago
    Wow. Thanks. I've been wondering about all those zombies.
    - cyphar 8y ago
      I would suggest https://github.com/openSUSE/catatonit https://github.com/openSUSE/catatonit (self-plug -- I wrote it). dumb-init (and tini which is what "docker run --init" uses) don't handle zombies in the most ideal fashion possible, catatonit does. And it does so in less code.
Kivutar 8y ago
Some years ago, I was developing a Linux distro for embedded systems. Some of these embedded systems come with old kernels, and you will never get a recent kernel for them. Systemd is written in C and relies on kernel headers. Which means that they need to have a layer that deals with different versions of the kernel headers. I remember it was headers about network. At the time I was working on this, I had to contribute to systemd to add support for my kernel version. Got a few patches accepted in their master. And things were working fine for a while. Until they suddenly decided to drop <= 3.10 kernel support. Yep. Afterwhat I had to maintain patch sets for systemd, which is a great amount of job, given that my upstream was updating systemd quite often. Now my good people, explain me how such a mess could have happen with a previous init based on shell?
- CaliforniaKarl 8y ago
  I'm curious: Did systemd drop 3.10 support before the kernel developers stopped supporting it (in 2017, see http://lkml.iu.edu/hypermail/linux/kernel/1711.0/03167.html http://lkml.iu.edu/hypermail/linux/kernel/1711.0/03167.html), or before that? If it was around the same time, then I could understand dropping support, since even the kernel developers had decided to move on.
- jononor 8y ago
  You could have stayed on an older systemd version? Preferably one used by a major distro, such that someone else might backport security fixes etc for you. Anyway the root cause here is bad vendor BSPs being stuck on old Linux versions.
- teddyh 8y ago
  GNU libc 2.26 (released 2017-08-02) requires Linux 3.2 or newer. It’s not like systemd is alone in requiring recent kernels.
aidenn0 8y ago
One of my "favorite" corner cases caused by systemd was that if an NIS user tried to login to the system, it would cause my X session to restart: 1. They changed the defaults for systemd-logind to disallow network access 2. When systemd-logind attempts to send a message to the NIS server, it instead hangs (or has a timeout greater than the systemd watchdog anyways) 3. The watchdog timer causes all systemd-logind services to be restart, which then tears down all of the child processes. My X session is a child process.
RemarkableMan 8y ago
I just don’t have time to beta test an init system with more LoC than the 2.6 kernel plus GCC 4. For now Devuan is where we landed but moving entirely away from Linux as Redhat moves system d into the kernel is the overall plan.
pgtan 8y ago
systemd broke UNIX. Everything that worked before, now is a mine field. Last week I had a SLES virtual machine refusing to boot, no network, just emergency shell. The reason was a forgotten line in fstab (without nofail option of course), so systemd couldn't mount some file system. Coming from Gentoo, this is just unbelievable.
- the_why_of_y 8y ago
  Systemd correctly implements the "noauto" and "nofail" keywords in fstab(5) that were added in 2008 in the way they have been documented since 2010: if neither of these is specified for a filesystem, then that file system will be mounted at boot time and a failed attempt to mount that filesystem implies a boot failure. Util-linux-ng 2.14 Release Notes (09-Jun-2008) ============================================== mount(8) supports new "nofail" mount option. https://github.com/karelzak/util-linux/commit/abe3d704b6aeb6b82ff32e8599edb56525cdfd72 https://github.com/karelzak/util-linux/commit/abe3d704b6aeb6... The fact that previous init scripts gladly ignored these failures does not reflect negatively on systemd but on those init scripts. Since 2011, FreeBSD has the same options with the same behavior, except that "nofail" is called "failok". https://svnweb.freebsd.org/base?view=revision&revision=222832 https://svnweb.freebsd.org/base?view=revision&revision=22283... Add a special mount option "failok" to indicate that the administrator wants the system to proceed to boot without bailing out into single user mode, even when the file system can not be successfully mounted. By your argument, FreeBSD broke UNIX.
- zlynx 8y ago
  systemd refusing to boot without required drives is actually a favorite feature of mine. Ever since I had an RH AS server boot without /var mounted. It was truly lovely with a gigabyte of trash filling up / then trying to decide if it was worth merging data or just throwing it out.
pdkl95 8y ago
> This argument, he said, seems to be predicated on the notion that systemd is a single, monolithic binary. Can we please stop misrepresenting the complaints against systemd? The only time I ever hear this "monolithic binary" argument is from systemd advocates. The actual complaint is about tightly coupling important features together. Not only does this make it difficult (often impossible) to replace individual components, when tight coupling happens at the (internal) protocol level, any replacement component ne4cessarily hast to implement a bunch of (sometimes unwanted) systemd baggage. Busybox implements all of its features in single monolithic binary, but it isn't a monolithic design that tightly couples those specific components together. Replacing one of busybox's components is often as simple as removing busybox's symlink and installing the replacement. This isn't even a "Unix philosophy" issue. Even inexperienced designers shouldn't have as hard time Understanding why systemd is a monolithic design but busybox isn't.
- chris_wot 8y ago
  What baggage are you specifically referring to?
  - lmm 8y ago
    It runs its own logging system with non-standard interfaces and formats. It runs its own DNS resolver with non-standard behaviour. It maintains compatibility only with a narrow range of udev versions, which in turn maintain compatibility only with a narrow range of kernel versions. And all the d-bus interfaces between these pieces may change at any point without notice. So you can't replace any piece of it, because even if you provide your own component that implements one of the systemd d-bus interfaces, you've got no forward compatibility.
    - pas 8y ago
      If there was a serious effort to replace/port parts of it, the needed internal APIs can be stabilized ( https://www.freedesktop.org/wiki/Software/systemd/InterfacePortabilityAndStabilityChart/ https://www.freedesktop.org/wiki/Software/systemd/InterfaceP... ).
ChuckMcM 8y ago
From the article -- Windows NT had a strong service model from the beginning, he said, and Mac OS has one now in the form of launchd. Other systems had to play a catch-up game to get there. Reminds me of the saying, "and if your friends all jumped off a cliff, would you jump off too?" which exasperated parents use to council their children that emulating others needs to be done responsibly.
- darkpuma 8y ago
  There is a big difference here. To continue with the cliff analogy: Launchd ran fast enough to clear the cliff and was rewarded by a refreshing dive into the cool water; it was implemented in a competent manner and consequently didn't cause people to become upset. Systemd on the other-hand can't run without stumbling, it hobbled off the cliff and bounced off every rock on the way down. It made lots of people pissed off because it interfered with their routines. If systemd didn't cause problems for people, virtually no user would even know what it was, and consequently, virtually no user would hate it. If you ask your average OSX user what launchd is, they'll come up with a blank because launchd isn't causing trouble for them. If you ask your average linux desktop user what systemd is, a large portion of them will recognize the name, and many of them will chew your ear off out of frustration. At the end of the day it's not about "adherence to philosophies" or any similarly dubious concepts. Whether or not software is hated is ultimately determined by whether or not it causes problems for people.
- JdeBP 8y ago
  That's not the problem. Windows NT did have that right from the beginning. The problem is the second sentence, because it ignores that so too did Unix from the same timeframe. AT&T System 5 Release 4 had the Service Access Facility from 1988. AIX had the System Resource Controller from version 3.2. Other systems were not playing catch-up. They had this stuff too, at around the same time. Benno Rice's explanation is actually ahistorical here. What actually happened is that the world spent a lot of time "cloning" existing Unix softwares during that time period, for reasons that we all known, and a lot of the time the clones were behind the times and did not catch up. By the time that Miquel van Smoorenburg cloned AT&T Unix init and rc for Minix, for example, what xe was cloning had already become years out of date. * http://jdebp.eu./FGA/inittab-getty-is-history.html http://jdebp.eu./FGA/inittab-getty-is-history.html
chroma 8y ago
It is my opinion that Lennart Poettering is the software equivalent of Thomas Midgley Jr., the inventor of both leaded gasoline and chlorofluorocarbons.[1] This is the person behind PulseAudio, Avahi, and systemd. In any sane world, these software projects would have been stillborn. Sensible alternatives would have outcompeted them. Instead, Red Hat did their best to foist them upon the community. They promised that after the initial adoption, all the wrinkles would be ironed out. All three times this was not true. I hate to appear so cynical, and I'm certain that Red Hat didn't intentionally pursue this strategy, but it sure is convenient for them that the harder Linux is to use, the more they can sell support contracts. Not so convenient for people who actually want to use Linux. 1. https://en.wikipedia.org/wiki/Thomas_Midgley_Jr https://en.wikipedia.org/wiki/Thomas_Midgley_Jr.
- darkpuma 8y ago
  You might be giving Lennart too much credit. As the author of the article points out, Lennart didn't invent PulseAudio and systemd so much as he ripped them off (poorly) from coreaudio and launchd. To compare him to any inventor seems too flattering.
  - mixmastamyk 8y ago
    Also not a huge fan but it works better than a bunch of shell scripts.
    - flukus 8y ago
      Don't you still have to resort to shell scripts for anything slightly complicated (mounting a drive on a specific network connection) and just have systemd invoke them?
      - arminiusreturns 8y ago
        So far this is a particular thing I have run into. DevOps (yelling): no more shell scripts Devs: systemd fstab is all kinds of fucked for nfsv4 Devs: Writes super complicated script to mount Me/DevOps: Makes ansible script to make systemd mount paired with service that is basically a wrapper for... a shell script...
fmajid 8y ago
Farce, not tragedy
yarrel 8y ago
First as tragedy...
_pmf_ 8y ago
If the current state is the tragedy, I'm terrified of the farce. My favorite systemdisms: * (already mentioned several times above; sorry) even detached and nohup'd processes will be killed if the systemd context that transitively led to the creation of the process is killed: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394 * mounts being in a separate "mount" namespace that is not accessible normally: http://www.volkerschatz.com/unix/advmount.html http://www.volkerschatz.com/unix/advmount.html
ComputerGuru 8y ago
For all those lamenting the death of standards, change for the sake of change, system breakage, bloat, and lack of user control: give FreeBSD a try (or any BSD, really).
progman 8y ago
I like to share the following links to those who also don't like systemd. http://without-systemd.org http://without-systemd.org Devuan (Debian without systemd): http://devuan.org http://devuan.org More Linux Distros without systemd: http://without-systemd.org/wiki/index.php/Linux_distributions_without_systemd http://without-systemd.org/wiki/index.php/Linux_distribution...
- fanf2 8y ago
  Debian Stretch works nicely for me without systemd
arminiusreturns 8y ago
I particularly like how it ripped out the old logging system but completely failed to replace it properly. Local might be ok, but shipping to a central log server? Just save yourself the headache and use syslog or some other shipper like filebeat, etc. How can something that can't even do logging properly suddenly become the defacto standard?
- newnewpdro 8y ago
  > I still think NSA used RedHat to push systemd for potentially nefarious reasons. Wow, just wow. Do you have any idea how insulting this is to the folks dedicating their lives to writing this GPL software? You should try having an actual conversation with the authors some time. These are real people and long-time GNU/Linux users and advocates you're implicitly accusing of being NSA collaborators.
  - arminiusreturns 8y ago
    RedHats collaboration with NSA is well documented. I think given what we know of past actions it's at best extremely naive to not at least entertain it as a possibility. Linus himself semi-openly admits he's been approached... https://www.youtube.com/watch?v=7gRsgkdfYJ8 https://www.youtube.com/watch?v=7gRsgkdfYJ8
CarlFK 8y ago
Notable quotes 22:10 "Unix is dead." 25:28 "Change is awesome when we are the ones doing it. 27:20 "No one needed to send death threats over a piece of software." 27:30 "Contempt isn't cool."
thekingofh 8y ago
Can I easily replace systemd on a distro that came with it? I wouldn't consider myself a Linux expert, but I got to a point where I could get around and started to understand the different init runlevels and how to set them, then everything was ripped out from under me and I no longer understand how to work everything anymore. Perhaps Linux boots faster now, I don't know, but it was really very frustrating I couldn't find a major user-friendly distro that worked the way I was used to.
- JdeBP 8y ago
  The writing was on the wall for run-levels for a long time. Longer than Linux has existed, in fact. * http://jdebp.eu./FGA/run-levels-are-history.html http://jdebp.eu./FGA/run-levels-are-history.html
gerbilly 8y ago
There are parallels with city planning. systemd reminds me of the efforts in some cities at clearing 'slums' to replace them with 'modern' apartment blocks.[1] sytemd seems like a similar technocratic, modernist effort aimed at UNIX. Operating systems are kind of like cities. One the surface they are often inefficient and messy, and people often wish they could come in and make them logical and rational, but people live in those cities and neighbourhoods, and sometimes the postbox is where it is for a reason. If Jane Jacobs[2] were around, she could write a book called: "The Death and Life of Great American Operating Systems" [1] See: https://en.wikipedia.org/wiki/Pruitt%E2%80%93Igoe https://en.wikipedia.org/wiki/Pruitt%E2%80%93Igoe, https://en.wikipedia.org/wiki/Quarry_Hill,_Leeds https://en.wikipedia.org/wiki/Quarry_Hill,_Leeds, https://www.youtube.com/watch?time_continue=862&v=Dxr2tEGlWUE https://www.youtube.com/watch?time_continue=862&v=Dxr2tEGlWU... [2] https://www.citymetric.com/fabric/against-modernist-nightmare-legacy-urbanist-jane-jacobs-2185 https://www.citymetric.com/fabric/against-modernist-nightmar...
dekhn 8y ago
I've used linux systems since '93. I waited until the last minute to adopt systemd. So far, for me at least, it's been great. It's a lot easier to express service dependencies, find problems when they occur. I hear a lot of hate, but in all, systemd seems to solve a ton of problems in a single place, which is something I think linux needed for a while.
jiveturkey 8y ago
more like abomination
merb 8y ago
a lot of people shit on systemd because of the "SIGHUP"-"bug" however what they don't realize is that there are more use cases where logging out a user SHOULD kill all users processes. I mean without that behavior linux is not a multi user system. The default is sane, however the lack of "permissions" in this kind is also a little bit bad, however since it is runtime configurable this should be enforced by the distribution, i.e. when installing the distri there should be a switch "this system is only used by a single user". it's basically the same as windows uac. while it looks bad for a single user, it's not that bad when you see computers as a tool which can have more than one user, with more than one "privilege". system just shows that linux is by far not ready for the masses cause a lot of behavior is undefined, like the uid/username stuff. usernames can be created as you wish, but most programs will behave broken if you create usernames that start with a number, it's a security nightmare. and the problem that there is no real process to actually get ALL people together who can solve this, it will never be fixed. there is only the posix standard but not everything applies to linux and not everything is a good idea in the sense of linux. systemd actually makes a lot of things in the right way, but of course it's not perfect and probably at a certain point in time people would come up with something better.

288 points by wyldfire 8y ago

ordu 8y ago

> "It's software" so of course it's buggy, he said. The notion that systemd has to be perfect, unlike any other system, raises the bar too high. systemd is a PID 1 program, it means it have to raise bar higher. When troubles begin, you would need tools to fix them, and if PID1 is crashed, you are out of luck. If system cannot boot into shell, you'd need to fix it from initrd shell. Or to boot other system, to fix this one. It sucks. Linux kernel chases very high standards of reliability, because when kernel panics it is even worse than PID1 crash. Init system should follow the same standards as linux.

geofft 8y ago
Have you ever had pid 1 (systemd or any other init) crash? For the last ~three years I've been paid to maintain high reliability algorithmic trading systems that ran systemd and a whole lot of other stuff, and systemd has never crashed on me. Lots of other stuff, including the kernel itself, has crashed. The bar is higher for pid 1 - if I were designing systemd I would have made a tiny pid 1 that just did message-passing to a more complex secondary process that could be restarted, or something, just to be safe - but I think systemd has empirically cleared the bar.
- CBLT 8y ago
  I've never experienced an outright crash, but I've been bitten by [0] on some of my servers. [0] https://github.com/systemd/systemd/issues/719 https://github.com/systemd/systemd/issues/719
- rachelbythebay 8y ago
  Yep. CentOS 6's upstart can be felled by generating a bunch of inotify events in /etc. http://rachelbythebay.com/w/2014/11/24/touch/ http://rachelbythebay.com/w/2014/11/24/touch/
- rurban 8y ago
  The current debian testing version crashes with a NULL pointer segv in the kernel module. You need to downgrade to the previous version.
  - newnewpdro 8y ago
    In what kernel module? There is no "the kernel module" in a systemd context.
    - rurban 8y ago
      There is. udev loads kernel modules. See eg. http://www.linuxfromscratch.org/lfs/view/development/chapter07/udev.html http://www.linuxfromscratch.org/lfs/view/development/chapter...
      - JdeBP 8y ago
        Waving in the direction of udev does not clarify what kernel module is supposedly the kernel module, which is what you were asked.
        geofft 8y ago
        'rurban is one of our resident trolls - see also https://news.ycombinator.com/item?id=13364173 https://news.ycombinator.com/item?id=13364173
- ordu 8y ago
  > Have you ever had pid 1 (systemd or any other init) crash? No, I have not. But I have seen how systemd gracefully failed to boot system to login, with good looking colorful error message. Something that reminded me "Keyboard is not found. Press F1 to run setup."
  - geofft 8y ago
    Oh, to be clear I'm real mad about how systemd fails boot if (say) one of your filesystems is unavailable and makes you log in with a root password to fix it. But OP was asserting that systemd crashes under normal operation because its pid 1 is too fragile, which is very different. At scale I already expect that there's a chance a machine won't come back if I reboot it - it's annoying if I can't ssh in, but, well, I already lost a disk I care about and it won't return to service and I need to fix it anyway. (And it's an easy fix, just add "nofail" to fstab.) At scale I don't expect init to crash under normal operation.
- ploxiln 8y ago
  I've had shutdown (for reboot) hang a few times after a systemd update, forcing me to cut the power. It's made me a bit paranoid, so I block the systemd package from having updates automatically installed, and every 6 months or so carefully manage update and reboot of each and every server ... EDITs: there's the classic case of the linux "debug" parameter: https://bugs.freedesktop.org/show_bug.cgi?id=76935 https://bugs.freedesktop.org/show_bug.cgi?id=76935 and the even more classic case of firmware loading events: https://lkml.org/lkml/2012/10/3/484 https://lkml.org/lkml/2012/10/3/484 and while "all software has bugs" systemd really has the most annoying bugs (by virtue of trying to do everything core to the system) and always insists that they are features and we are backwards whiny geeks for complaining.
- 65a 8y ago
  There was that time it was bricking computers by erasing UEFI variables, but I'll allocate equal blame between systemd and UEFI
  - chris_wot 8y ago
    Actually, that was nothing to do with systemd. That was definitely a UEFI implementation issue. And systemd didn't delete anything, the user did - they ran: rm -rf --no-preserve-root / https://lwn.net/Articles/674940/ https://lwn.net/Articles/674940/
    - majewsky 8y ago
      The bug was in the kernel (it should not have allowed userspace to write arbitrary UEFI variables), but AFAIR it was exposed by systemd because it eagerly mounted the UEFI variable filesystem provided by the kernel into /sys/something/efivars.
      - chris_wot 8y ago
        Indeed, but again that was a firmware issue. systemd didn't delete the variables. And systemd was setting EFI variables, so consequently it needed it to be mounted as read/write. The configuration files should have set that to read only after boot. The kernel patch where this was fixed can be found here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0389075ecfb6231818de9b0225d3a5a21a661171 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
  - voxadam 8y ago
    Personally, I lay the blame for this issue squarely at the feet of various UEFI implementations which fail to boot when the system's EFI variables are, for whatever reason, wiped clean. The UEFI spec explicitly states that clearing all of the variables on a system must not result in an unbootable system.
  - JdeBP 8y ago
    You shouldn't, because the maintainer of the kernel subsystem concerned told us all that systemd wasn't to blame for it. * https://news.ycombinator.com/item?id=15973577 https://news.ycombinator.com/item?id=15973577 * https://news.ycombinator.com/item?id=11152880 https://news.ycombinator.com/item?id=11152880
- setquk 8y ago
  I’ve had Systemd completely stop responding before on numerous occasions on centos 7. As in can’t reboot or hangs rebooting or all commands hang. Only recourse has been to reboot the instance from AWS dashboard. I can’t get to the bottom of it because the tools don’t work when it’s down and there’s nothing there when it comes back up. I am not enjoying boiling to death in this pot of shit. And then there’s the situation where it just won’t boot. I just fire up a new instance then because it’s easier than debugging it.
- dvfjsdhgfv 8y ago
  > Have you ever had pid 1 (systemd or any other init) crash? For the last ~three years I've been paid to maintain high reliability algorithmic trading systems that ran systemd and a whole lot of other stuff, and systemd has never crashed on me. You see, that's the argument I hear a lot from Systemd advocates. The problem with anecdotal evidence is obvious. When you hear people opposing Systemd, practically all of them have some real-life issues with it, often related to functionality that would otherwise be non-essential (i.e. doesn't really need to be handled by PID 1). Of course if you don't have a particular problem, you don't feel it's important. That's precisely the attitude people resent.
  - geofft 8y ago
    > When you hear people opposing Systemd, practically all of them have some real-life issues with it Yes, but a lot of people have real-life issues with it on their desktop of the form "It's too complicated." I'm asking specifically about real-life issues on production servers at scale. There will of course be tools that are poorly suited for a personal machine (even a personal server) but well suited for a team that wants to run a bunch of reliable servers. For instance I would never be happy running RHEL on my desktop, but that doesn't mean RHEL is useless.
    - dvfjsdhgfv 8y ago
      I can't quote any statistics but have the impression that a large part of non-Systemd crowd are old-time admins who maintain a large number of servers, myself included. When you break something on a desktop machine, that's easily fixable. When you need to deal with a large heterogeneous environment, you prefer to have things handled a bit more gracefully. Linus is a good example of a person who got this right.
- nonbirithm 8y ago
  Very recently I had this issue[0] as the result of a systemd upgrade, requiring the use of a recovery disk to downgrade to the previous version as the keyboard input had failed to be initialized. [0] https://github.com/systemd/systemd/issues/11314 https://github.com/systemd/systemd/issues/11314
  - majewsky 8y ago
    If this bug hits you in staging, no big problem, just don't promote that particular update to production. If this bug hits you in production, your lack of a staging environment is the bigger concern IMO. (I have been hit by the same issue on my private notebook, but I have procedures in place to cleanly recover from failed upgrades on all systems, so it was not a big deal.)
    - yjftsjthsd-h 8y ago
      "Yeah the software completely broke, but that's fine because you should be able to deal with that" does not make me feel better about the software in question.
- RX14 8y ago
  Using systemd automount and NFS you can easily get pid1 unresponsive, hung in uninteruptible sleep forever.
- stewartm 8y ago
  Yep. 3AM, deep slumber, called out to look at a stricken server. Its problems included that systemd was frozen. Reluctantly I came to the conclusion that a restart was the only route forward. Cept, that is when you discover that the commands that have served you well for 2 decades don't work, as they are all wrappers for systemd, which has keeled over. To this day, the `shutdown` man page, which I was checking in, makes no mention of how to resolve, tho in fairness the other commands (poweroff, halt, init) do. I discovered this after stumbling across https://github.com/systemd/systemd/issues/3282 https://github.com/systemd/systemd/issues/3282 If you find yourself stuck in the middle of the night, reading through docs to try and figure out how recover a machine with a crashed systemd, then `systemctl reboot -ff` or equivalent is what you are now looking for, the `-ff` being the key to "JUST £&*(ing RESTART THE MACHINE!!!". Experiences like that, don't win you friends.
  - Twirrim 8y ago
    Did raising elephants not work (SysRq + R E I S U B)
    - auscompgeek 8y ago
      systemd disables the magic sysrq keys by default.
  - setquk 8y ago
    The worst thing about this is when stuff goes down, it does so at the least convenient time. Back in 2003 I was on a customer site who had a RH server and there was no internet connection available (as it was routed through the box) and my phone was a Treo 180G which had precisely fuck all useful internet on it. The company still exists and is in the middle of nowhere on the end of a shonky ADSL line and no mobile phone reception so the story hasn't improved. If this happened to me today with systemd I'd be up shit creek without a paddle.
- JdeBP 8y ago
  No. Not mine. Not systemd. Not others. And I touched upon how rare this was in practice in my experience some years ago on Hacker News. * https://news.ycombinator.com/item?id=8384251 https://news.ycombinator.com/item?id=8384251 But it does happen to other people. * https://unix.stackexchange.com/questions/440229/ https://unix.stackexchange.com/questions/440229/ And there was one crash that made the headlines. * https://news.ycombinator.com/item?id=12600413 https://news.ycombinator.com/item?id=12600413

andrewstuart 8y ago

I really like systemd. It's got tons of really great functionality. systemd feels like it is well thought out and modern, like it's an integrated rethink and rebuild of lots of various software utilities that have evolved over many years. I'm investing all my learning in systemd where appropriate instead of things I used previously like cron and supervisor.

CaliforniaKarl 8y ago
I agree with that. But at the same time, systemd makes assumptions that I don’t think it should. Case in point: I deal with several clusters that use Stanford Central LDAP for account info, and our UID numbers have gotten pretty high. So much so that it overlaps the range systemd uses for dynamic service UIDs. The biggest annoyance is that the UID range is hard-coded, so I can’t provide an alternate range. More details: https://github.com/systemd/systemd/issues/9843 https://github.com/systemd/systemd/issues/9843 (but please don’t spam the GitHub issue!)
- inferiorhuman 8y ago
  Do you remember when systemd would take usernames it couldn't parse (due to undocumented restrictions on valid usernames) in unit files and just translate them to uid 0?
  - CaliforniaKarl 8y ago
    It seems that was fixed in https://github.com/systemd/systemd/commit/bb28e68477a3a39796e4999a6cbc6ac6345a9159 https://github.com/systemd/systemd/commit/bb28e68477a3a39796...
- tyingq 8y ago
  Wish my customers could file bugs and feature requests with that kind of thoughtful detail.
  - CaliforniaKarl 8y ago

darkpuma 8y ago

> "I'm not going to defend Lennart's approach to community interaction, but what I will say is you have to admire the willpower and determination of somebody who at work one day says 'I'm gonna write my own init system- actually no it's going to be a full on system layer for linux' [...] and then he gets it put into pretty much every major linux distribution." Yeah, I suppose I also admire the willpower and determination of Genghis Khan. Corbet is almost as dismissive as Lennart himself of the life-disruptive bugs real people encounter due to Lennart's approach to software engineering and project management. "It has bugs? It's software" This attitude is positively correlated with the amount of hate systemd (and pulseaudio) have gotten over the years.

HarryHirsch 8y ago
If planes fell out of the sky due to buggy software as often as systemd crashes due to yet another bug, air travel would get much hate, too.
- freddie_mercury 8y ago
  If systemd cost as much as a plane, it would have fewer bugs. We can go back and forth with dumb comparisons all day but I don't really think we're advancing the conversation.
  - peterwwillis 8y ago
    I'm in the dumb comparisons camp, so here's mine: Systemd is communism, BSD init is capitalism.
  - HarryHirsch 8y ago
    I disagree about the value of the conversation. Airplane software is meticulously tested because of the safety implications. If we can't have proper testing for a widely used, sprawling piece of software that runs with root privileges and PID 1 we may be better off without that piece of software. Also this: the equally widespread SQLite seems to be programmed with a much better level of discipline and comes with much better testing. It isn't nearly as bug-ridden.

deleted 8y ago

[deleted]

shittyadmin 8y ago

This article is a bit of a joke "It's software so of course it's buggy" isn't a great argument when you're replacing something that didn't suffer the same issues. I just count my blessings that runit is widely packaged in every major distro because it can just happily sit on top of sysvinit, systemd, upstart, pretty much any init system and does things in a very simple shell script style, I really wasn't a fan of the weird ini-like format for systemd or several different tools I'm expected to learn just to read my (now binary) log files competently. If you're sick of switching init systems constantly or don't want to have to write separate scripts for your linux box and your freebsd box even, I highly recommend checking runit out. I'm sure I'll give it a serious shot eventually... in about 3 years once they work all the Poettering kinks out, just like PulseAudio. They're doing some cool things with cgroups and stuff, so I hope it gets there eventually.

cetra3 8y ago
I second using runit. We use runit to be able to use the same service definitions inside docker, on a VM or bare metal. If you've ever tried to use systemd inside docker to bring up a couple of services, you would know the hoops you have to jump through to get it working. (I understand that docker wasn't invented to run multiple services in the one container, but sometimes it can't be avoided and simplifies app deployment vastly I.e, using CI to test your service actually starts up as per its definition: just run up a quick docker image with runit and a service definition file)
- pestaa 8y ago
  I've only seen supervisord as the root process in multi-purpose containers. Is there any significant gain to using systemd instead?
  - cetra3 8y ago
    If you use systemd, you can use standard packages from your distro to run up services inside a container. That's basically the only reason I considered it.
8y ago

Lazare 8y ago

> When Lennart Poettering started thinking about the problem, he first looked at Upstart, which was an event-based system that was still based around scripts, but he concluded that he could do a better job. I think I'd like systemd more if I had more confidence that he had done a better job. I never really understood why Upstart didn't get more traction; was it just the Canonical backing made other distros avoid it, or did it have drawbacks I never ran across?

na85 8y ago
Indeed. Early systemd was a fucking dumpster fire and part of the reason I no longer use Linux.
- danbolt 8y ago
  What do you use nowadays? Do you feel it addresses things better?
  - dijit 8y ago
    Can’t speak to the parent but I have a similar feeling. Personally I’ve been using openbsd quite heavily for personal projects, for work my team transitioned from RHEL6 to FreeBSD- and while it has quirks (mostly on installation of software) it is incredibly stable. It’s a shame my chosen cloud provider doesn’t treat it like a first class citizen, but that’s fine since the community projects seem to work on making good images. I still use Linux on my desktop and laptop; since I feel like the design of systemd is more suited to those roles (systemd’s design in general feels like windows service activation) but I’ve been having issues that seem to be related to systemd. So it’s not winning me over.
    - busterarm 8y ago
      Ditto. I've switched to OpenBSD for nearly all personal work. We're still on RHEL/CentOS at work though. I haven't figured out how to get xmonad running on OpenBSD yet or my workstation at work would be on it too.

anfilt 8y ago

Systemd just makes things worse. I remeber seeing someone post one of the simplest init process one could write. It was only about 100 lines of C.

geofft 8y ago
It's much less than that, but it's sort of an uninteresting demo I think - you could pretty easily write an init process that just spawns systemd in a container, for instance, and run the whole system in a container. Or write an init process that spawns systemd as pid 2, runs prctl(PR_SET_CHILD_SUBREAPER) to make pid 2 responsible for reaping instead of pid 1, and then sits around carefully doing nothing. If systemd crashes, whatever, hopefully sshd etc. are still up. Of course, if you do that then the question is why even bother with an init process. Patch the kernel to not treat pid 1 specially. If a process gets orphaned, don't reparent it to init, just reparent it to nonexistent pid 0. Have the kernel deal with the thing that the 10 lines of C would be doing (waiting on processes to terminate). I think there are meaningful complaints about systemd but the fact that it's a complicated pid 1 is not one of those. If it were actually a problem someone would have written and productionalized one of the above two approaches.
- anfilt 8y ago
  It was code that even let one make simple files to start up one shot process or daemons. If a long running daemon died it would restart it. Plus it handled the getting the status of orphaned processes, and few other things. Really the biggest issue with systemd is is scope creep scope creep means more and more packages will have systemd as a hard dependency. Like there some aspects of gnome that will not work without systemd, but at least it's not entirely a hard dependency just means you lose features unless you patch gnome.
  - geggam 8y ago
    Yet OpenBSD has Gnome without systemd. So it is possible.
    - 8y ago

djsumdog 8y ago

I hate how he talks about knee-jerk reactions to change. because I don't think that's what's going on here. I remember I first saw systemd on a release of opensuse. I didn't think anything of it at first, except that I didn't really like the command line interface (systemctl) and I found the flags and options cumbersome. I often see new software in new releases (including the old HAL/DBUS layer) and didn't have the same reactions (although HAL had a lot of issues and was later removed or merged into dbus). I've seen the BSD talk on this and I agree, having a system layer is helpful. It'd be nice if it was plugable, NetworkManager (or others that have some standard messages you can send/get via dbus), consolekit OR logind, etc. systemd does make it nice that I only have to write startup/shutdown scripts once for each distro, but I'm not happy with the layout of target files, the way mounts are handled, some of the weird race conditions I've found between systemd mount targets and fstab, etc. systemd is modular, but the modules are still all part of the whole and are not easily replaceable. The same can be said when Docker went to a modeler refactor, but there are alternative implementations of the entire docker engine. Every attempt to create alternative implementations of systemd have eventually gone unmaintained because systemd keeps getting more and more complex and engulfing more systems. If it wasn't for distros like Void, Gentoo, Alpine, Slackware, et. al, we'd no longer have a choice at all. There would be some things that simply couldn't be deployed on embedded systems because all of the dbus shims just wouldn't exist. It's not that people are opposed to change, it's that there are legit concerns about some of the ways systemd works and is implemented, and the way it's been ham-fisted as a political move in a lot of ways. Honestly, I don't think it will matter in a few years. I think the way things are going, eventually all services will be hosted via docker containers and it will be much easier to make Linux distros that have a tiny init layer that just launches a docker daemon and services. RacherOS already does this, with the init process being a container, which can be uses to start up shell environment containers and other service containers.

lsc 8y ago
>It's not that people are opposed to change, it's that there are legit concerns about some of the ways systemd works and is implemented, and the way it's been ham-fisted as a political move in a lot of ways. I personally think the industry needs a lot more resistance to change when it comes to interfaces and other things humans have to understand. I mean, I'm not talking about systemd in particular; I'm talking about in general about how interfaces change over time and people don't seem to take into account the cognitive costs of that change. Sure, ss is better than netstat and IP is better than ifconfig... but how much of that 'better' could you have done in a way that didn't toss away the historical knowledge so many people have of those tools? And really, sysadmin tools are the least of it; I mean, they are operated by professionals, so if you want to pay for retraining (or pay the costs associated with there being fewer of us) People change customer facing interfaces to no benefit all the time, forcing people who are trying to do other things to put effort into re-learning their interface. I mean, my point is that interface changes are expensive, and should not be undertaken without a really good argument that they bring more benefit than the cost of retraining.

mantlepro 8y ago

A well-written, thoughtful article, worth the read. > Nerds have a complicated relationship to change; it's awesome when we are the ones creating the change, but it's untrustworthy when it comes from outside.

mantlepro 8y ago
Yikes! How in the world did I lose 2 of my 3 karma points on this?! Somebody please explain and upvote? Linux user since the 90s but totally new to Hacker News. I'm good with constructive criticism, opinions, etc. but not a fan of downvotes without explanation.

stevefan1999 8y ago

I reckon there's a SystemD bug that didn't mount LVM volumes correctly, some has lost data that's real tragedy

JoshTriplett 8y ago
Surely if there's an issue that serious, you have a bug report number?
- shakna 8y ago
  Here you go: [0], from 2017. [0] https://github.com/systemd/systemd/issues/6066 https://github.com/systemd/systemd/issues/6066
  - JoshTriplett 8y ago
    So, fstab says to mount a filesystem, and systemd mounts the filesystem. The reporter explicitly says they want the filesystem mounted at boot time. The response is correct: there's no such thing as "mounted at boot time but not 'after system boot process completed'", because there's no magic "after system boot process completed" point at which that behavior could change. When the device becomes available it gets mounted. sysvinit and typical distribution init scripts couldn't implement the behavior seemingly desired by the user, either.
    - shakna 8y ago
      ... All I did was find the issue number. The ticket follows the sensible approach: Document it, and close it as expected behaviour, and it did lead eventually to an LVM issue that did require a fix [0]. systemd did good here, above board, but there was a faff about it, because the expected behaviour of the audience differed from the reality of the situation. [0] https://github.com/systemd/systemd/pull/6174 https://github.com/systemd/systemd/pull/6174
  - stevefan1999 8y ago
    plus this: https://github.com/systemd/systemd/issues/11255 https://github.com/systemd/systemd/issues/11255,

voltagex_ 8y ago

I'd recommend everyone watch the video: https://www.youtube.com/watch?v=o_AIw9bGogo https://www.youtube.com/watch?v=o_AIw9bGogo

bitwize 8y ago

How about no? How about you not lecture to me that I only hate systemd because I'm just butthurt that Linux is changing, and I need to get with the program? Because that's how people who want to wrest control talk. It's like telling people that the only reason why they dislike Ghostbusters (2016) is because they're misogynists who feel threatened by strong women. Who are you to tell me how I feel and why I feel that way? I really don't like systemd, but I'm glad that it exists. Because more free software is better than less free software. I just don't want to use it on the systems I run. And I don't want the entire Linux ecosystem to hard-depend on it. Lennart can implement all the cockamamie ideas he wants; as a software developer that's his right, and as a software developer myself I would be remiss in defending software freedom if I did not acknowledge and support his exercise of that right. Where I draw the line is the subsequent push to ensure that the Linux ecosystem breaks without systemd.

dingo_bat 8y ago

My only issue is the binary log files. When you're writing out the logs to disk, just make sure it is in text! How hard can it be?? It creates unnecessary friction when dealing with logs from a crashed system.

XorNot 8y ago
Does this ever actually come up for all the people complaining about it? Where are these people working that they're not willing to install rsyslog which reads journald logs directly and will write plaintext, but who have this problem enough to warrant all the complaining? Like, previously to have any logs at all you had to have a syslog daemon. This is not a new situation.
- inferiorhuman 8y ago
  > Like, previously to have any logs at all you had to have a syslog daemon. This is not a new situation. What's new is that now more and more-difficult-to-debug pieces are required. For what benefit?
- yarrel 8y ago
  This is a "now you have two problems" solution.

Karrot_Kream 8y ago

We specifically use runit because we've had systemd issues in the past. Systemd may have matured now but at this point we're moving over to containers anyway, so it doesn't matter.

makz 8y ago

As a conspiracy theorist, I'm going to say that systemd is aimed towards destroying Linux and everything it represents. They want to push the cloud, to better control us, take away power from us, leave us at their mercy. The next step is, when everything that matters is on the cloud, they are going to replace Linux with something else that only they know how it works on the inside, and that's going to be the end of it, we'll be left only with paywalled APIs and services, but they'll say it's better for everyone and all of that. It's sad.

newnewpdro 8y ago
You're confusing systemd with docker and kubernetes.
- busterarm 8y ago
  I hate how accurate this is. The more I work on Kubernetes implementations, the more I feel like it's a tool that 99% of companies can't effectively operate. The attitude is very reminiscent of the mainframe era where ops didn't want mere users touching their precious systems. I feel like as an industry we're wasting tons of cycles solving the wrong problem.

squirrelicus 8y ago

I hate how the Unix philosophy argument is boiled down to "one vs many" binaries. It's about independence to the degree of reusability. There's nothing in systemd suite that's useful unless you're two feet in. You can say things about streams and text and binary formats, but those are distractions. Systemd tooling is tightly coupled to each other. It's a huge codebase, a huge system that shoddily replaces that which was historically done with less. And it's the biggest reason I don't run a Linux workstation anymore.

progman 8y ago
Basically said, systemd follows quite the opposite of the KISS principle which made Unix and Linux so great.

jiqiren 8y ago

My impression of systemd: shoddy version of launchd.

myrandomcomment 8y ago

I hate systemd with a passion, however I understand the reason for it. I think that systemd is a bit Franklinstien by the nature of having to pull udev and d-bus into it. So I guess it is a victim of having to deal with the system it is on. Oh well. I think the only Unix system to get this right was likely Solaris via SMF. https://en.m.wikipedia.org/wiki/Service_Management_Facility https://en.m.wikipedia.org/wiki/Service_Management_Facility WindowsNT also did it correctly.

j16sdiz 8y ago

I hate parallel service start up on server. Yes, it is faster. But when it breaks because bad dependency, it is hard to reason the breakage on a initrd shell on a small remote console. I value debuggability, reliablity and manual fallback options on server.

kakwa_ 8y ago
I hate writing systemd .service files. I'm always confused with the Type (should it be simple? forking?) and my program ends-up starting but not stopping or things like that. I know I lack the knowledge to write a good .service file in one go, as I write one every other month at best and forget all the details in between. It would be the same with a traditional init scripts. However, with init scripts, the way to debug is obvious: just add a '-x' somewhere and here we go, you have a detailed execution. I don't know the equivalent with systemd and I end-up bashing my head against the wall trying everything blindly.

throw2016 8y ago

The author goes into 'contempt culture' which is valid in any group of people ie 'PHP haters' but then tries to slip through the old trope of 'resistance to change' which is a clever bait and switch that positions change as 'always good' and refashions dissenters as anti-change which is a time tested PR strategy to push through unpopular changes that benefit a few by positioning any 'dissent' as 'anti change' and thus avoiding the meat of the discussion - the change in question. We can agree on the problem and then on the solution, but any proposed solution can't just pass through 'without scrutiny' because a problem exists. Systemd proponents since the beginning have positioned any criticism as 'motivated', 'greybeards', 'haters' or anti-change which tells you that there is no space for discussion here, only 'acceptance'. Can there be any legitimate criticism of systemd without this kind of politics, hypersensitivity and now posing as victims?

aidenn0 8y ago
Also the whole "One solution for people who don't like it is to create their own alternative; that is a good way to find out just how much fun that task is." This seems to imply that those criticizing it didn't have alternatives, which is just not true. However it is so monolithic that as soon as a tiny part is depended on by something else (e.g. gnome) alternatives quickly become a non-starter. systemd seems to be getting rewarded by not playing well with others which is something that plays strongly into a lot of people's dislike with it.

jimrandomh 8y ago

Back in 2016, systemd started killing user processes on logout (rather than send them the SIGHUP signal, as POSIX says should happen). This caused problems for programs like nohup, screen and tmux, which deliberately keep running. Systemd's response was to say that they should incorporate systemd's library, and use systemd's new daemonization API. As far as I know, none of them did. Two years later, you can find hundreds of support requests across the internet, from frustrated users who are having their sessions killed by systemd. Bugs are annoying, but that's life. On the other hand, when you're an impacted user who's lost work, and researching the bug leads you to a years-old discussion in which someone is actively denying that the bug exists and refusing to fix it, that's infuriating. I don't think systemd's developers deserve the trust that maintaining a core piece of infrastructure requires; they don't seem to care enough about whether they've broken things.

MereInterest 8y ago
Absolutely. I can understand implementing this feature for some special cases, like containers that should clear all hint of a user away on log off. It should never have been the default, and breaks an entire category of software. In my standard .bashrc file, I have the following snippet to warn me if I am on a system with that stupid setting enabled. if which loginctl > /dev/null && loginctl >& /dev/null; then if loginctl show-user | grep KillUserProcesses | grep -q yes; then echo "systemd is set to kill user processes on logoff" echo "This will break screen, tmux, emacs --daemon, nohup, etc" echo "Tell the sysadmin to set KillUserProcesses=no in /etc/systemd/login.conf" fi fi
- m45t3r 8y ago
  Thanks, now I know why Emacs daemon keeps delaying my restarts in the system (just discovered that NixOS defaults KillUserProcesses to false). Turning this on to true, for me it does no make sense to a user service (yeah, I run emacs as a user's systemd service) to keep running after I logout of my system. P.S.: And the fact that for some people this behavior makes sense is why I think Lenart decision to put this as an option makes sense.

m463 8y ago

I believe systemd solves a problem or two. However, I believe the implementation is not great. A couple specific observations: * lots of binaries. /lib/systemd, the logs. This more than anything else is a tragedy. Binaries get in the way of viewing, diagnosing, understanding and changing things. It seems like premature optimization to me - I don't think speed requires it. I think the majority are completely unnecessary. * copying from launchd? Commercial software vendors ship binaries. They are aligned with them. They resist disclosure, which protects from reverse engineering, lawsuits and user modifications. Does that align with linux? * the config files are a disorganized mess. It is like /etc/init.d but poorly understood and then /etc/rc.d was lumped into the same directory and organization became chaos. I looked at one system here and in /etc/systemd there are 16 directories, 23 symbolic links and just 11 files. Try unraveling the interdependencies. additionally the config files (which mimic windows config files) lack depth. Simple config files can have benefits, say if a GUI read and wrote them to change settings, but I don't think that applies here. What I noticed is that they cannot do anything, so additional logic requires an intermediate script or compiled binary elsewhere. * it is responsible for too many functions. I bet it didn't solve so many in its first iteration, but sucked in more over time. It reminds me of the accounting program that grew to rule everything in Tron. I wonder if these problems could be resolved. It's hard to remove complexity and add elegance later.

atombender 8y ago
Launchd has been open source (Apache license) pretty much from the beginning: https://wiki.freebsd.org/launchd https://wiki.freebsd.org/launchd
mixmastamyk 8y ago
Systemd as the new MCP, interesting. “Sark, prepare that daemon for immediate deresolution.”
poettering 8y ago
I kinda like how a good chunk of the community complains we are "too monolithic", and the other side of the community complains we have "too many binaries". We can't win with you guys, can we? Lennart

mirimir 8y ago

Damn. So I'm just a simple Debian user. Systemd showed up, so I learned as much as I needed. And in some ways, I found it easier to use than init scripts. Now I've been using Docker. And there is no systemd in containers. But of course, why would you want that? But the problem is that many packages now depend on systemd. Or at least, useful features do. So I'm back to fighting with scripts. It's funny, no?

inferiorhuman 8y ago
> And there is no systemd in containers. But of course, why would you want that? I knew a guy who invented an orchestration system that involved shoving an init process into docker containers. Much sadness ensued.
- mirimir 8y ago
  Are you referring to dumb-init? Is there a better way to reap zombies?
  - inferiorhuman 8y ago
    > Are you referring to dumb-init? Nope.
tomjakubowski 8y ago
Maybe you wouldn't want systemd, but a basic init system as PID 1 is wise for a Docker container. See https://github.com/Yelp/dumb-init https://github.com/Yelp/dumb-init
- mirimir 8y ago
  Wow. Thanks. I've been wondering about all those zombies.
  - cyphar 8y ago
    I would suggest https://github.com/openSUSE/catatonit https://github.com/openSUSE/catatonit (self-plug -- I wrote it). dumb-init (and tini which is what "docker run --init" uses) don't handle zombies in the most ideal fashion possible, catatonit does. And it does so in less code.

Kivutar 8y ago

Some years ago, I was developing a Linux distro for embedded systems. Some of these embedded systems come with old kernels, and you will never get a recent kernel for them. Systemd is written in C and relies on kernel headers. Which means that they need to have a layer that deals with different versions of the kernel headers. I remember it was headers about network. At the time I was working on this, I had to contribute to systemd to add support for my kernel version. Got a few patches accepted in their master. And things were working fine for a while. Until they suddenly decided to drop <= 3.10 kernel support. Yep. Afterwhat I had to maintain patch sets for systemd, which is a great amount of job, given that my upstream was updating systemd quite often. Now my good people, explain me how such a mess could have happen with a previous init based on shell?

CaliforniaKarl 8y ago
I'm curious: Did systemd drop 3.10 support before the kernel developers stopped supporting it (in 2017, see http://lkml.iu.edu/hypermail/linux/kernel/1711.0/03167.html http://lkml.iu.edu/hypermail/linux/kernel/1711.0/03167.html), or before that? If it was around the same time, then I could understand dropping support, since even the kernel developers had decided to move on.
jononor 8y ago
You could have stayed on an older systemd version? Preferably one used by a major distro, such that someone else might backport security fixes etc for you. Anyway the root cause here is bad vendor BSPs being stuck on old Linux versions.
teddyh 8y ago
GNU libc 2.26 (released 2017-08-02) requires Linux 3.2 or newer. It’s not like systemd is alone in requiring recent kernels.

aidenn0 8y ago

One of my "favorite" corner cases caused by systemd was that if an NIS user tried to login to the system, it would cause my X session to restart: 1. They changed the defaults for systemd-logind to disallow network access 2. When systemd-logind attempts to send a message to the NIS server, it instead hangs (or has a timeout greater than the systemd watchdog anyways) 3. The watchdog timer causes all systemd-logind services to be restart, which then tears down all of the child processes. My X session is a child process.

RemarkableMan 8y ago

I just don’t have time to beta test an init system with more LoC than the 2.6 kernel plus GCC 4. For now Devuan is where we landed but moving entirely away from Linux as Redhat moves system d into the kernel is the overall plan.

pgtan 8y ago

systemd broke UNIX. Everything that worked before, now is a mine field. Last week I had a SLES virtual machine refusing to boot, no network, just emergency shell. The reason was a forgotten line in fstab (without nofail option of course), so systemd couldn't mount some file system. Coming from Gentoo, this is just unbelievable.

the_why_of_y 8y ago
Systemd correctly implements the "noauto" and "nofail" keywords in fstab(5) that were added in 2008 in the way they have been documented since 2010: if neither of these is specified for a filesystem, then that file system will be mounted at boot time and a failed attempt to mount that filesystem implies a boot failure. Util-linux-ng 2.14 Release Notes (09-Jun-2008) ============================================== mount(8) supports new "nofail" mount option. https://github.com/karelzak/util-linux/commit/abe3d704b6aeb6b82ff32e8599edb56525cdfd72 https://github.com/karelzak/util-linux/commit/abe3d704b6aeb6... The fact that previous init scripts gladly ignored these failures does not reflect negatively on systemd but on those init scripts. Since 2011, FreeBSD has the same options with the same behavior, except that "nofail" is called "failok". https://svnweb.freebsd.org/base?view=revision&revision=222832 https://svnweb.freebsd.org/base?view=revision&revision=22283... Add a special mount option "failok" to indicate that the administrator wants the system to proceed to boot without bailing out into single user mode, even when the file system can not be successfully mounted. By your argument, FreeBSD broke UNIX.
zlynx 8y ago
systemd refusing to boot without required drives is actually a favorite feature of mine. Ever since I had an RH AS server boot without /var mounted. It was truly lovely with a gigabyte of trash filling up / then trying to decide if it was worth merging data or just throwing it out.

pdkl95 8y ago

> This argument, he said, seems to be predicated on the notion that systemd is a single, monolithic binary. Can we please stop misrepresenting the complaints against systemd? The only time I ever hear this "monolithic binary" argument is from systemd advocates. The actual complaint is about tightly coupling important features together. Not only does this make it difficult (often impossible) to replace individual components, when tight coupling happens at the (internal) protocol level, any replacement component ne4cessarily hast to implement a bunch of (sometimes unwanted) systemd baggage. Busybox implements all of its features in single monolithic binary, but it isn't a monolithic design that tightly couples those specific components together. Replacing one of busybox's components is often as simple as removing busybox's symlink and installing the replacement. This isn't even a "Unix philosophy" issue. Even inexperienced designers shouldn't have as hard time Understanding why systemd is a monolithic design but busybox isn't.

chris_wot 8y ago
What baggage are you specifically referring to?
- lmm 8y ago
  It runs its own logging system with non-standard interfaces and formats. It runs its own DNS resolver with non-standard behaviour. It maintains compatibility only with a narrow range of udev versions, which in turn maintain compatibility only with a narrow range of kernel versions. And all the d-bus interfaces between these pieces may change at any point without notice. So you can't replace any piece of it, because even if you provide your own component that implements one of the systemd d-bus interfaces, you've got no forward compatibility.
  - pas 8y ago
    If there was a serious effort to replace/port parts of it, the needed internal APIs can be stabilized ( https://www.freedesktop.org/wiki/Software/systemd/InterfacePortabilityAndStabilityChart/ https://www.freedesktop.org/wiki/Software/systemd/InterfaceP... ).

ChuckMcM 8y ago

From the article -- Windows NT had a strong service model from the beginning, he said, and Mac OS has one now in the form of launchd. Other systems had to play a catch-up game to get there. Reminds me of the saying, "and if your friends all jumped off a cliff, would you jump off too?" which exasperated parents use to council their children that emulating others needs to be done responsibly.

darkpuma 8y ago
There is a big difference here. To continue with the cliff analogy: Launchd ran fast enough to clear the cliff and was rewarded by a refreshing dive into the cool water; it was implemented in a competent manner and consequently didn't cause people to become upset. Systemd on the other-hand can't run without stumbling, it hobbled off the cliff and bounced off every rock on the way down. It made lots of people pissed off because it interfered with their routines. If systemd didn't cause problems for people, virtually no user would even know what it was, and consequently, virtually no user would hate it. If you ask your average OSX user what launchd is, they'll come up with a blank because launchd isn't causing trouble for them. If you ask your average linux desktop user what systemd is, a large portion of them will recognize the name, and many of them will chew your ear off out of frustration. At the end of the day it's not about "adherence to philosophies" or any similarly dubious concepts. Whether or not software is hated is ultimately determined by whether or not it causes problems for people.
JdeBP 8y ago
That's not the problem. Windows NT did have that right from the beginning. The problem is the second sentence, because it ignores that so too did Unix from the same timeframe. AT&T System 5 Release 4 had the Service Access Facility from 1988. AIX had the System Resource Controller from version 3.2. Other systems were not playing catch-up. They had this stuff too, at around the same time. Benno Rice's explanation is actually ahistorical here. What actually happened is that the world spent a lot of time "cloning" existing Unix softwares during that time period, for reasons that we all known, and a lot of the time the clones were behind the times and did not catch up. By the time that Miquel van Smoorenburg cloned AT&T Unix init and rc for Minix, for example, what xe was cloning had already become years out of date. * http://jdebp.eu./FGA/inittab-getty-is-history.html http://jdebp.eu./FGA/inittab-getty-is-history.html

chroma 8y ago

It is my opinion that Lennart Poettering is the software equivalent of Thomas Midgley Jr., the inventor of both leaded gasoline and chlorofluorocarbons.[1] This is the person behind PulseAudio, Avahi, and systemd. In any sane world, these software projects would have been stillborn. Sensible alternatives would have outcompeted them. Instead, Red Hat did their best to foist them upon the community. They promised that after the initial adoption, all the wrinkles would be ironed out. All three times this was not true. I hate to appear so cynical, and I'm certain that Red Hat didn't intentionally pursue this strategy, but it sure is convenient for them that the harder Linux is to use, the more they can sell support contracts. Not so convenient for people who actually want to use Linux. 1. https://en.wikipedia.org/wiki/Thomas_Midgley_Jr https://en.wikipedia.org/wiki/Thomas_Midgley_Jr.

darkpuma 8y ago
You might be giving Lennart too much credit. As the author of the article points out, Lennart didn't invent PulseAudio and systemd so much as he ripped them off (poorly) from coreaudio and launchd. To compare him to any inventor seems too flattering.
- mixmastamyk 8y ago
  Also not a huge fan but it works better than a bunch of shell scripts.
  - flukus 8y ago
    Don't you still have to resort to shell scripts for anything slightly complicated (mounting a drive on a specific network connection) and just have systemd invoke them?
    - arminiusreturns 8y ago
      So far this is a particular thing I have run into. DevOps (yelling): no more shell scripts Devs: systemd fstab is all kinds of fucked for nfsv4 Devs: Writes super complicated script to mount Me/DevOps: Makes ansible script to make systemd mount paired with service that is basically a wrapper for... a shell script...

fmajid 8y ago

Farce, not tragedy

yarrel 8y ago

First as tragedy...

_pmf_ 8y ago

If the current state is the tragedy, I'm terrified of the farce. My favorite systemdisms: * (already mentioned several times above; sorry) even detached and nohup'd processes will be killed if the systemd context that transitively led to the creation of the process is killed: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394 * mounts being in a separate "mount" namespace that is not accessible normally: http://www.volkerschatz.com/unix/advmount.html http://www.volkerschatz.com/unix/advmount.html

ComputerGuru 8y ago

For all those lamenting the death of standards, change for the sake of change, system breakage, bloat, and lack of user control: give FreeBSD a try (or any BSD, really).

progman 8y ago

I like to share the following links to those who also don't like systemd. http://without-systemd.org http://without-systemd.org Devuan (Debian without systemd): http://devuan.org http://devuan.org More Linux Distros without systemd: http://without-systemd.org/wiki/index.php/Linux_distributions_without_systemd http://without-systemd.org/wiki/index.php/Linux_distribution...

fanf2 8y ago
Debian Stretch works nicely for me without systemd

arminiusreturns 8y ago

I particularly like how it ripped out the old logging system but completely failed to replace it properly. Local might be ok, but shipping to a central log server? Just save yourself the headache and use syslog or some other shipper like filebeat, etc. How can something that can't even do logging properly suddenly become the defacto standard?

newnewpdro 8y ago
> I still think NSA used RedHat to push systemd for potentially nefarious reasons. Wow, just wow. Do you have any idea how insulting this is to the folks dedicating their lives to writing this GPL software? You should try having an actual conversation with the authors some time. These are real people and long-time GNU/Linux users and advocates you're implicitly accusing of being NSA collaborators.
- arminiusreturns 8y ago
  RedHats collaboration with NSA is well documented. I think given what we know of past actions it's at best extremely naive to not at least entertain it as a possibility. Linus himself semi-openly admits he's been approached... https://www.youtube.com/watch?v=7gRsgkdfYJ8 https://www.youtube.com/watch?v=7gRsgkdfYJ8

CarlFK 8y ago

Notable quotes 22:10 "Unix is dead." 25:28 "Change is awesome when we are the ones doing it. 27:20 "No one needed to send death threats over a piece of software." 27:30 "Contempt isn't cool."

thekingofh 8y ago

Can I easily replace systemd on a distro that came with it? I wouldn't consider myself a Linux expert, but I got to a point where I could get around and started to understand the different init runlevels and how to set them, then everything was ripped out from under me and I no longer understand how to work everything anymore. Perhaps Linux boots faster now, I don't know, but it was really very frustrating I couldn't find a major user-friendly distro that worked the way I was used to.

JdeBP 8y ago
The writing was on the wall for run-levels for a long time. Longer than Linux has existed, in fact. * http://jdebp.eu./FGA/run-levels-are-history.html http://jdebp.eu./FGA/run-levels-are-history.html

gerbilly 8y ago

There are parallels with city planning. systemd reminds me of the efforts in some cities at clearing 'slums' to replace them with 'modern' apartment blocks.[1] sytemd seems like a similar technocratic, modernist effort aimed at UNIX. Operating systems are kind of like cities. One the surface they are often inefficient and messy, and people often wish they could come in and make them logical and rational, but people live in those cities and neighbourhoods, and sometimes the postbox is where it is for a reason. If Jane Jacobs[2] were around, she could write a book called: "The Death and Life of Great American Operating Systems" [1] See: https://en.wikipedia.org/wiki/Pruitt%E2%80%93Igoe https://en.wikipedia.org/wiki/Pruitt%E2%80%93Igoe, https://en.wikipedia.org/wiki/Quarry_Hill,_Leeds https://en.wikipedia.org/wiki/Quarry_Hill,_Leeds, https://www.youtube.com/watch?time_continue=862&v=Dxr2tEGlWUE https://www.youtube.com/watch?time_continue=862&v=Dxr2tEGlWU... [2] https://www.citymetric.com/fabric/against-modernist-nightmare-legacy-urbanist-jane-jacobs-2185 https://www.citymetric.com/fabric/against-modernist-nightmar...

dekhn 8y ago

I've used linux systems since '93. I waited until the last minute to adopt systemd. So far, for me at least, it's been great. It's a lot easier to express service dependencies, find problems when they occur. I hear a lot of hate, but in all, systemd seems to solve a ton of problems in a single place, which is something I think linux needed for a while.

jiveturkey 8y ago

more like abomination

merb 8y ago

a lot of people shit on systemd because of the "SIGHUP"-"bug" however what they don't realize is that there are more use cases where logging out a user SHOULD kill all users processes. I mean without that behavior linux is not a multi user system. The default is sane, however the lack of "permissions" in this kind is also a little bit bad, however since it is runtime configurable this should be enforced by the distribution, i.e. when installing the distri there should be a switch "this system is only used by a single user". it's basically the same as windows uac. while it looks bad for a single user, it's not that bad when you see computers as a tool which can have more than one user, with more than one "privilege". system just shows that linux is by far not ready for the masses cause a lot of behavior is undefined, like the uid/username stuff. usernames can be created as you wish, but most programs will behave broken if you create usernames that start with a number, it's a security nightmare. and the problem that there is no real process to actually get ALL people together who can solve this, it will never be fixed. there is only the posix standard but not everything applies to linux and not everything is a good idea in the sense of linux. systemd actually makes a lot of things in the right way, but of course it's not perfect and probably at a certain point in time people would come up with something better.