22 ms·

The bad side of systemd: two recent systemd failures

121 points by lreeves 12y ago

ultramancool 12y ago
I'm no fan of systemd, but do we really need to be nitpicking its every failing? The alternatives are certainly not perfect either. If you don't like it, don't use it. Many distros are still committed to supporting other init systems along side systemd and many BSDs have a very good philosophy with things like this and may be worth a look too. Can we just agree to move on? I'm sick of the endless banter. If you like systemd, use it, if you don't like it, don't. If you think it needs more time to mature, stay away from it for a while and experiment occasionally. That's my plan at least.
- broodbucket 12y ago
  Within all the noise about systemd I think posts like this are quite beneficial. Instead of discussing politics or throwing out buzzwords, it presents a specific technical shortcoming. Positive or negative, these are the types of things that people should be reading about in relation to systemd. The problem is that people are deciding whether or not they like systemd based on "do you like Lennart" or "do you like The UNIX Philosophy", because that's what people are making noise about.
  - Mister_Snuggles 12y ago
    As I said in another comment, I really wish it went into more detail on the actual problem. The problems with journalctl are usability ones, which doesn't make them any less of a problem, but doesn't point to an underlying technical failure either. The problems with systemd are basically stated as "a Fedora upgrade causes systemd to go bad." Is this a systemd problem or a Fedora problem?
    - tankenmate 12y ago
      Some anti-fragility on the part of both Fedora and systemd would help here. Maybe a statically compiled init would be a start?
      - vezzy-fnord 12y ago
        systemd does not support static linking due to its use of NSS. See: http://lists.freedesktop.org/archives/systemd-devel/2014-March/017493.html http://lists.freedesktop.org/archives/systemd-devel/2014-Mar...
        ultramancool 12y ago
        > Only the most trivial libraries actually support it. ... what kind of nonsense is that? Almost every library supports static linking to a basic degree at least. Heck, I statically link giant things like Qt and Boost every day at work.
        vodik 12y ago
        Well the core of the issue here is you can't really staticly link against a library that depends on dlopen/dlsym (nss system is built around this). I mean, you could, but now your staticly linked binary still depends on other dlsym'd code. So whats the point? (Not trying to make an argument that this is good or bad behaviour)
        ultramancool 12y ago
        I understand, but the question then becomes why your library depends on dlopen and why you don't offer an alternative static method as Qt's "plugins" do. Doesn't seem like good design on the part of the NSS devs to me, and probably a poor choice of library on the part of the systemd devs to me. But maybe I'm missing something.
        the_why_of_y 12y ago
        The library that depends on dlopen is glibc.
    - digi_owl 12y ago
      Given that Fedora and systemd has been developing in lockstep over the last few years, one would think they worked well together...
      - Mister_Snuggles 12y ago
        I'd hope that any distribution would be packaged to work well with it's init system, regardless of which init system that is. After all, the whole point of a distribution is to have a core set of software[0] that works together reasonably well. [0] With the number of different packages in a modern Linux, I don't expect everything in a distributions repository to work perfectly, but core stuff (like the stuff that gets you from a cold boot to a login prompt, etc) should be rock solid.
  - felixgallo 12y ago
    those are completely valid criteria. What's wrong with wanting your system to follow the closest thing to a known axiom in operational computer science that has ever existed?
  - noselasd 12y ago
    I don't really have a good or bad opinion about systemd, to me the init system and things around it are just pieces that needs to work. The points discussed here are serious, should be filed as bug reports, and should be acted on by the developers. But the article also just sounds like systemd bashing, software has bugs and poor thought out solutions - which can be fixed or improved - it's just code. I could probably have written a similar article about many, many Fedora (and Debian) upgrades over the course of a decade or so about core components such as distro upgrades making the init system crashing, failing to start services, glibc upgrades screwing up servers, grub upgrades calling for a 2 hour car drive to fix and so on. But I didn't, I either filed a bug report, fixed the code, or was just too plain busy getting things back to working.
- vezzy-fnord 12y ago
  Given the huge ambitions of the project maintainers and how everyone's rushing to adopt it even amidst sloppy integration work, yes, the criticism is worth it. "The alternatives are certainly not perfect" is no consolation. At all. Especially considering the vast majority were never fairly considered to begin with.
  - ultramancool 12y ago
    I had the same sort of annoyances, but instead of whining, I simply ditched it for now. I don't see why people have such a problem doing this.
- e40 12y ago
  if you like systemd, use it, if you don't like it, don't. That is so naive. No one chooses to use systemd. People choose a distro. It's being forced on long time users of distros without any of the users having a say. Yeah, I can switch from one distro to another. That's a choice I have. But it's not so simple as you make it sound.
  - deleted 12y ago
    [deleted]
  - ansible 12y ago
    Yeah, I can switch from one distro to another. That's a choice I have. But it's not so simple as you make it sound. Switching distros is like moving to a new city. If you stay in the same state (or at least in the same country) it isn't too disorienting. But switching to something completely different (like on another continent, or going from an RPM-based to deb-based distro) will require some significant effort to adapt to the new situation. We pick our distro of choice based on a number of factors. Just because it has some downsides doesn't mean we have to move immediately. But that doesn't mean we can't give (constructive!) feedback if there are problems.
  - vidarh 12y ago
    > No one chooses to use systemd. I did. I've been systematically installing systemd on my Debian machines.
  - chimeracoder 12y ago
    > It's being forced on long time users of distros without any of the users having a say. It's being adopted by distributions in accordance with the stated and implied wishes of the developers, maintainers, and users of the distributions. The reason that (e.g.) Arch dropped support for legacy init so quickly was because they literally failed to find anyone who wanted to go through all the work of maintaining the old init system. If none of the users want to do the work, then nothing's being "forced" - that's the way community-organized FOSS projects work![0] Same goes for Debian, who made the switch in accordiance with their normal development process (including a general resolution in support of systemd!). Debian has promised to maintain systemd-shim (supporting other init systems) through at least Jessie, and possibly longer, depending on whether there is enough demand for alternative init systems, and support from developers for maintaining it in future releases. Ironically, the biggest threat to systemd-shim in Jessie's successor are people who claim they want to fork Debian - if all the people who hate systemd leave Debian, then nobody who wants to actually do the work to maintain the alternative init systems will remain with the Debian project[1]. [0] Even then, you can still find an AUR package for it (https://aur.archlinux.org/packages/initscripts-fork/ https://aur.archlinux.org/packages/initscripts-fork/), though as with everything in the AUR, it's subject to much less testing. Note that the package has only 27 votes - this means that only 27 people actually said that they care about moving this package back into the main distro. [1] This may be a bad or a good thing, depending on who you ask.
- zobzu 12y ago
  Not sure if /sbin/init failing is a nitpick. its kind of a catastrophic failure as far operating systems are concerned.
  - digi_owl 12y ago
    Never mind that it results in a zombie state where you still have a shell, but no commands actually work.
- howeyc 12y ago
  > Can we just agree to move on? No, we can't. Sorry but that's just how the world works. Just like every Go language post has "generics or crap" mentioned, every systemd post will have "The Unix Way" mentioned. A price of freedom of speech is having to deal with self-filtering idiotic viewpoints. Worth it in my opinion, but I'm an anti-censorship for EVERYTHING person.
  - ultramancool 12y ago
    I'm not suggesting censorship of any sort. I'm certainly the same way. I'm just suggesting level-headedness and a willingness to take or leave it and move on with life.
    - howeyc 12y ago
      I understand, I also agree with your suggestion. I guess my point is really that even though a nice ideal, based on my experience, likely not attainable.
- rdtsc 12y ago
  > If you don't like it, don't use it. Have you tried removing systemd from latest Ubuntu? I didn't go very well for me. I imagine removing it from RHEL 7 won't go well over either. > I'm sick of the endless banter. Ok but "use something else" or "fork it" is not exactly solving the problem either.
  - digi_owl 12y ago
    And thats the thing. If it was "just a init", it would be just as replaceable as sysv, openrc or runit. But it is not, so it is not. As best i can tell, a major argument for going systemd is that something more user facing (Gnome) depends on a subsystem of systemd, and that in turn requires that systemd is used as init. In other words, systemd as a project is invasive and insidious.
    - EmanueleAina 12y ago
      The people that develop GNOME chose to depend on the D-Bus interfaces exported by some systemd-provided services (eg. logind) because they actually solve some problems they were facing with a clean and robust solution. Yet they made sure that the dependency is optional, to accomodate those that for a reason of another do not use systemd. This means that systemd is not insidious, it's valuable.
  - EmanueleAina 12y ago
    > Have you tried removing systemd from latest Ubuntu? I didn't go very well for me. I imagine removing it from RHEL 7 won't go well over either. Ubuntu is not the Universal Operating System, that's Debian. This means that Ubuntu is definitely opinionated in its software selection, and rightly so. You couldn't replace Upstart and now Ubuntu relies on systemd. If you have the skills to fully evaluate the costs and benefits of mixing and matching random pieces of software, distribution like Debian or Gentoo may be a better fit for you. Indeed, switching to sysvinit in Debian is trivial and the systemd team worked really really hard to make sure the transition is smooth even if one keeps switching back and forth. They should really be praised for the effort they've put in it.
- _cipher_ 12y ago
  > I'm no fan of systemd, but do we really need to be nitpicking its every failing? Yes. When some developers are pushing down your (and mine) throat critical software that sucks _that_ much, that's the least someone can do.
  - EmanueleAina 12y ago
    Some developers wrote some code and published it in a software package under a free software license. Other developers picked up said package and added it to the package collection they are maintaining. You choose to use the package collection maintained by said developers, which means that you trust their technical decisions otherwise you'd be running software from people you don't trust, which is not a great idea. Nobody is pushing software down anyone's throat. Please avoid such hyperbolic mischaracterization, it really does not contribute anything to the discussion.
    - yarrel 12y ago
      > You choose to use the package collection maintained by said developers, which means that you trust their technical decisions This does not however mean that I must silently accept their bad choices. If they were about to step out in front of a bus I wouldn't remain silent because I trust their judgement. Sometimes people you respect excercise poor judgement. This is one of those cases. Feedback is a thing. You're giving it here. Why shouldn't I give it to the distros that certainly are trying to force systemd on me?
      - EmanueleAina 12y ago
        > This does not however mean that I must silently accept their bad choices. But there are proper places to discuss this issue with them, and HN is totally not one of those. > Sometimes people you respect excercise poor judgement. This is one of those cases. Sure, but have you considered that the one exercising poor judgement in this case may be you? I don't know which distro you're using, but Debian has extensively discussed the issue and the first tech CTTE resolution made lots of valid technical points in favour of systemd. For sure it's not an obviously bad choice, and if your maintainers have chosen to ship it they may have valid reasons to do so. Not to mention that's still entirely possible to avoid systemd for whatever reason, at least if you're using Debian it's trivial. > Feedback is a thing. You're giving it here. Why shouldn't I give it to the distros that certainly are trying to force systemd on me? Right, but I'm here trying to give feedback to you for your actions in this thread. You're here giving feedback on HN to people that probably don't read HN, and doing so with a rather inflammatory tone. If you want to give feedback to your distro maintainers, please get in touch with them in the appropriate channels (IRC, mailing lists) and ask them their reasons, but please first check if they haven't already stated those, one can just get annoyed if every person in the world keeps asking stuff that has been answered countless times.
- skywhopper 12y ago
  I think ongoing constructive discussion about the technical and philosophical issues with software design, particularly of open-source software that affects huge numbers of users, is always valuable. The OP here even likes systemd, so the article isn't nitpicking. As others have pointed out, when your init process faults, that's a major problem. In the real world, for the core elements of an operating system, whatever features they have are less important than how well they fail. In this case, the OP has identified a very very bad failure case. More importantly, this report makes concrete the theoretical complaint against systemd that its monolithic design is not just a philosophical issue, but a practical one. The init process needs to be rock solid, but a law of software is that bigger software is less stable. So if the sprawling nature of systemd means it will tend to become less stable over time, then we have a major practical issue that needs to be addressed. This is not a matter of not moving on. Rather, given that systemd has been adopted by several major Linux distributions, it's actually far more important now to criticize its design failures, to draw attention to get things fixed. If "moving on" means we shouldn't critique the software, then how will it ever improve?
- mrweasel 12y ago
  >If you like systemd, use it, if you don't like it, don't. I think people would care less about systemd if that was a realistic choice. I strongly believe that the whole systemd debate would go away if systemd wasn't a dependency of GNOME and other stuff. Personally I have no strong feelings one way or the other, but I do fear that the BSDs are going to be left behind as they to some extend have been piggybacking on Linux in terms of desktop software.
- exabrial 12y ago
  you're missing the point... SystemD is about RedHat trying to take over the entire Linux ecosystem. It's intentionally designed so if you use any part of it (which there are good parts) then you have to use all of it (which there are many bad parts). Oh and conveniently RedHat is the main project maintainers.
XorNot 12y ago
I can't speak to the first problem, but the second is more of "this is strange and I don't care to learn how it works". journalctl --no-pager Problem solved. I mean, you have the command, it's right there in the usage text. Which, awesomely, does invoke a pager when you do journalctl --help so if you're stuck on a 80x25 console (as you might be on a broken system) you can easily actually read the help options.
- masklinn 12y ago
  > I can't speak to the first problem, but the second is more of "this is strange and I don't care to learn how it works". I can't speak for you not speaking to the first problem, but the second is more of "this is strange and I didn't care to read the text" > In the grand systemd tradition, there is no option to control this; all you can do is force journalctl to not use a pager or work out how to change things inside the pager to not do this. (emphasis mine) > Problem solved. I mean, you have the command, it's right there in the usage text. Which, awesomely, does invoke a pager when you do journalctl --help so if you're stuck on a 80x25 console (as you might be on a broken system) you can easily actually read the help options. From reading the text, I'm under the impression that the issue is the mangling of $LESS. One would expect that $LESS is set up how the user wants it, if the user wants -S he'll set that.
  - darklajid 12y ago
    Okay, okay. I understand that the user might be surprised. man journalctl $SYSTEMD_LESS Override the default options passed to less ("FRSXMK"). So - whereever the user defines $LESS I'd add a line a la export SYSTEMD_LESS=$LESS and that should solve the problem nicely, right? Surprising? Not intuitive? Maybe. But easily fixable with a quick peek at the man page.
    - zaphar 12y ago
      So I have no dog in the systemd fight. However. $SYSTEMD_LESS is quite possibly one of the most aggressively stupid ways to force a bad UX on someone. I mean seriously. If you have $LESS defined then you obviously have an expectation that less will behave in a certain way. An expectation that has in all probability built up over years of tweaking your shell. For some other application to ignore that preference and in fact force it's own preference on you and only allow you to use your own preference for how less should work by reading not the docs for less but the docs for an unrelated application? That breaks so many rules it's just sad. Yes it's easily fixable with a quick peek at the man page for systemd which might not be the first thing you check given you've been using less for years. Systemd's manpage is the last thing I would check. First would be what is my $LESS set to? Second would be is less still less? At some point it would occur to me that maybe systemd is being aggresively stupid and check their manpage. But given I have no particular preference I would be inclined to assume it couldn't possibly be systemd's fault. I must have messed something up. Put this minor annoyance in the middle of attempting to debug a segfaulting init? That's what we call a major annoyance.
lstyls 12y ago
How much of the blame should systemd be taking for these bugs? Unless the same segfault is happening on other distros, this seems like a problem in Fedora rather than systemd. I suppose this does give some substance to the systemd criticisms that it is too system-critical for a process that is technically outside of the Linux kernel.
- kosinus 12y ago
  If you're saying, a bug should always be investigated downstream before reporting it upstream, then I agree. If you're saying, there are scenarios where segfaults caused by upstream code are not upstream bugs, then I disagree.
Mister_Snuggles 12y ago
It's interesting to see examples of where systemd falls down, though I'd be interested to hear what the actual problem is. The Fedora bug report and linked page basically say "systemd fell over during an upgrade", but is this really a systemd problem or is it a Fedora problem?
- digi_owl 12y ago
  At this point in time they seem joined at the hip. I think a Pottering retort about Debian users complaining about the size of systemd was that it was introduced over time on Fedora. So if any distro should make systemd shine, it would be Fedora.
- cesarb 12y ago
  It's failing during an upgrade. My guess would be that it's a systemd problem, but one triggered by mixing systemd versions. Probably the systemctl from Fedora 21 does something that the systemd from Fedora 20 doesn't like. If that's the case, the bug might have even already been fixed in systemd upstream, but the fix wasn't applied to the older systemd in Fedora 20. The solution would be to backport the fix to Fedora 20's systemd package, so users who do a "yum update" before upgrading to Fedora 21 would have it. An alternative workaround might be to, immediately after unpacking the new systemd, send a SIGTERM to PID 1. According to http://www.freedesktop.org/software/systemd/man/systemd.html http://www.freedesktop.org/software/systemd/man/systemd.html, that signal directly tells PID 1 to re-exec itself, without using systemctl in the process. That would make PID 1 be the new systemd before anything tries to use the new systemctl.
- siebenmann 12y ago
  Almost any time PID 1 segfaults, it's PID 1's fault. With PID 1 being systemd, that makes it systemd's fault here. Based on gdb stack backtraces in the Fedora bug report and looking at the systemd code, it seems like some sort of memory corruption or overwrite, perhaps a use after free issue. The segfault itself comes from dereferencing a clearly invalid pointer and said pointer was obtained by dereferencing a structure field through another pointer, so you'd get exactly this result if the structure was overwritten with other data at some point. (In my grumpy sysadmin view, it is PID 1's fault even if the distribution is doing odd things around PID 1. Init processes need to be absolutely rock solid and extremely defensively coded, precisely because the world basically dies if they ever fall over.) (I am the author of the original post.)
darklajid 12y ago
The first problem, PID 1 crashing, is certainly worth complaining about. The rest seems mostly nitpicking. I wouldn't work myself up about the pager issue as discussed elsewhere here. I do somewhat agree with a rather useless default for the range of things to show (i.e. I'd prefer -b [1] to be implied unless specified otherwise, for example). 1: -b means 'since the last boot' for people not using journalctl. The default is instead to show the full journal from its beginning, as stated in the article.
raverbashing 12y ago
" Oh, the full message is available, all right, but journalctl specifically and deliberately invokes the pager in a mode where you have to scroll sideways to see long lines. Under no circumstance is all of a long line visible on screen at once so that you may, for example, copy it into a bug report." Well, that's what you get by supporting a system built by obnoxious incompetents that think they can replace the most important process in the machine "(Oh, and journalctl goes out of its way to set up this behavior. Not by passing command line arguments to less, because that would be too obvious (you might spot it in a ps listing, for example); instead it mangles $LESS to effectively add the '-S' option, among other things.)" Yes, of course.
wtbob 12y ago
Regarding the first problem, as someone else said, 'Remember how awesome PulseAudio was? Imagine that for PID 1!'
- cbd1984 12y ago
  PulseAudio gives me a better out-of-the-box experience than any other sound system, bar none.
  - eots 12y ago
    Guess I'm the only one who had a cron job restarting the PulseAudio process regularly.
    - bkeroack 12y ago
      # keep systemd happy 0 * * * * /bin/shutdown -r now
      - digi_owl 12y ago
        You know you are old when the first thing that comes to mind is a User Friendly strip... http://ars.userfriendly.org/cartoons/?id=19990302 http://ars.userfriendly.org/cartoons/?id=19990302
    - amluto 12y ago
      I had to regularly restart PulseAudio to keep Firefox's context menu working :( On the one hand, PulseAudio really would have benefitted from a built-in watchdog to restart it its early days. On the other hand, before PulseAudio, I never really had working sound at all, so I consider PulseAudio to have been an improvement.
  - Wilya 12y ago
    That might be true today (I no longer have linux desktops, so I have no idea), but it took years of buggy (or extremely hard to configure) releases to reach that point.
upofadown 12y ago
I felt dirty, but this made me laugh out loud: >While I'm here, let me mention that journalctl's default behavior of 'show all messages since the beginning of time in forward chronological order' is about the most useless default I can imagine.
- cesarb 12y ago
  Yeah, that's an annoying default. At least it also uses a pager by default, so it stops after the first screenful. On the other hand, "journalctl -b" (show all messages since the machine has been last powered on, in forward chronological order) is quite useful.
  - the_why_of_y 12y ago
    Yep... even better is that it takes an index, so you can "journalctl --list-boots" to look up boot by timestamp and "journalctl -b -2" to look at the log of the second-to-last boot to copy and report the kernel oops in there.
userbinator 12y ago
Looks like there is another bug open already with a similar backtrace, also triggered by an upgrade: https://bugzilla.redhat.com/show_bug.cgi?id=1130633 https://bugzilla.redhat.com/show_bug.cgi?id=1130633
staticshock 12y ago
Guys, the `less -S` problem is bunk, because you can easily change it at runtime. From the OPTIONS section of `man less`: Most options may be changed while less is running, via the "-" command. So, just start `journalctl -l` as you would, then type in `-S` and scroll the screen in any direction to force it to re-draw. Discussing it any further is bikeshedding, and distracts from the much more interesting segfault discussion.
- deleted 12y ago
  [deleted]
- deathcakes 12y ago
  To which I would say that you are completely missing the point of the complaint levelled here - that you can work around a stupid design decision does not negate the stupidity of that design decision. Personally I think the default of show all logs ever is worse but both mean that what should be an integral and useful tool is initially off putting at best and constantly irritating at worst. I say this as an arch linux user for the past three years.
roller 12y ago
With apologies for drifting off topic: I love less's command line interface for some of this stuff. The command line switch to chop long lines: -S The in-app command to toggle chop lines: -S The reverse is almost true for things like /search, F (follow) or G (go to end). They can be used from the command line with +/search, +F or +G
fngegngkg 12y ago
The systemfailureD apologists were quick on the postings here today. All the progress Linux made across desktop and server is mostly at risk now due to the instability of this rogue growing cancer - I won't call it an init system anymore, it is already subsuming more than that, and it really shows redhat's end aim to own the Linux market by making it too hard to run unless you are using of their installs. BSDS were there all along, we just never saw how good they were until pootering waged war on the Linux distros.
- digi_owl 12y ago
  Hacker News is a startup hangout. statups makes more and more use of cloud services. Systemd is in large part about cloud...
  - felixgallo 12y ago
    could you point me to the large part of systemd that is cloud? Is it the "desktop bus", or maybe mdns and dhcp, or perhaps the core files being stored in the journal?
    - EmanueleAina 12y ago
      Drawing a clear line is difficult, and the fact that many people are enraged with systemd for the opposite reason (systemd is too desktop-oriented, it has no place on servers) just proves that systemd does a lot to cater for different use cases (which in some case may even overlap), but I guess some features that may be useful in cloud settings are: * service lifetime tracking, restarting them if they go down * lots of resources-tracking knobs thanks to cgroups * support for read-only root filesystem * strong integration with containers: http://0pointer.net/blog/systemd-for-administrators-part-xxi.html * many batteries included (no need to ship ntp on VMs thanks timesyncd, journald can replace syslogd for basic usage, networkd tries to do the right thing without any manual setup when run in a VM, etc.) I'm sure there are more, these are just the few ones I could think off the top of my head.
      - 12y ago
angersock 12y ago
Man, anybody else remember being able to use tail and grep to decode system logs? And not having the init process crash? Seriously, this is clownshoes. Their developers and evangelists should get hit by a SIGBUS.
jdlyga 12y ago
All I've been seeing lately are posts very critical of systemd. I really don't know much about it, and I would love to hear the other side of the story. Why is it so popular lately?

121 points by lreeves 12y ago

I'm no fan of systemd, but do we really need to be nitpicking its every failing? The alternatives are certainly not perfect either. If you don't like it, don't use it. Many distros are still committed to supporting other init systems along side systemd and many BSDs have a very good philosophy with things like this and may be worth a look too. Can we just agree to move on? I'm sick of the endless banter. If you like systemd, use it, if you don't like it, don't. If you think it needs more time to mature, stay away from it for a while and experiment occasionally. That's my plan at least.

broodbucket 12y ago
Within all the noise about systemd I think posts like this are quite beneficial. Instead of discussing politics or throwing out buzzwords, it presents a specific technical shortcoming. Positive or negative, these are the types of things that people should be reading about in relation to systemd. The problem is that people are deciding whether or not they like systemd based on "do you like Lennart" or "do you like The UNIX Philosophy", because that's what people are making noise about.
- Mister_Snuggles 12y ago
  As I said in another comment, I really wish it went into more detail on the actual problem. The problems with journalctl are usability ones, which doesn't make them any less of a problem, but doesn't point to an underlying technical failure either. The problems with systemd are basically stated as "a Fedora upgrade causes systemd to go bad." Is this a systemd problem or a Fedora problem?
  - tankenmate 12y ago
    Some anti-fragility on the part of both Fedora and systemd would help here. Maybe a statically compiled init would be a start?
    - vezzy-fnord 12y ago
      systemd does not support static linking due to its use of NSS. See: http://lists.freedesktop.org/archives/systemd-devel/2014-March/017493.html http://lists.freedesktop.org/archives/systemd-devel/2014-Mar...
      - ultramancool 12y ago
        > Only the most trivial libraries actually support it. ... what kind of nonsense is that? Almost every library supports static linking to a basic degree at least. Heck, I statically link giant things like Qt and Boost every day at work.
        vodik 12y ago
        Well the core of the issue here is you can't really staticly link against a library that depends on dlopen/dlsym (nss system is built around this). I mean, you could, but now your staticly linked binary still depends on other dlsym'd code. So whats the point? (Not trying to make an argument that this is good or bad behaviour)
        ultramancool 12y ago
        I understand, but the question then becomes why your library depends on dlopen and why you don't offer an alternative static method as Qt's "plugins" do. Doesn't seem like good design on the part of the NSS devs to me, and probably a poor choice of library on the part of the systemd devs to me. But maybe I'm missing something.
        the_why_of_y 12y ago
        The library that depends on dlopen is glibc.
  - digi_owl 12y ago
    Given that Fedora and systemd has been developing in lockstep over the last few years, one would think they worked well together...
    - Mister_Snuggles 12y ago
      I'd hope that any distribution would be packaged to work well with it's init system, regardless of which init system that is. After all, the whole point of a distribution is to have a core set of software[0] that works together reasonably well. [0] With the number of different packages in a modern Linux, I don't expect everything in a distributions repository to work perfectly, but core stuff (like the stuff that gets you from a cold boot to a login prompt, etc) should be rock solid.
- felixgallo 12y ago
  those are completely valid criteria. What's wrong with wanting your system to follow the closest thing to a known axiom in operational computer science that has ever existed?
- noselasd 12y ago
  I don't really have a good or bad opinion about systemd, to me the init system and things around it are just pieces that needs to work. The points discussed here are serious, should be filed as bug reports, and should be acted on by the developers. But the article also just sounds like systemd bashing, software has bugs and poor thought out solutions - which can be fixed or improved - it's just code. I could probably have written a similar article about many, many Fedora (and Debian) upgrades over the course of a decade or so about core components such as distro upgrades making the init system crashing, failing to start services, glibc upgrades screwing up servers, grub upgrades calling for a 2 hour car drive to fix and so on. But I didn't, I either filed a bug report, fixed the code, or was just too plain busy getting things back to working.
vezzy-fnord 12y ago
Given the huge ambitions of the project maintainers and how everyone's rushing to adopt it even amidst sloppy integration work, yes, the criticism is worth it. "The alternatives are certainly not perfect" is no consolation. At all. Especially considering the vast majority were never fairly considered to begin with.
- ultramancool 12y ago
  I had the same sort of annoyances, but instead of whining, I simply ditched it for now. I don't see why people have such a problem doing this.
e40 12y ago
if you like systemd, use it, if you don't like it, don't. That is so naive. No one chooses to use systemd. People choose a distro. It's being forced on long time users of distros without any of the users having a say. Yeah, I can switch from one distro to another. That's a choice I have. But it's not so simple as you make it sound.
- deleted 12y ago
  [deleted]
- ansible 12y ago
  Yeah, I can switch from one distro to another. That's a choice I have. But it's not so simple as you make it sound. Switching distros is like moving to a new city. If you stay in the same state (or at least in the same country) it isn't too disorienting. But switching to something completely different (like on another continent, or going from an RPM-based to deb-based distro) will require some significant effort to adapt to the new situation. We pick our distro of choice based on a number of factors. Just because it has some downsides doesn't mean we have to move immediately. But that doesn't mean we can't give (constructive!) feedback if there are problems.
- vidarh 12y ago
  > No one chooses to use systemd. I did. I've been systematically installing systemd on my Debian machines.
- chimeracoder 12y ago
  > It's being forced on long time users of distros without any of the users having a say. It's being adopted by distributions in accordance with the stated and implied wishes of the developers, maintainers, and users of the distributions. The reason that (e.g.) Arch dropped support for legacy init so quickly was because they literally failed to find anyone who wanted to go through all the work of maintaining the old init system. If none of the users want to do the work, then nothing's being "forced" - that's the way community-organized FOSS projects work![0] Same goes for Debian, who made the switch in accordiance with their normal development process (including a general resolution in support of systemd!). Debian has promised to maintain systemd-shim (supporting other init systems) through at least Jessie, and possibly longer, depending on whether there is enough demand for alternative init systems, and support from developers for maintaining it in future releases. Ironically, the biggest threat to systemd-shim in Jessie's successor are people who claim they want to fork Debian - if all the people who hate systemd leave Debian, then nobody who wants to actually do the work to maintain the alternative init systems will remain with the Debian project[1]. [0] Even then, you can still find an AUR package for it (https://aur.archlinux.org/packages/initscripts-fork/ https://aur.archlinux.org/packages/initscripts-fork/), though as with everything in the AUR, it's subject to much less testing. Note that the package has only 27 votes - this means that only 27 people actually said that they care about moving this package back into the main distro. [1] This may be a bad or a good thing, depending on who you ask.
zobzu 12y ago
Not sure if /sbin/init failing is a nitpick. its kind of a catastrophic failure as far operating systems are concerned.
- digi_owl 12y ago
  Never mind that it results in a zombie state where you still have a shell, but no commands actually work.
howeyc 12y ago
> Can we just agree to move on? No, we can't. Sorry but that's just how the world works. Just like every Go language post has "generics or crap" mentioned, every systemd post will have "The Unix Way" mentioned. A price of freedom of speech is having to deal with self-filtering idiotic viewpoints. Worth it in my opinion, but I'm an anti-censorship for EVERYTHING person.
- ultramancool 12y ago
  I'm not suggesting censorship of any sort. I'm certainly the same way. I'm just suggesting level-headedness and a willingness to take or leave it and move on with life.
  - howeyc 12y ago
    I understand, I also agree with your suggestion. I guess my point is really that even though a nice ideal, based on my experience, likely not attainable.
rdtsc 12y ago
> If you don't like it, don't use it. Have you tried removing systemd from latest Ubuntu? I didn't go very well for me. I imagine removing it from RHEL 7 won't go well over either. > I'm sick of the endless banter. Ok but "use something else" or "fork it" is not exactly solving the problem either.
- digi_owl 12y ago
  And thats the thing. If it was "just a init", it would be just as replaceable as sysv, openrc or runit. But it is not, so it is not. As best i can tell, a major argument for going systemd is that something more user facing (Gnome) depends on a subsystem of systemd, and that in turn requires that systemd is used as init. In other words, systemd as a project is invasive and insidious.
  - EmanueleAina 12y ago
    The people that develop GNOME chose to depend on the D-Bus interfaces exported by some systemd-provided services (eg. logind) because they actually solve some problems they were facing with a clean and robust solution. Yet they made sure that the dependency is optional, to accomodate those that for a reason of another do not use systemd. This means that systemd is not insidious, it's valuable.
- EmanueleAina 12y ago
  > Have you tried removing systemd from latest Ubuntu? I didn't go very well for me. I imagine removing it from RHEL 7 won't go well over either. Ubuntu is not the Universal Operating System, that's Debian. This means that Ubuntu is definitely opinionated in its software selection, and rightly so. You couldn't replace Upstart and now Ubuntu relies on systemd. If you have the skills to fully evaluate the costs and benefits of mixing and matching random pieces of software, distribution like Debian or Gentoo may be a better fit for you. Indeed, switching to sysvinit in Debian is trivial and the systemd team worked really really hard to make sure the transition is smooth even if one keeps switching back and forth. They should really be praised for the effort they've put in it.
_cipher_ 12y ago
> I'm no fan of systemd, but do we really need to be nitpicking its every failing? Yes. When some developers are pushing down your (and mine) throat critical software that sucks _that_ much, that's the least someone can do.
- EmanueleAina 12y ago
  Some developers wrote some code and published it in a software package under a free software license. Other developers picked up said package and added it to the package collection they are maintaining. You choose to use the package collection maintained by said developers, which means that you trust their technical decisions otherwise you'd be running software from people you don't trust, which is not a great idea. Nobody is pushing software down anyone's throat. Please avoid such hyperbolic mischaracterization, it really does not contribute anything to the discussion.
  - yarrel 12y ago
    > You choose to use the package collection maintained by said developers, which means that you trust their technical decisions This does not however mean that I must silently accept their bad choices. If they were about to step out in front of a bus I wouldn't remain silent because I trust their judgement. Sometimes people you respect excercise poor judgement. This is one of those cases. Feedback is a thing. You're giving it here. Why shouldn't I give it to the distros that certainly are trying to force systemd on me?
    - EmanueleAina 12y ago
      > This does not however mean that I must silently accept their bad choices. But there are proper places to discuss this issue with them, and HN is totally not one of those. > Sometimes people you respect excercise poor judgement. This is one of those cases. Sure, but have you considered that the one exercising poor judgement in this case may be you? I don't know which distro you're using, but Debian has extensively discussed the issue and the first tech CTTE resolution made lots of valid technical points in favour of systemd. For sure it's not an obviously bad choice, and if your maintainers have chosen to ship it they may have valid reasons to do so. Not to mention that's still entirely possible to avoid systemd for whatever reason, at least if you're using Debian it's trivial. > Feedback is a thing. You're giving it here. Why shouldn't I give it to the distros that certainly are trying to force systemd on me? Right, but I'm here trying to give feedback to you for your actions in this thread. You're here giving feedback on HN to people that probably don't read HN, and doing so with a rather inflammatory tone. If you want to give feedback to your distro maintainers, please get in touch with them in the appropriate channels (IRC, mailing lists) and ask them their reasons, but please first check if they haven't already stated those, one can just get annoyed if every person in the world keeps asking stuff that has been answered countless times.
skywhopper 12y ago
I think ongoing constructive discussion about the technical and philosophical issues with software design, particularly of open-source software that affects huge numbers of users, is always valuable. The OP here even likes systemd, so the article isn't nitpicking. As others have pointed out, when your init process faults, that's a major problem. In the real world, for the core elements of an operating system, whatever features they have are less important than how well they fail. In this case, the OP has identified a very very bad failure case. More importantly, this report makes concrete the theoretical complaint against systemd that its monolithic design is not just a philosophical issue, but a practical one. The init process needs to be rock solid, but a law of software is that bigger software is less stable. So if the sprawling nature of systemd means it will tend to become less stable over time, then we have a major practical issue that needs to be addressed. This is not a matter of not moving on. Rather, given that systemd has been adopted by several major Linux distributions, it's actually far more important now to criticize its design failures, to draw attention to get things fixed. If "moving on" means we shouldn't critique the software, then how will it ever improve?
mrweasel 12y ago
>If you like systemd, use it, if you don't like it, don't. I think people would care less about systemd if that was a realistic choice. I strongly believe that the whole systemd debate would go away if systemd wasn't a dependency of GNOME and other stuff. Personally I have no strong feelings one way or the other, but I do fear that the BSDs are going to be left behind as they to some extend have been piggybacking on Linux in terms of desktop software.
exabrial 12y ago
you're missing the point... SystemD is about RedHat trying to take over the entire Linux ecosystem. It's intentionally designed so if you use any part of it (which there are good parts) then you have to use all of it (which there are many bad parts). Oh and conveniently RedHat is the main project maintainers.

XorNot 12y ago

I can't speak to the first problem, but the second is more of "this is strange and I don't care to learn how it works". journalctl --no-pager Problem solved. I mean, you have the command, it's right there in the usage text. Which, awesomely, does invoke a pager when you do journalctl --help so if you're stuck on a 80x25 console (as you might be on a broken system) you can easily actually read the help options.

masklinn 12y ago
> I can't speak to the first problem, but the second is more of "this is strange and I don't care to learn how it works". I can't speak for you not speaking to the first problem, but the second is more of "this is strange and I didn't care to read the text" > In the grand systemd tradition, there is no option to control this; all you can do is force journalctl to not use a pager or work out how to change things inside the pager to not do this. (emphasis mine) > Problem solved. I mean, you have the command, it's right there in the usage text. Which, awesomely, does invoke a pager when you do journalctl --help so if you're stuck on a 80x25 console (as you might be on a broken system) you can easily actually read the help options. From reading the text, I'm under the impression that the issue is the mangling of $LESS. One would expect that $LESS is set up how the user wants it, if the user wants -S he'll set that.
- darklajid 12y ago
  Okay, okay. I understand that the user might be surprised. man journalctl $SYSTEMD_LESS Override the default options passed to less ("FRSXMK"). So - whereever the user defines $LESS I'd add a line a la export SYSTEMD_LESS=$LESS and that should solve the problem nicely, right? Surprising? Not intuitive? Maybe. But easily fixable with a quick peek at the man page.
  - zaphar 12y ago
    So I have no dog in the systemd fight. However. $SYSTEMD_LESS is quite possibly one of the most aggressively stupid ways to force a bad UX on someone. I mean seriously. If you have $LESS defined then you obviously have an expectation that less will behave in a certain way. An expectation that has in all probability built up over years of tweaking your shell. For some other application to ignore that preference and in fact force it's own preference on you and only allow you to use your own preference for how less should work by reading not the docs for less but the docs for an unrelated application? That breaks so many rules it's just sad. Yes it's easily fixable with a quick peek at the man page for systemd which might not be the first thing you check given you've been using less for years. Systemd's manpage is the last thing I would check. First would be what is my $LESS set to? Second would be is less still less? At some point it would occur to me that maybe systemd is being aggresively stupid and check their manpage. But given I have no particular preference I would be inclined to assume it couldn't possibly be systemd's fault. I must have messed something up. Put this minor annoyance in the middle of attempting to debug a segfaulting init? That's what we call a major annoyance.

lstyls 12y ago

How much of the blame should systemd be taking for these bugs? Unless the same segfault is happening on other distros, this seems like a problem in Fedora rather than systemd. I suppose this does give some substance to the systemd criticisms that it is too system-critical for a process that is technically outside of the Linux kernel.

kosinus 12y ago
If you're saying, a bug should always be investigated downstream before reporting it upstream, then I agree. If you're saying, there are scenarios where segfaults caused by upstream code are not upstream bugs, then I disagree.

Mister_Snuggles 12y ago

It's interesting to see examples of where systemd falls down, though I'd be interested to hear what the actual problem is. The Fedora bug report and linked page basically say "systemd fell over during an upgrade", but is this really a systemd problem or is it a Fedora problem?

digi_owl 12y ago
At this point in time they seem joined at the hip. I think a Pottering retort about Debian users complaining about the size of systemd was that it was introduced over time on Fedora. So if any distro should make systemd shine, it would be Fedora.
cesarb 12y ago
It's failing during an upgrade. My guess would be that it's a systemd problem, but one triggered by mixing systemd versions. Probably the systemctl from Fedora 21 does something that the systemd from Fedora 20 doesn't like. If that's the case, the bug might have even already been fixed in systemd upstream, but the fix wasn't applied to the older systemd in Fedora 20. The solution would be to backport the fix to Fedora 20's systemd package, so users who do a "yum update" before upgrading to Fedora 21 would have it. An alternative workaround might be to, immediately after unpacking the new systemd, send a SIGTERM to PID 1. According to http://www.freedesktop.org/software/systemd/man/systemd.html http://www.freedesktop.org/software/systemd/man/systemd.html, that signal directly tells PID 1 to re-exec itself, without using systemctl in the process. That would make PID 1 be the new systemd before anything tries to use the new systemctl.
siebenmann 12y ago
Almost any time PID 1 segfaults, it's PID 1's fault. With PID 1 being systemd, that makes it systemd's fault here. Based on gdb stack backtraces in the Fedora bug report and looking at the systemd code, it seems like some sort of memory corruption or overwrite, perhaps a use after free issue. The segfault itself comes from dereferencing a clearly invalid pointer and said pointer was obtained by dereferencing a structure field through another pointer, so you'd get exactly this result if the structure was overwritten with other data at some point. (In my grumpy sysadmin view, it is PID 1's fault even if the distribution is doing odd things around PID 1. Init processes need to be absolutely rock solid and extremely defensively coded, precisely because the world basically dies if they ever fall over.) (I am the author of the original post.)

darklajid 12y ago

The first problem, PID 1 crashing, is certainly worth complaining about. The rest seems mostly nitpicking. I wouldn't work myself up about the pager issue as discussed elsewhere here. I do somewhat agree with a rather useless default for the range of things to show (i.e. I'd prefer -b [1] to be implied unless specified otherwise, for example). 1: -b means 'since the last boot' for people not using journalctl. The default is instead to show the full journal from its beginning, as stated in the article.

raverbashing 12y ago

" Oh, the full message is available, all right, but journalctl specifically and deliberately invokes the pager in a mode where you have to scroll sideways to see long lines. Under no circumstance is all of a long line visible on screen at once so that you may, for example, copy it into a bug report." Well, that's what you get by supporting a system built by obnoxious incompetents that think they can replace the most important process in the machine "(Oh, and journalctl goes out of its way to set up this behavior. Not by passing command line arguments to less, because that would be too obvious (you might spot it in a ps listing, for example); instead it mangles $LESS to effectively add the '-S' option, among other things.)" Yes, of course.

wtbob 12y ago

Regarding the first problem, as someone else said, 'Remember how awesome PulseAudio was? Imagine that for PID 1!'

cbd1984 12y ago
PulseAudio gives me a better out-of-the-box experience than any other sound system, bar none.
- eots 12y ago
  Guess I'm the only one who had a cron job restarting the PulseAudio process regularly.
  - bkeroack 12y ago
    # keep systemd happy 0 * * * * /bin/shutdown -r now
    - digi_owl 12y ago
      You know you are old when the first thing that comes to mind is a User Friendly strip... http://ars.userfriendly.org/cartoons/?id=19990302 http://ars.userfriendly.org/cartoons/?id=19990302
  - amluto 12y ago
    I had to regularly restart PulseAudio to keep Firefox's context menu working :( On the one hand, PulseAudio really would have benefitted from a built-in watchdog to restart it its early days. On the other hand, before PulseAudio, I never really had working sound at all, so I consider PulseAudio to have been an improvement.
- Wilya 12y ago
  That might be true today (I no longer have linux desktops, so I have no idea), but it took years of buggy (or extremely hard to configure) releases to reach that point.

upofadown 12y ago

I felt dirty, but this made me laugh out loud: >While I'm here, let me mention that journalctl's default behavior of 'show all messages since the beginning of time in forward chronological order' is about the most useless default I can imagine.

cesarb 12y ago
Yeah, that's an annoying default. At least it also uses a pager by default, so it stops after the first screenful. On the other hand, "journalctl -b" (show all messages since the machine has been last powered on, in forward chronological order) is quite useful.
- the_why_of_y 12y ago
  Yep... even better is that it takes an index, so you can "journalctl --list-boots" to look up boot by timestamp and "journalctl -b -2" to look at the log of the second-to-last boot to copy and report the kernel oops in there.

userbinator 12y ago

Looks like there is another bug open already with a similar backtrace, also triggered by an upgrade: https://bugzilla.redhat.com/show_bug.cgi?id=1130633 https://bugzilla.redhat.com/show_bug.cgi?id=1130633

staticshock 12y ago

Guys, the `less -S` problem is bunk, because you can easily change it at runtime. From the OPTIONS section of `man less`: Most options may be changed while less is running, via the "-" command. So, just start `journalctl -l` as you would, then type in `-S` and scroll the screen in any direction to force it to re-draw. Discussing it any further is bikeshedding, and distracts from the much more interesting segfault discussion.

deleted 12y ago
[deleted]
deathcakes 12y ago
To which I would say that you are completely missing the point of the complaint levelled here - that you can work around a stupid design decision does not negate the stupidity of that design decision. Personally I think the default of show all logs ever is worse but both mean that what should be an integral and useful tool is initially off putting at best and constantly irritating at worst. I say this as an arch linux user for the past three years.

roller 12y ago

With apologies for drifting off topic: I love less's command line interface for some of this stuff. The command line switch to chop long lines: -S The in-app command to toggle chop lines: -S The reverse is almost true for things like /search, F (follow) or G (go to end). They can be used from the command line with +/search, +F or +G

fngegngkg 12y ago

The systemfailureD apologists were quick on the postings here today. All the progress Linux made across desktop and server is mostly at risk now due to the instability of this rogue growing cancer - I won't call it an init system anymore, it is already subsuming more than that, and it really shows redhat's end aim to own the Linux market by making it too hard to run unless you are using of their installs. BSDS were there all along, we just never saw how good they were until pootering waged war on the Linux distros.

digi_owl 12y ago
Hacker News is a startup hangout. statups makes more and more use of cloud services. Systemd is in large part about cloud...
- felixgallo 12y ago
  could you point me to the large part of systemd that is cloud? Is it the "desktop bus", or maybe mdns and dhcp, or perhaps the core files being stored in the journal?
  - EmanueleAina 12y ago
    Drawing a clear line is difficult, and the fact that many people are enraged with systemd for the opposite reason (systemd is too desktop-oriented, it has no place on servers) just proves that systemd does a lot to cater for different use cases (which in some case may even overlap), but I guess some features that may be useful in cloud settings are: * service lifetime tracking, restarting them if they go down * lots of resources-tracking knobs thanks to cgroups * support for read-only root filesystem * strong integration with containers: http://0pointer.net/blog/systemd-for-administrators-part-xxi.html * many batteries included (no need to ship ntp on VMs thanks timesyncd, journald can replace syslogd for basic usage, networkd tries to do the right thing without any manual setup when run in a VM, etc.) I'm sure there are more, these are just the few ones I could think off the top of my head.
    - 12y ago

angersock 12y ago

Man, anybody else remember being able to use tail and grep to decode system logs? And not having the init process crash? Seriously, this is clownshoes. Their developers and evangelists should get hit by a SIGBUS.

jdlyga 12y ago

All I've been seeing lately are posts very critical of systemd. I really don't know much about it, and I would love to hear the other side of the story. Why is it so popular lately?