12 ms·

How to Crash Systemd in One Command

224 points by agwa 10y ago

tcoppi 10y ago
> "Systemd is defective by design." The most succinct explanation of Systemd possible.
- mSparks 10y ago
  I've complained a few times that systemd haters havent explained the hate. having read this post I'd say a slightly less succinct but infinitely more informative version would be > "Systemd is defective by design because they totally disregard the fail-safe by design rules"
  - qwertyuiop924 10y ago
    Yes. I wish people would explain their hatred. I understand that there are valid reasons to hate systemd, which is why I avoid it like the plague. But so many people get sucked in by the initial usefulness, and don't see the problems lurking beneath the surface.
    - 5ilv3r 10y ago
      Binary only logs was enough for me to hate it.
      - qwertyuiop924 10y ago
        That's it? Man that's just the start...
    - fapjacks 10y ago
      Yes, a large part of which is the process behind systemd. You're adopting more than just software when you adopt software, and I think investigating what the systemd developers are like is a worthy exercise -- in particular how they approach bugs. I think most people would be surprised to learn what that is like.
      - qwertyuiop924 10y ago
        Ah, yes: When is a bug not a bug? According to the systemd developers, it's when we don't encounter it on our personal machines.
        fapjacks 10y ago
        ... And also when they can pass the buck to kernel developers, even when it's not a kernel bug.
        digi_owl 10y ago
        Reminds me that i ran into an email chain a while back where the kernel devs pondered forking or taking over udev development because they were tired of power games from the maintainer. This was from back before udev became part of systemd btw.
        qwertyuiop924 10y ago
        Greg-KH is both one of the original developers on udev and a major force behind systemd. He maintains -stable on the linux kernel. So either the kerbel devs did take over udev development...
        digi_owl 10y ago
        I do wonder if the kernel will split between GregKH and T'so once Torvalds steps down. BTW GKH was oddly absent in that thread, and the kernel devs ire was aimed at Sievers...
        qwertyuiop924 10y ago
        Well, GKH is One Of Them. He's less likely to attract their ire, and if he does, at the end of the day, they have to answer to him. But in a fight between GKH and T'so, I'll take T'so. /dev/random's ...interesting design aside, I frankly trust T'so far more than I trust GKH, who backs LP, a person I emphatically do not trust, in a lot of things.
justin_vanw 10y ago
It seems like it is quite fashionable to hate on systemd, and it seems like systemd is kindof a piece of crap - in some ways. However, linux is missing basic functionality other os's offer, and systemd is showing up and trying to fill in those blanks. This is open source, if you don't like systemd, if you think it's crap, if you think there are obviously better ways to do it, well, what are you waiting for then? Systemd is winning because they showed up and basically nobody else did. It's open source fork it and 'do it right'. Or point out the problems and hope that someone else will step up instead? I think the world would better place if people spent less time griping and more time fighting to get their patch that fixes the problems merged. I guess coding is harder than writing blog posts though. EDIT: I'm not saying systemd is perfect or even good. My point is that the people writing the code are going to have the final say, whether it's right or wrong, and it is unlikely that people writing blog posts lampooning it are really going to make a difference. I mean systemd has been despised and highly criticized from the beginning, and look where we are.
- korethr 10y ago
  > However, linux is missing basic functionality other os's offer, and systemd is showing up and trying to fill in those blanks. I'm curious what functionality other OSes had that Linux did not before systemd came along. It seems to me that it has been subsuming functions of other systems on Linux that were already present, as opposed to offering anything new. Socket activation? inetd/xinetd. Timed start of programs? Cron and At Logging? rsyslog/syslog-ng/etc. DNS? -- dnsmasq, powerdns, bind, etc. Honestly, I think if system had stopped at doing init, and focused solely on that and doing the best damned possible job of it, there'd be less hate. I think what's pissing off a lot of the systemd haters is the scope creep of the project has undergone. I was personally fine with the idea of systemd when all it was promising was init and service management. But then oh, were going to take over /dev, and cron and logging, and logins and DNS and (to be determined). Ahh, thanks, but no thanks.
  - bjourne 10y ago
    > I'm curious what functionality other OSes had that Linux did not before systemd came along. Uniform daemon/service handling. Just look at the scripts in /etc/init.d (or wherever they are located in your particular SysV init setup). It's a total mess. The rest of what systemd offers I can't comment on because I'm not familiar with it. But it is clear to me that systemd's declarative .service files are a billion times better than shell scripts.
milankragujevic 10y ago
Archived page, since it's not loading for me... http://archive.is/8xUSY http://archive.is/8xUSY
- mavdi 10y ago
  How to crash a blog with one submission
- agwa 10y ago
  My hosting provider thought I was under DDoS attack and started aggressively throttling port 443. I failed over to my backup provider, probably permanently.
belovedeagle 10y ago
This is an excellent writeup explaining without hysterics why systemd is terrible software. Unfortunately, I expect that this post will be brigaded by systemd-apologists anyways, denying newcomers to the debate the opportunity to hear the facts...
- andrewstuart2 10y ago
  I'm not sure how this comment is helpful to the discussion. As far as I can tell, you'd rather only hear the facts that fit your opinion of systemd as terrible software. I think the obvious right answer is that both sides present only facts and let newcomers decide based on that.
  - fapjacks 10y ago
    I completely agree with you. That would be nice, if people were presented with facts and a choice. But they weren't, and aren't.
    - andrewstuart2 10y ago
      Well, it's never too late to start. :-) As history shows, you can try to suppress truth for a while, but it is usually either unstoppable, or insignificant.
  - belovedeagle 10y ago
    As the other reply said, the problem is that newcomers aren't allowed to hear the anti-systemd facts, as the systemd gods have already decided that all Linux users (or at least all distros) must bow down and accept them, and no dissenting opinion is allowed to stay on the front page of HN for long (this post has already been brigaded off the front page, apparently).
    - fapjacks 10y ago
      It could be the "controversial topic" penalty, which is extremely heavy-handed. But I also notice that both anti-systemd and pro-other-init-system posts and comments are barraged with downvotes on all kinds of forums, including HN. It is definitely something that certain people don't want out in the world.
geofft 10y ago
I've been thinking for a bit that it's reasonably easy to write a PID 1 that reaps children and sends siginfo_t structs over a pipe to PID 2, which can be an init system as monolithic as you want. Of course, it would also be simple for the kernel to allow PID 1 to die/crash, and just restart it instead of panicking. I don't know why the kernel continues to treat PID 1 so specially—if the ewontfix.com PID 1 is sufficient, it's simpler for the kernel not to bother with the reparenting logic and stop treating PID 1 as a special process.
- mschuster91 10y ago
  > I don't know why the kernel continues to treat PID 1 so specially By design all processes inherit from PID 1 (directly or indirectly), so you always have an unbroken tree from every process back to PID 1. PID1 aka init is supposed to do bookkeeping (e.g. zombie processing), and if PID1 crashes the bookkeeping records are lost. Because of this, it is better to fail hard, instead of allowing for corrupted process trees or similar. Also, ps, top and basically every program that works with processes expects one root process for the tree. Breaking this would basically break compatibility with every existing userland (which the kernel cares the most about!)
  - justinsaccount 10y ago
    > Breaking this would basically break compatibility with every existing userland By "Breaking this" you mean "fixing the fact that pid 1 can't crash and restart"? > Because of this, it is better to fail hard It can't be fixed because it would "break compatibility" where that "compatibility" is that the machine should panic? You're not really making a good case for why this can't be changed. Why not * pid 1 exits * kernel re-executes init as a new pid 1 * userspace continues on as if nothing happened.
    - geofft 10y ago
      Right. If init opts into it, e.g. by setting some sysctl or something, the kernel changes behavior in the following way: 1. Processes that are reparented to init aren't actually considered children of init for the purposes of the wait() system call. Their ppid still becomes 1, so parent process walks work, but when they exit, the kernel immediately cleans them up. Zombies still exist in this change, as long as their parent still exists; the only thing that changes is that processes (zombies or not) are not reparented to init when their parent dies. Direct children of pid 1 are not treated specially. 2. When init is about to exit or crash, the kernel turns it into a zombie. It then starts a new init process and reparents the old init to the new one, allowing it to do things depending on the cause of the crash. Direct children of the old init have lost their parent, and are treated like any other parentless process as described above. 3. If execing the new init fails, the kernel panics. This would be completely backwards-compatible. (I might even argue against 3, to remove all worry about init crashing; since pid 1 is no longer required to do anything, if exec fails, the kernel can just leave pid 1 as a stuck process table entry, let the system continue running, and log something to dmesg so the sysadmin can diagnose the problem and reboot the machine cleanly.)
5ilv3r 10y ago
Ran as a normal user on debian jessie, and now ssh logins are delayed by 30 seconds. I don't see any other problems yet, but this is a VM host and so leave it running for a while for fun. :)
- aorth 10y ago
  Ran it on an up to date Ubuntu 16.04 and lost the ability to run any systemctl commands like status, restart, etc. Web server and other system services were still working, but I couldn't `reboot` or `systemctl halt` etc... had to hard reboot the VPS from hosting console. :)
  - andreyv 10y ago
    "systemctl reboot --force --force" should work in such cases. The reboot will be executed by systemctl itself.
rcarmo 10y ago
First they came for PID 1, and we were silent. Oh, wait, we weren't. But although I don't miss the rickety init scripts, I've grown more and more worried about stuff like this, enough to ponder replacements (in Go or Python, which seem to be generally acceptable for systems software these days). Ignoring for the moment the usual biases against Python, how feasible/how much support would it be/have to build something to replace systemd (by having PID 1 do the minimum possible, in C, and having the rest done with Python)?
- djsumdog 10y ago
  I've been working at a Docker shop for a while and I have a feeling it will be the future for running services. It makes systemd somewhat irrelevant. There are still a lot of issues I've had with Docker, which I won't go into. It also can't replace everything running on your Linux system. But it's a neat idea and big players use it in production for huge workloads. Now the whole AWS-containers, docker-machine, swarm, mesos, coreos, kubernetes thing is a cluster fuck of choices .. but you have choices. And unlike running an Ansible script on some provides base image (which is totally different from Rackspace, Linode, etc.), you are guaranteed your container will work on any docker engine with little modification (your orchestration files may change though). I have a feeling, for better or worse, containers will be the future and will stick around long after systemd hopefully dies.
  - digi_owl 10y ago
    Nah, systemd is angling to supplant docker for containers. There is already a great deal of bad blood between the two because of the reworked cgroups management. And there is also the likes of CoreOS that is basically systemd-the-distro. Never mind that we have the likes of xdg-app/flatpak that is basically aiming for what you are thinking about, and it is systemd dependent.
    - fridsun 10y ago
      It was quite amusing seeing systemd causing systemd-specific trouble in CoreOS development though, such as systemd requiring too much privilege when running inside container. Many people just use s6 happily.
deleted 10y ago
[deleted]
packetized 10y ago
Erm, this does not actually seem to work on Centos7, running systemd-219-19.el7_2.13. Even tried it as root inside of a sandbox - nothing.
- lake99 10y ago
  It's the same on Archlinux, systemd-231. Nothing happened. I was able to start and stop a couple of services as quickly as before.
- SysArchitect 10y ago
  while true; do NOTIFY_SOCKET=/run/systemd/notify systemd-notify ""; done Try this. As soon as you loop it, the bug is hit pretty quickly.
  - deleted 10y ago
    [deleted]
  - nilved 10y ago
    I don't understand why that makes a difference if the bug is `assert(n > 0)`
    - SysArchitect 10y ago
      I have no idea either. Race condition somewhere? It was mentioned here: https://github.com/systemd/systemd/issues/4234 https://github.com/systemd/systemd/issues/4234
    - JdeBP 10y ago
      There is a race in the systemd-notify mechanism that means that some notification messages get discarded if the systemd-notify process happens to complete its work and exit before systemd picks up the message. They never reach the point of the assertion that is triggered here, because systemd is unable to determine the service unit that encompasses the message sender. * http://jdebp.eu./FGA/unix-daemon-readiness-protocol-problems.html#SynchronousProtocol http://jdebp.eu./FGA/unix-daemon-readiness-protocol-problems...
xemdetia 10y ago
At this point in time my biggest frustration is that even compared to upstart writing concise and clear config files for network services is just really hard, and there isn't great documentation or a short book around that covers all the edge cases other than TIAS. I feel like I've wasted more time trying to get a tidy systemd configuration than I ever had even with upstart or some of the other ones.
bitmapbrother 10y ago
>Despite the banality, the bug is serious, as it allows any local user to trivially perform a denial-of-service attack against a critical system component. Is it really that serious of a bug? You have to be a local user in order to execute it. Are other OS's somehow resilient to the actions of local user to degrade the OS? >The Linux ecosystem has fallen behind other operating systems in writing secure and robust software. While Microsoft was hardening Windows and Apple was developing iOS, open source software became complacent. He must be unaware of the numerous patches Microsoft and Apple issue on a regular basis to fix their "secure and robust software".
- falcolas 10y ago
  Any remote code execution bug can exploit this. Normally, a remote execution code bug when the user is non-root isn't a huge issue, because the user effectively sandboxes them away. This bypasses all of these benefits. Containers may prove to be a sufficient sandbox, assuming this is not executable against kdbus (which is shared across containers since it's in the kernel).
redbeard0x0a 10y ago
This article got me wondering if maybe the NSA or similar has somebody on the inside of the systemd project that is helping the project along. Systemd has replaced a lot of init systems in the linux ecosystem and the general development practices of the project is leaving a huge footprint of code that might be exploited. If they aren't involved, I'm sure they probably know a few different exploits for the system already.
- pdkl95 10y ago
  If you want to go down that rabbit hole, see: https://igurublog.wordpress.com/2014/02/17/biography-of-a-cypherpunk-and-how-cryptography-affects-your-life/ https://igurublog.wordpress.com/2014/02/17/biography-of-a-cy...
- johnfjacobi 10y ago
  I've heard this a lot and first just dismissed it as a conspiracy theory. Is there actually any substance to these thoughts, or are they only speculation?
xaduha 10y ago
Don't really care about that bug, it will be fixed I'm sure. But thank you for this post anyway, the word LLMNR caught my eye, very nice to have if it works. EDIT: works fine with Windows, without installing Bonjour/iTunes.
deleted 10y ago
[deleted]
SysArchitect 10y ago
The issue has a bug report on the systemd repo on Github: https://github.com/systemd/systemd/issues/4234 https://github.com/systemd/systemd/issues/4234
deleted 10y ago
[deleted]
walter_bishop 10y ago
Doesn't work here on Ubuntu 16.04 xenial
- Twirrim 10y ago
  I've put a comment on the github bug with an updated PoC. IT does work on 16.04, but you seem to have to do it over and over until it triggers. A working PoC seems to be: while true; do NOTIFY_SOCKET=/run/systemd/notify systemd-notify ""; done
zzxxmm11aajj 10y ago
Note: 112 comments from the peanut gallery before someone looked at the code enough to submit the trivial fix.
m45t3r 10y ago
You could simple do a DoS attack with a local user using a fork bomb too. It would be even shorter than the command showed in this post blog. So while this is a bug, the author is overreacting just to make systemd look bad.
- rascul 10y ago
  your fork bomb can be mitigated with ulimit
  - m45t3r 10y ago
    I am not saying that this is not a bug, just that this isn't the problem that the author is making. If you have local access, you probably can do other things if your only objective is to do a DoS.
    - reactor4 10y ago
      Your point is covered in the article. "If you think systemd doesn't need privilege separation because it only parses messages from local users, keep in mind that in the Internet era, local attacks tend to acquire remote vectors. Consider Shellshock, or the presentation at this year's systemd conference which is titled "Talking to systemd from a Web Browser.""
aorth 10y ago
A CVE has been requested, albeit after publication of the bug: http://www.openwall.com/lists/oss-security/2016/09/28/9 http://www.openwall.com/lists/oss-security/2016/09/28/9
- JdeBP 10y ago
  Discussed on Hacker News at https://news.ycombinator.com/item?id=12600994 https://news.ycombinator.com/item?id=12600994 .
wyldfire 10y ago
> The immediate question raised by this bug is what kind of quality assurance process would allow such a simple bug to exist for over two years (it was introduced in systemd 209). Isn't the empty string an obvious test case? All bugs are terribly easy to spot in hindsight. I don't think it's fair to criticize this one in that fashion. Systemd has lots of pros and cons, but the age of this bug isn't interesting IMO. Its severity is, but that only speaks to the critical need to patch it ASAP.
JdeBP 10y ago
It's interesting to see a "hacker" discussion of this that entirely misses several topics that one would expect programmers to at least mention: * Whether release builds should have assert() enabled. * Whether a process #1 program should use assertions at all. * Whether assert() is the right mechanism to use for input validation. * How a process #1 program should recover from assertion failures.
fridsun 10y ago
Lennart Poettering has replied on Reddit. https://www.reddit.com/r/linux/comments/54yfcd/how_to_crash_systemd_in_one_tweet/d86rs74 https://www.reddit.com/r/linux/comments/54yfcd/how_to_crash_... > So let's summarize this. There's a bug in some software. OMG! Shock! This of course never happened before! > The bug is in not exploitable remotely. The bug does not allow privilege escalation nor insertion of code. This of course makes the bug a massive vulnerability like there was no other on the planet ... ever. As bad as heartbleed multiplied by the Debian OpenSSL random generator bug to the power of 10. > The bug is caused by an error check that filters out garbage sent to PID 1. The check works correctly, except that the resulting action is a too harsh: instead of complaining and dropping it will abort the process. Such a bug is of course unprecedented and the authors of said software should be stoned and flogged right away given the severity of the issue: after all a safety check worked a bit too well, and we really can't have that because undefined behaviour of course would be a lot better than a local DoS. > The project the error was found in is large. Yet the number of CVEs collect so far is pretty small comparing it witht other projects of similar extent. Given that another bug was discovered now this obviously shows how incompetent the programmers are and that security is a unknown concept to them. > The program the bug was found in is longer than 50 lines of code but runs with privileges, all written in a low-level programming langauge that many call little more than a fancy macro assembler. The code runs on top of an operating system kernel written in the same language but running with a lot higher privileges and consisting of expoentially more lines of code including drivers of questionnable quality. This together is of course proof that the project at hand is flawed conceptually to its core. > Dha! > Lennart > (More seriously: yes this is a bug, we should fix it. But it's very low impact and the bruhaha it generated appears wildly out of scale. If all bugs in the wider Open Source ecosystem would have a similarly low impact we'd live in a much much safer world!)
JdeBP 10y ago
Lennart Poettering's response to this was interesting: > The bug is caused by an error check that filters out garbage sent to PID 1. The check works correctly, except that the resulting action is a too harsh: instead of complaining and dropping it will abort the process. * https://www.reddit.com/r/linux/comments/54yfcd/how_to_crash_systemd_in_one_tweet/d86rs74 https://www.reddit.com/r/linux/comments/54yfcd/how_to_crash_... In fact, the opposite was the case. The root of the problem was identified in the GitHub bug report: * https://github.com/systemd/systemd/issues/4234#issuecomment-250431337 https://github.com/systemd/systemd/issues/4234#issuecomment-... * hhttps://github.com/systemd/systemd/commit/d875aa8ce10b458dc218c0d98f4a82c8904d6d03 https://github.com/systemd/systemd/commit/d875aa8ce10b458dc2... What happened was that Lennart Poettering removed an error check that filtered out zero-length messages. This left the flow of control to fall through to a later point where an assertion, that had been earlier added by Lennart Poettering with the assumption that this check for zero was in place (as it had been at the time), then triggered. Of course, no-one that I have seen has yet asked why "" as a command-line argument to systemd-notify results in a zero-length message in the first place. After all, according to the doco that would be length 1, a single terminating LF byte, not zero length. That said, the server still should be proof against zero-length network input.

224 points by agwa 10y ago

tcoppi 10y ago

> "Systemd is defective by design." The most succinct explanation of Systemd possible.

mSparks 10y ago
I've complained a few times that systemd haters havent explained the hate. having read this post I'd say a slightly less succinct but infinitely more informative version would be > "Systemd is defective by design because they totally disregard the fail-safe by design rules"
- qwertyuiop924 10y ago
  Yes. I wish people would explain their hatred. I understand that there are valid reasons to hate systemd, which is why I avoid it like the plague. But so many people get sucked in by the initial usefulness, and don't see the problems lurking beneath the surface.
  - 5ilv3r 10y ago
    Binary only logs was enough for me to hate it.
    - qwertyuiop924 10y ago
      That's it? Man that's just the start...
  - fapjacks 10y ago
    Yes, a large part of which is the process behind systemd. You're adopting more than just software when you adopt software, and I think investigating what the systemd developers are like is a worthy exercise -- in particular how they approach bugs. I think most people would be surprised to learn what that is like.
    - qwertyuiop924 10y ago
      Ah, yes: When is a bug not a bug? According to the systemd developers, it's when we don't encounter it on our personal machines.
      - fapjacks 10y ago
        ... And also when they can pass the buck to kernel developers, even when it's not a kernel bug.
        digi_owl 10y ago
        Reminds me that i ran into an email chain a while back where the kernel devs pondered forking or taking over udev development because they were tired of power games from the maintainer. This was from back before udev became part of systemd btw.
        qwertyuiop924 10y ago
        Greg-KH is both one of the original developers on udev and a major force behind systemd. He maintains -stable on the linux kernel. So either the kerbel devs did take over udev development...
        digi_owl 10y ago
        I do wonder if the kernel will split between GregKH and T'so once Torvalds steps down. BTW GKH was oddly absent in that thread, and the kernel devs ire was aimed at Sievers...
        qwertyuiop924 10y ago
        Well, GKH is One Of Them. He's less likely to attract their ire, and if he does, at the end of the day, they have to answer to him. But in a fight between GKH and T'so, I'll take T'so. /dev/random's ...interesting design aside, I frankly trust T'so far more than I trust GKH, who backs LP, a person I emphatically do not trust, in a lot of things.

justin_vanw 10y ago

It seems like it is quite fashionable to hate on systemd, and it seems like systemd is kindof a piece of crap - in some ways. However, linux is missing basic functionality other os's offer, and systemd is showing up and trying to fill in those blanks. This is open source, if you don't like systemd, if you think it's crap, if you think there are obviously better ways to do it, well, what are you waiting for then? Systemd is winning because they showed up and basically nobody else did. It's open source fork it and 'do it right'. Or point out the problems and hope that someone else will step up instead? I think the world would better place if people spent less time griping and more time fighting to get their patch that fixes the problems merged. I guess coding is harder than writing blog posts though. EDIT: I'm not saying systemd is perfect or even good. My point is that the people writing the code are going to have the final say, whether it's right or wrong, and it is unlikely that people writing blog posts lampooning it are really going to make a difference. I mean systemd has been despised and highly criticized from the beginning, and look where we are.

korethr 10y ago
> However, linux is missing basic functionality other os's offer, and systemd is showing up and trying to fill in those blanks. I'm curious what functionality other OSes had that Linux did not before systemd came along. It seems to me that it has been subsuming functions of other systems on Linux that were already present, as opposed to offering anything new. Socket activation? inetd/xinetd. Timed start of programs? Cron and At Logging? rsyslog/syslog-ng/etc. DNS? -- dnsmasq, powerdns, bind, etc. Honestly, I think if system had stopped at doing init, and focused solely on that and doing the best damned possible job of it, there'd be less hate. I think what's pissing off a lot of the systemd haters is the scope creep of the project has undergone. I was personally fine with the idea of systemd when all it was promising was init and service management. But then oh, were going to take over /dev, and cron and logging, and logins and DNS and (to be determined). Ahh, thanks, but no thanks.
- bjourne 10y ago
  > I'm curious what functionality other OSes had that Linux did not before systemd came along. Uniform daemon/service handling. Just look at the scripts in /etc/init.d (or wherever they are located in your particular SysV init setup). It's a total mess. The rest of what systemd offers I can't comment on because I'm not familiar with it. But it is clear to me that systemd's declarative .service files are a billion times better than shell scripts.

milankragujevic 10y ago

Archived page, since it's not loading for me... http://archive.is/8xUSY http://archive.is/8xUSY

mavdi 10y ago
How to crash a blog with one submission
agwa 10y ago
My hosting provider thought I was under DDoS attack and started aggressively throttling port 443. I failed over to my backup provider, probably permanently.

belovedeagle 10y ago

This is an excellent writeup explaining without hysterics why systemd is terrible software. Unfortunately, I expect that this post will be brigaded by systemd-apologists anyways, denying newcomers to the debate the opportunity to hear the facts...

andrewstuart2 10y ago
I'm not sure how this comment is helpful to the discussion. As far as I can tell, you'd rather only hear the facts that fit your opinion of systemd as terrible software. I think the obvious right answer is that both sides present only facts and let newcomers decide based on that.
- fapjacks 10y ago
  I completely agree with you. That would be nice, if people were presented with facts and a choice. But they weren't, and aren't.
  - andrewstuart2 10y ago
    Well, it's never too late to start. :-) As history shows, you can try to suppress truth for a while, but it is usually either unstoppable, or insignificant.
- belovedeagle 10y ago
  As the other reply said, the problem is that newcomers aren't allowed to hear the anti-systemd facts, as the systemd gods have already decided that all Linux users (or at least all distros) must bow down and accept them, and no dissenting opinion is allowed to stay on the front page of HN for long (this post has already been brigaded off the front page, apparently).
  - fapjacks 10y ago
    It could be the "controversial topic" penalty, which is extremely heavy-handed. But I also notice that both anti-systemd and pro-other-init-system posts and comments are barraged with downvotes on all kinds of forums, including HN. It is definitely something that certain people don't want out in the world.

geofft 10y ago

I've been thinking for a bit that it's reasonably easy to write a PID 1 that reaps children and sends siginfo_t structs over a pipe to PID 2, which can be an init system as monolithic as you want. Of course, it would also be simple for the kernel to allow PID 1 to die/crash, and just restart it instead of panicking. I don't know why the kernel continues to treat PID 1 so specially—if the ewontfix.com PID 1 is sufficient, it's simpler for the kernel not to bother with the reparenting logic and stop treating PID 1 as a special process.

mschuster91 10y ago
> I don't know why the kernel continues to treat PID 1 so specially By design all processes inherit from PID 1 (directly or indirectly), so you always have an unbroken tree from every process back to PID 1. PID1 aka init is supposed to do bookkeeping (e.g. zombie processing), and if PID1 crashes the bookkeeping records are lost. Because of this, it is better to fail hard, instead of allowing for corrupted process trees or similar. Also, ps, top and basically every program that works with processes expects one root process for the tree. Breaking this would basically break compatibility with every existing userland (which the kernel cares the most about!)
- justinsaccount 10y ago
  > Breaking this would basically break compatibility with every existing userland By "Breaking this" you mean "fixing the fact that pid 1 can't crash and restart"? > Because of this, it is better to fail hard It can't be fixed because it would "break compatibility" where that "compatibility" is that the machine should panic? You're not really making a good case for why this can't be changed. Why not * pid 1 exits * kernel re-executes init as a new pid 1 * userspace continues on as if nothing happened.
  - geofft 10y ago
    Right. If init opts into it, e.g. by setting some sysctl or something, the kernel changes behavior in the following way: 1. Processes that are reparented to init aren't actually considered children of init for the purposes of the wait() system call. Their ppid still becomes 1, so parent process walks work, but when they exit, the kernel immediately cleans them up. Zombies still exist in this change, as long as their parent still exists; the only thing that changes is that processes (zombies or not) are not reparented to init when their parent dies. Direct children of pid 1 are not treated specially. 2. When init is about to exit or crash, the kernel turns it into a zombie. It then starts a new init process and reparents the old init to the new one, allowing it to do things depending on the cause of the crash. Direct children of the old init have lost their parent, and are treated like any other parentless process as described above. 3. If execing the new init fails, the kernel panics. This would be completely backwards-compatible. (I might even argue against 3, to remove all worry about init crashing; since pid 1 is no longer required to do anything, if exec fails, the kernel can just leave pid 1 as a stuck process table entry, let the system continue running, and log something to dmesg so the sysadmin can diagnose the problem and reboot the machine cleanly.)

5ilv3r 10y ago

Ran as a normal user on debian jessie, and now ssh logins are delayed by 30 seconds. I don't see any other problems yet, but this is a VM host and so leave it running for a while for fun. :)

aorth 10y ago
Ran it on an up to date Ubuntu 16.04 and lost the ability to run any systemctl commands like status, restart, etc. Web server and other system services were still working, but I couldn't `reboot` or `systemctl halt` etc... had to hard reboot the VPS from hosting console. :)
- andreyv 10y ago
  "systemctl reboot --force --force" should work in such cases. The reboot will be executed by systemctl itself.

rcarmo 10y ago

First they came for PID 1, and we were silent. Oh, wait, we weren't. But although I don't miss the rickety init scripts, I've grown more and more worried about stuff like this, enough to ponder replacements (in Go or Python, which seem to be generally acceptable for systems software these days). Ignoring for the moment the usual biases against Python, how feasible/how much support would it be/have to build something to replace systemd (by having PID 1 do the minimum possible, in C, and having the rest done with Python)?

djsumdog 10y ago
I've been working at a Docker shop for a while and I have a feeling it will be the future for running services. It makes systemd somewhat irrelevant. There are still a lot of issues I've had with Docker, which I won't go into. It also can't replace everything running on your Linux system. But it's a neat idea and big players use it in production for huge workloads. Now the whole AWS-containers, docker-machine, swarm, mesos, coreos, kubernetes thing is a cluster fuck of choices .. but you have choices. And unlike running an Ansible script on some provides base image (which is totally different from Rackspace, Linode, etc.), you are guaranteed your container will work on any docker engine with little modification (your orchestration files may change though). I have a feeling, for better or worse, containers will be the future and will stick around long after systemd hopefully dies.
- digi_owl 10y ago
  Nah, systemd is angling to supplant docker for containers. There is already a great deal of bad blood between the two because of the reworked cgroups management. And there is also the likes of CoreOS that is basically systemd-the-distro. Never mind that we have the likes of xdg-app/flatpak that is basically aiming for what you are thinking about, and it is systemd dependent.
  - fridsun 10y ago
    It was quite amusing seeing systemd causing systemd-specific trouble in CoreOS development though, such as systemd requiring too much privilege when running inside container. Many people just use s6 happily.

deleted 10y ago

[deleted]

packetized 10y ago

Erm, this does not actually seem to work on Centos7, running systemd-219-19.el7_2.13. Even tried it as root inside of a sandbox - nothing.

lake99 10y ago
It's the same on Archlinux, systemd-231. Nothing happened. I was able to start and stop a couple of services as quickly as before.
SysArchitect 10y ago
while true; do NOTIFY_SOCKET=/run/systemd/notify systemd-notify ""; done Try this. As soon as you loop it, the bug is hit pretty quickly.
- deleted 10y ago
  [deleted]
- nilved 10y ago
  I don't understand why that makes a difference if the bug is `assert(n > 0)`
  - SysArchitect 10y ago
    I have no idea either. Race condition somewhere? It was mentioned here: https://github.com/systemd/systemd/issues/4234 https://github.com/systemd/systemd/issues/4234
  - JdeBP 10y ago
    There is a race in the systemd-notify mechanism that means that some notification messages get discarded if the systemd-notify process happens to complete its work and exit before systemd picks up the message. They never reach the point of the assertion that is triggered here, because systemd is unable to determine the service unit that encompasses the message sender. * http://jdebp.eu./FGA/unix-daemon-readiness-protocol-problems.html#SynchronousProtocol http://jdebp.eu./FGA/unix-daemon-readiness-protocol-problems...

xemdetia 10y ago

At this point in time my biggest frustration is that even compared to upstart writing concise and clear config files for network services is just really hard, and there isn't great documentation or a short book around that covers all the edge cases other than TIAS. I feel like I've wasted more time trying to get a tidy systemd configuration than I ever had even with upstart or some of the other ones.

bitmapbrother 10y ago

>Despite the banality, the bug is serious, as it allows any local user to trivially perform a denial-of-service attack against a critical system component. Is it really that serious of a bug? You have to be a local user in order to execute it. Are other OS's somehow resilient to the actions of local user to degrade the OS? >The Linux ecosystem has fallen behind other operating systems in writing secure and robust software. While Microsoft was hardening Windows and Apple was developing iOS, open source software became complacent. He must be unaware of the numerous patches Microsoft and Apple issue on a regular basis to fix their "secure and robust software".

falcolas 10y ago
Any remote code execution bug can exploit this. Normally, a remote execution code bug when the user is non-root isn't a huge issue, because the user effectively sandboxes them away. This bypasses all of these benefits. Containers may prove to be a sufficient sandbox, assuming this is not executable against kdbus (which is shared across containers since it's in the kernel).

redbeard0x0a 10y ago

This article got me wondering if maybe the NSA or similar has somebody on the inside of the systemd project that is helping the project along. Systemd has replaced a lot of init systems in the linux ecosystem and the general development practices of the project is leaving a huge footprint of code that might be exploited. If they aren't involved, I'm sure they probably know a few different exploits for the system already.

pdkl95 10y ago
If you want to go down that rabbit hole, see: https://igurublog.wordpress.com/2014/02/17/biography-of-a-cypherpunk-and-how-cryptography-affects-your-life/ https://igurublog.wordpress.com/2014/02/17/biography-of-a-cy...
johnfjacobi 10y ago
I've heard this a lot and first just dismissed it as a conspiracy theory. Is there actually any substance to these thoughts, or are they only speculation?

xaduha 10y ago

Don't really care about that bug, it will be fixed I'm sure. But thank you for this post anyway, the word LLMNR caught my eye, very nice to have if it works. EDIT: works fine with Windows, without installing Bonjour/iTunes.

deleted 10y ago

[deleted]

SysArchitect 10y ago

The issue has a bug report on the systemd repo on Github: https://github.com/systemd/systemd/issues/4234 https://github.com/systemd/systemd/issues/4234

deleted 10y ago

[deleted]

walter_bishop 10y ago

Doesn't work here on Ubuntu 16.04 xenial

Twirrim 10y ago
I've put a comment on the github bug with an updated PoC. IT does work on 16.04, but you seem to have to do it over and over until it triggers. A working PoC seems to be: while true; do NOTIFY_SOCKET=/run/systemd/notify systemd-notify ""; done

zzxxmm11aajj 10y ago

Note: 112 comments from the peanut gallery before someone looked at the code enough to submit the trivial fix.

m45t3r 10y ago

You could simple do a DoS attack with a local user using a fork bomb too. It would be even shorter than the command showed in this post blog. So while this is a bug, the author is overreacting just to make systemd look bad.

rascul 10y ago
your fork bomb can be mitigated with ulimit
- m45t3r 10y ago
  I am not saying that this is not a bug, just that this isn't the problem that the author is making. If you have local access, you probably can do other things if your only objective is to do a DoS.
  - reactor4 10y ago
    Your point is covered in the article. "If you think systemd doesn't need privilege separation because it only parses messages from local users, keep in mind that in the Internet era, local attacks tend to acquire remote vectors. Consider Shellshock, or the presentation at this year's systemd conference which is titled "Talking to systemd from a Web Browser.""

aorth 10y ago

A CVE has been requested, albeit after publication of the bug: http://www.openwall.com/lists/oss-security/2016/09/28/9 http://www.openwall.com/lists/oss-security/2016/09/28/9

JdeBP 10y ago
Discussed on Hacker News at https://news.ycombinator.com/item?id=12600994 https://news.ycombinator.com/item?id=12600994 .

wyldfire 10y ago

> The immediate question raised by this bug is what kind of quality assurance process would allow such a simple bug to exist for over two years (it was introduced in systemd 209). Isn't the empty string an obvious test case? All bugs are terribly easy to spot in hindsight. I don't think it's fair to criticize this one in that fashion. Systemd has lots of pros and cons, but the age of this bug isn't interesting IMO. Its severity is, but that only speaks to the critical need to patch it ASAP.

JdeBP 10y ago

It's interesting to see a "hacker" discussion of this that entirely misses several topics that one would expect programmers to at least mention: * Whether release builds should have assert() enabled. * Whether a process #1 program should use assertions at all. * Whether assert() is the right mechanism to use for input validation. * How a process #1 program should recover from assertion failures.

fridsun 10y ago

Lennart Poettering has replied on Reddit. https://www.reddit.com/r/linux/comments/54yfcd/how_to_crash_systemd_in_one_tweet/d86rs74 https://www.reddit.com/r/linux/comments/54yfcd/how_to_crash_... > So let's summarize this. There's a bug in some software. OMG! Shock! This of course never happened before! > The bug is in not exploitable remotely. The bug does not allow privilege escalation nor insertion of code. This of course makes the bug a massive vulnerability like there was no other on the planet ... ever. As bad as heartbleed multiplied by the Debian OpenSSL random generator bug to the power of 10. > The bug is caused by an error check that filters out garbage sent to PID 1. The check works correctly, except that the resulting action is a too harsh: instead of complaining and dropping it will abort the process. Such a bug is of course unprecedented and the authors of said software should be stoned and flogged right away given the severity of the issue: after all a safety check worked a bit too well, and we really can't have that because undefined behaviour of course would be a lot better than a local DoS. > The project the error was found in is large. Yet the number of CVEs collect so far is pretty small comparing it witht other projects of similar extent. Given that another bug was discovered now this obviously shows how incompetent the programmers are and that security is a unknown concept to them. > The program the bug was found in is longer than 50 lines of code but runs with privileges, all written in a low-level programming langauge that many call little more than a fancy macro assembler. The code runs on top of an operating system kernel written in the same language but running with a lot higher privileges and consisting of expoentially more lines of code including drivers of questionnable quality. This together is of course proof that the project at hand is flawed conceptually to its core. > Dha! > Lennart > (More seriously: yes this is a bug, we should fix it. But it's very low impact and the bruhaha it generated appears wildly out of scale. If all bugs in the wider Open Source ecosystem would have a similarly low impact we'd live in a much much safer world!)

JdeBP 10y ago

Lennart Poettering's response to this was interesting: > The bug is caused by an error check that filters out garbage sent to PID 1. The check works correctly, except that the resulting action is a too harsh: instead of complaining and dropping it will abort the process. * https://www.reddit.com/r/linux/comments/54yfcd/how_to_crash_systemd_in_one_tweet/d86rs74 https://www.reddit.com/r/linux/comments/54yfcd/how_to_crash_... In fact, the opposite was the case. The root of the problem was identified in the GitHub bug report: * https://github.com/systemd/systemd/issues/4234#issuecomment-250431337 https://github.com/systemd/systemd/issues/4234#issuecomment-... * hhttps://github.com/systemd/systemd/commit/d875aa8ce10b458dc218c0d98f4a82c8904d6d03 https://github.com/systemd/systemd/commit/d875aa8ce10b458dc2... What happened was that Lennart Poettering removed an error check that filtered out zero-length messages. This left the flow of control to fall through to a later point where an assertion, that had been earlier added by Lennart Poettering with the assumption that this check for zero was in place (as it had been at the time), then triggered. Of course, no-one that I have seen has yet asked why "" as a command-line argument to systemd-notify results in a zero-length message in the first place. After all, according to the doco that would be length 1, a single terminating LF byte, not zero length. That said, the server still should be proof against zero-length network input.