5 ms·

Which was ultimately caused by a bug in the kernel, and the systemd developer ultimately accepted the proposed changes just to avoid any kind of confusion (to m

Which was ultimately caused by a bug in the kernel, and the systemd developer ultimately accepted the proposed changes just to avoid any kind of confusion (to me, it looked perfectly acceptable to parse the kernel command-line, multiple applications still do for other purposes). Arguably, the tone of the discussion resembles more the usual LKML tone rather than the one observable on the systemd mailing list.

yellowapple 11y ago
I fail to see how systemd hijacking a kernel boot argument for its own purposes is a "bug in the kernel".
- the_why_of_y 11y ago
  http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01537.html http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01537.html
  - yellowapple 11y ago
    That says nothing about why the systemd folks are entitled to hijacking a kernel flag instead of (more sanely) namespacing their own debug flag, which is the actual bug that was filed (and in turn met with "it's not a bug it's a feature, so go whine to the kernel folks about the thing of theirs that we broke and don't feel like fixing", as if the systemd folks have become one with the Microsoft). Whether or not there are other underlying issues stemming from such misbehavior is irrelevant to the fact that the misbehavior exists. It's like sticking your hand in a fire, then blaming your hand instead of your own stupidity for your third-degree burns.
    - EmanueleAina 11y ago
      Honestly I find that a single global "debug" option would have been useful, instead of having to know which component (the kernel, the ramdisk, sysvinit, sysv-rc, systemd, upstart, udev, whatever) is at fault. Unfortunately such mechanism introduced a feedback loop in the presence of a bug in the kernel. I understand that it would be difficult for the kernel to guard from such loops, but my sincere impression is that they threw out the the baby with the bathwater.
      - yellowapple 11y ago
        I agree that such a flag would be useful, too, in which case the systemd folks could very well have consulted with the kernel folks in advance to work out a deprecation path for the kernel-only behavior and an introduction path for a generic behavior that activates the respective kernel-specific and systemd-specific flags. The problem is that such collaboration didn't happen, and the systemd folks assumed that they were the center of the universe and decided "well fuck the kernel, we're going to use the 'debug' flag for ourselves, everything else be damned", thus causing breakage and flamewars. Had they taken the 10 seconds to write up an email on LKML or somesuch saying "hey, we think the 'debug' flag should apply to everything, not just the kernel; y'all game?", the vast majority of the fallout would have never existed.
        EmanueleAina 11y ago
        > the systemd folks assumed that they were the center of the universe and decided "well fuck the kernel, we're going to use the 'debug' flag for ourselves, everything else be damned" Note that even Linus said that "parsing [the 'debug' and 'quiet' flag] and doing something sane with them is not a bug, it's a feature". So the issue revolved on what people believe is "sane", not about userspace using kernel bootparams (systemd still does and Linus is ok with it). The systemd folks argued that the kernel should not affected if something broke and started throwing too many messages at it during early boot (indeed, the kernel was fixed to rate limit in those cases). The kernel people instead said that it's not ok for userspace to flood the kernel, not even in broken setups (indeed, systemd was fixed to stop sending messages to dmesg as soon as journald is up). I believe that sending emails here and there to ask "it's ok to do that?" would not work. LKML is already high traffic as is, such messages would just get ignored until things break.
    - the_why_of_y 11y ago
      > That says nothing about why the systemd folks are entitled to hijacking a kernel flag instead of (more sanely) namespacing their own debug flag "No, we very much expose /proc/cmdline for a reason. System services are supposed to parse it, because it gives a unified way for people to pass in various flags. [...] And yes, that does include "quiet" and "debug". Parsing them and doing something sane with them is not a bug, it's a feature." http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01488.html http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01488.html > which is the actual bug that was filed (and in turn met with "it's not a bug it's a feature, And that is the appropriate response to the bug that was filed, with the title "Do not parse "debug" command line parameter". The bug report makes it immediately clear that this is not really about a technical problem that ought to be solved - no, the dirty user-space programmers should get their stinking paws off of our good kernel command line flags, how dare they! Kay pointed out in his first comment that there is already a different parameter that turns on debug output only for the kernel; the reporter said he's too lazy to start using it. And this is not about "entitlement": there is a clear administrative use-case for systemd using the debug parameter: if your system doesn't boot you want a simple and obvious way to turn on debug logging in all of the relevant plumbing code, whether it's in the kernel, in the initrd or during service startup. You shouldn't have to remember five different parameters because then you'll forget one and your outage lasts 10 minutes longer. If the bug report had been something more like "I've enabled debug and my system failed to boot" without prescribing a particular fix and leaving that at the maintainer's discretion, it wouldn't have been resolved as NOTABUG but fixed quickly. Of course, a wiser and more socially skilled maintainer than Kay Sievers would have immediately filed a follow-up bug report, or re-titled the bug report instead of lowering himself to the level of the reporter and playing resolve-reopen ping pong. (Fortunately the systemd project also also has some more patient maintainers like Tom and Zbigniew.)
      - yellowapple 11y ago
        Read the rest of the email you linked, where Linus clarifies that systemd's use of the 'debug' parameter seemed to assume that systemd was the only user of it, thus going against the whole point of a readable /proc/cmdline. Thus, feedback loops were caused and pain was created and flamewars were started. Again, if you shove your hand into a fire, don't blame the hand (or, for that matter, the fire) for your own arrogance and stupidity. Common bloody sense.
- EmanueleAina 11y ago
  It worked before, it stopped after a change in the kernel, another change made the problem disappear: the root of the problem was definitely a bug in the kernel. That said, it's perfectly fine for the kernel developers to clarify that a flag is for their own exclusive use, and indeed it seems that this is how things ended up. Note that there are multiple userspace applications parsing the flags on the kernel command line: the most striking example when discussing the "debug" flag is the "quiet" flag.
  - weland 11y ago
    > That said, it's perfectly fine for the kernel developers to clarify that a flag is for their own exclusive use Oh for fuck's sake. It's a parameter passed to the kernel. Programs shouldn't parse it for precisely the same reason why Firefox doesn't parse sshd's arguments. I.e. because they're passed to sshd, not to Firefox. It's a ridiculous clunky hack that stinks of large company coding practice: what they actually needed was a way to pass arguments to systemd itself. The logical way to do that in an OS that has to be portable and flexible is to implement a mechanism for the kernel to pass some of the arguments it receives to init. Because there's no reason to assume that init is going to be systemd, that there's going to be a procfs ready for it so as to be able to read the kernel's parameters, or that the init process can read it for that matter. The way the systemd development team chose to do that (widely seen in many other projects from the same lineage, like Pulseaudio) was ha look at that we can see the kernel's arguments so how about we use those and what do you mean "this can have unwanted side-effects", it works perfectly fine on my system. The fact that the developers (again, it's a known pattern) decided to play the PR game and call it "expected behaviour" doesn't make it less of a bug. It's also expected that a division by zero will crash your program. It doesn't mean that blindly accepting any input values is not a bug, nor that the correct way to solve "your program crashes if I pass 0 as the first argument" is "well stop passing 0 as a first argument ya evil bastard."
    - EmanueleAina 11y ago
      > It's a parameter passed to the kernel. Just like "quiet", which is totally meant to affect userspace programs. > It's a ridiculous clunky hack that stinks of large company coding practice: what they actually needed was a way to pass arguments to systemd itself. Having a global "debug" knob would have been useful when one does not know if the problme lies in the ramdisk, plymouth, systemd, udev, lvm, whatever. Having a systemd-specific flag defeats that purpose. > The logical way to do that in an OS that has to be portable and flexible is to implement a mechanism for the kernel to pass some of the arguments it receives to init. Because there's no reason to assume that init is going to be systemd, that there's going to be a procfs ready for it so as to be able to read the kernel's parameters, or that the init process can read it for that matter. The kernel command line, readable from procfs, has been used for this purpose (see the "quiet" flag) for a very long time now. > what do you mean "this can have unwanted side-effects", it works perfectly fine on my system. It worked perfectly on every system until the kernel bug triggered the feedback loop. > The fact that the developers (again, it's a known pattern) decided to play the PR game and call it "expected behaviour" doesn't make it less of a bug. It was "expected behaviour" in the presence of a kernel bug, which is why it was obviously not generally "expected behaviour" as you seem to imply.
weland 11y ago
If this one isn't satisfying, digi_owl has a nice one, too: https://news.ycombinator.com/item?id=9723476 https://news.ycombinator.com/item?id=9723476
- EmanueleAina 11y ago
  Never heard about it, is there any link with more details?
  - weland 11y ago
    https://bugzilla.redhat.com/show_bug.cgi?id=753882 https://bugzilla.redhat.com/show_bug.cgi?id=753882 Enjoy.
    - EmanueleAina 11y ago
      Interesting read, thanks! However I would not characterize that as "Poettering completely misunderstanding a perfectly valid use of su". It seems a quite complex issue about a workflow that is only half-working even without systemd. A systemd patch has been committed that fixes the most blatant error, but of course it doesn't solve all problems since most of them existed before systemd.
      - weland 11y ago
        I don't know in what corner of the universe was that "only half-working". It was working absolutely fine. It has been working fine since... I don't know, v7, when it was first introduced.
        EmanueleAina 11y ago
        The few times I tried to run GUI programs through su I always got plenty of warning about DBus and stuff. But I don't do that very often, so that's all I know.

yellowapple 11y ago

I fail to see how systemd hijacking a kernel boot argument for its own purposes is a "bug in the kernel".

the_why_of_y 11y ago
http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01537.html http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01537.html
- yellowapple 11y ago
  That says nothing about why the systemd folks are entitled to hijacking a kernel flag instead of (more sanely) namespacing their own debug flag, which is the actual bug that was filed (and in turn met with "it's not a bug it's a feature, so go whine to the kernel folks about the thing of theirs that we broke and don't feel like fixing", as if the systemd folks have become one with the Microsoft). Whether or not there are other underlying issues stemming from such misbehavior is irrelevant to the fact that the misbehavior exists. It's like sticking your hand in a fire, then blaming your hand instead of your own stupidity for your third-degree burns.
  - EmanueleAina 11y ago
    Honestly I find that a single global "debug" option would have been useful, instead of having to know which component (the kernel, the ramdisk, sysvinit, sysv-rc, systemd, upstart, udev, whatever) is at fault. Unfortunately such mechanism introduced a feedback loop in the presence of a bug in the kernel. I understand that it would be difficult for the kernel to guard from such loops, but my sincere impression is that they threw out the the baby with the bathwater.
    - yellowapple 11y ago
      I agree that such a flag would be useful, too, in which case the systemd folks could very well have consulted with the kernel folks in advance to work out a deprecation path for the kernel-only behavior and an introduction path for a generic behavior that activates the respective kernel-specific and systemd-specific flags. The problem is that such collaboration didn't happen, and the systemd folks assumed that they were the center of the universe and decided "well fuck the kernel, we're going to use the 'debug' flag for ourselves, everything else be damned", thus causing breakage and flamewars. Had they taken the 10 seconds to write up an email on LKML or somesuch saying "hey, we think the 'debug' flag should apply to everything, not just the kernel; y'all game?", the vast majority of the fallout would have never existed.
      - EmanueleAina 11y ago
        > the systemd folks assumed that they were the center of the universe and decided "well fuck the kernel, we're going to use the 'debug' flag for ourselves, everything else be damned" Note that even Linus said that "parsing [the 'debug' and 'quiet' flag] and doing something sane with them is not a bug, it's a feature". So the issue revolved on what people believe is "sane", not about userspace using kernel bootparams (systemd still does and Linus is ok with it). The systemd folks argued that the kernel should not affected if something broke and started throwing too many messages at it during early boot (indeed, the kernel was fixed to rate limit in those cases). The kernel people instead said that it's not ok for userspace to flood the kernel, not even in broken setups (indeed, systemd was fixed to stop sending messages to dmesg as soon as journald is up). I believe that sending emails here and there to ask "it's ok to do that?" would not work. LKML is already high traffic as is, such messages would just get ignored until things break.
  - the_why_of_y 11y ago
    > That says nothing about why the systemd folks are entitled to hijacking a kernel flag instead of (more sanely) namespacing their own debug flag "No, we very much expose /proc/cmdline for a reason. System services are supposed to parse it, because it gives a unified way for people to pass in various flags. [...] And yes, that does include "quiet" and "debug". Parsing them and doing something sane with them is not a bug, it's a feature." http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01488.html http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01488.html > which is the actual bug that was filed (and in turn met with "it's not a bug it's a feature, And that is the appropriate response to the bug that was filed, with the title "Do not parse "debug" command line parameter". The bug report makes it immediately clear that this is not really about a technical problem that ought to be solved - no, the dirty user-space programmers should get their stinking paws off of our good kernel command line flags, how dare they! Kay pointed out in his first comment that there is already a different parameter that turns on debug output only for the kernel; the reporter said he's too lazy to start using it. And this is not about "entitlement": there is a clear administrative use-case for systemd using the debug parameter: if your system doesn't boot you want a simple and obvious way to turn on debug logging in all of the relevant plumbing code, whether it's in the kernel, in the initrd or during service startup. You shouldn't have to remember five different parameters because then you'll forget one and your outage lasts 10 minutes longer. If the bug report had been something more like "I've enabled debug and my system failed to boot" without prescribing a particular fix and leaving that at the maintainer's discretion, it wouldn't have been resolved as NOTABUG but fixed quickly. Of course, a wiser and more socially skilled maintainer than Kay Sievers would have immediately filed a follow-up bug report, or re-titled the bug report instead of lowering himself to the level of the reporter and playing resolve-reopen ping pong. (Fortunately the systemd project also also has some more patient maintainers like Tom and Zbigniew.)
    - yellowapple 11y ago
      Read the rest of the email you linked, where Linus clarifies that systemd's use of the 'debug' parameter seemed to assume that systemd was the only user of it, thus going against the whole point of a readable /proc/cmdline. Thus, feedback loops were caused and pain was created and flamewars were started. Again, if you shove your hand into a fire, don't blame the hand (or, for that matter, the fire) for your own arrogance and stupidity. Common bloody sense.
EmanueleAina 11y ago
It worked before, it stopped after a change in the kernel, another change made the problem disappear: the root of the problem was definitely a bug in the kernel. That said, it's perfectly fine for the kernel developers to clarify that a flag is for their own exclusive use, and indeed it seems that this is how things ended up. Note that there are multiple userspace applications parsing the flags on the kernel command line: the most striking example when discussing the "debug" flag is the "quiet" flag.
- weland 11y ago
  > That said, it's perfectly fine for the kernel developers to clarify that a flag is for their own exclusive use Oh for fuck's sake. It's a parameter passed to the kernel. Programs shouldn't parse it for precisely the same reason why Firefox doesn't parse sshd's arguments. I.e. because they're passed to sshd, not to Firefox. It's a ridiculous clunky hack that stinks of large company coding practice: what they actually needed was a way to pass arguments to systemd itself. The logical way to do that in an OS that has to be portable and flexible is to implement a mechanism for the kernel to pass some of the arguments it receives to init. Because there's no reason to assume that init is going to be systemd, that there's going to be a procfs ready for it so as to be able to read the kernel's parameters, or that the init process can read it for that matter. The way the systemd development team chose to do that (widely seen in many other projects from the same lineage, like Pulseaudio) was ha look at that we can see the kernel's arguments so how about we use those and what do you mean "this can have unwanted side-effects", it works perfectly fine on my system. The fact that the developers (again, it's a known pattern) decided to play the PR game and call it "expected behaviour" doesn't make it less of a bug. It's also expected that a division by zero will crash your program. It doesn't mean that blindly accepting any input values is not a bug, nor that the correct way to solve "your program crashes if I pass 0 as the first argument" is "well stop passing 0 as a first argument ya evil bastard."
  - EmanueleAina 11y ago
    > It's a parameter passed to the kernel. Just like "quiet", which is totally meant to affect userspace programs. > It's a ridiculous clunky hack that stinks of large company coding practice: what they actually needed was a way to pass arguments to systemd itself. Having a global "debug" knob would have been useful when one does not know if the problme lies in the ramdisk, plymouth, systemd, udev, lvm, whatever. Having a systemd-specific flag defeats that purpose. > The logical way to do that in an OS that has to be portable and flexible is to implement a mechanism for the kernel to pass some of the arguments it receives to init. Because there's no reason to assume that init is going to be systemd, that there's going to be a procfs ready for it so as to be able to read the kernel's parameters, or that the init process can read it for that matter. The kernel command line, readable from procfs, has been used for this purpose (see the "quiet" flag) for a very long time now. > what do you mean "this can have unwanted side-effects", it works perfectly fine on my system. It worked perfectly on every system until the kernel bug triggered the feedback loop. > The fact that the developers (again, it's a known pattern) decided to play the PR game and call it "expected behaviour" doesn't make it less of a bug. It was "expected behaviour" in the presence of a kernel bug, which is why it was obviously not generally "expected behaviour" as you seem to imply.

weland 11y ago

If this one isn't satisfying, digi_owl has a nice one, too: https://news.ycombinator.com/item?id=9723476 https://news.ycombinator.com/item?id=9723476

EmanueleAina 11y ago
Never heard about it, is there any link with more details?
- weland 11y ago
  https://bugzilla.redhat.com/show_bug.cgi?id=753882 https://bugzilla.redhat.com/show_bug.cgi?id=753882 Enjoy.
  - EmanueleAina 11y ago
    Interesting read, thanks! However I would not characterize that as "Poettering completely misunderstanding a perfectly valid use of su". It seems a quite complex issue about a workflow that is only half-working even without systemd. A systemd patch has been committed that fixes the most blatant error, but of course it doesn't solve all problems since most of them existed before systemd.
    - weland 11y ago
      I don't know in what corner of the universe was that "only half-working". It was working absolutely fine. It has been working fine since... I don't know, v7, when it was first introduced.
      - EmanueleAina 11y ago
        The few times I tried to run GUI programs through su I always got plenty of warning about DBus and stuff. But I don't do that very often, so that's all I know.