7 ms·

>My question was more about the GP elaborating why a new system would be something "[no one] wants to go through this again." Because the migration was terribl

by esaym 11y ago

>My question was more about the GP elaborating why a new system would be something "[no one] wants to go through this again." Because the migration was terrible that is why. And I still continue to find stuff I hate about systemd. The latest was just yesterday, while troubleshooting some disk issues I wanted to fsck root before rebooting. Which normally means just remount root as read-only. Tried that and it didn't work, "disk was busy". Jumped into runlevel 1, still disk was busy. Rebooted into rescue mode, remount ro, still disk busy. After googling and digging through lsof, I could see it was systemd processes and journal that was holding onto files and preventing a remount. Tried killing systemd, journald, even kill -9, it just restarts it self and holds the disk. Darn. Did the touch /forcefsck trick, rebooted, yet no fsck. Darn again. More googling later, I see you have pass kernel parameters at boot time to force a fsck, and no, you can't select which FS either, it just does all of them. Rebooted again, fsck is running of course. I just wanted root checked but now it is doing all my terabyte partitions as well. Gave up at this point and just left the machine and hoped for the best.

lisivka 11y ago
I hate it too, but workaround is not too hard: chmod a-x journald && pkill journald of course, it is bad idea to touch damaged FS before fsck.
- simoncion 11y ago
  Wait. To run an unscheduled fsck of your root fs (and only your root fs), you have to remove the execute bit from a systemd component, kill that component, then remember to reset the bit after you're done? That's nuts.
  - morganvachon 11y ago
    That kind of thing is why I stick with Slackware and OpenBSD for my production systems. systemd has some great ideas, but it has some down right silliness as well, and I personally don't feel it's stable enough for daily use yet.
  - rdtsc 11y ago
    (Just to make sure everyone gets it) It will prevent it from running again. So it is basically a big FU! to systemd. "So you you think you gonna restart yourself? Oh, no you won't"
  - pdkl95 11y ago
    That's nothing - systemd panics (pid1 crash) if you restart dbus, which requires either the Magic SysRq or the reset button.. https://forums.gentoo.org/viewtopic-p-7811550.html#7811550 https://forums.gentoo.org/viewtopic-p-7811550.html#7811550 Pottering was apparently told about this in 2013, when he simply declared it expected behavior because dbus was "rock solid". http://lists.freedesktop.org/archives/systemd-devel/2013-January/008456.html http://lists.freedesktop.org/archives/systemd-devel/2013-Jan... Incidentally, this might be the original reason for kdbus - it's harder to restart the daemon after it becomes a kernel module.
    - simoncion 11y ago
      > Incidentally, this might be the original reason for kdbus... Yeah, especially given that its stated reason (Moving into the kernel gives us a 2x speedup over userspace dbus! Dbus is slow and we NEEED the speedup!) is mooted by the 10x speedup that cleaning up the userspace dbus libraries has achieved. [0] Having said that, in the systemd-devel post Poettering says that they're working on... > ...improving this in two ways: even if it still brings down the system, at least allow logind to handle this case nicely, so that you get a sane getty. [The other way is to push dbus into the kernel.] Systemd is a sprawling project, so I could see why it has taken more than two-and-a-half years to fix a local DOS caused by the perfectly normal method of responding to a DBUS upgrade. ;) [0] Fun fact: Those cleaned up libraries were supposed to be released after kdbus was merged into the kernel in 4.1. The Systemd Cabal was so disappointed that they had to ship the performant userspace dbus libs before the dbus daemon got pushed into kernelspace.
      - tmd83 11y ago
        I am by no means knowledgable in the relevant domain yet this is scary at so many levels. The team that is taking charge of so many critical infrastructure not doing a minimal level of fact finding before pushing for a new kernel module for RPC is a bit obscene. I may have read things that are biased but I have a feeling it is a useful project and it does a lot of thing elegantly or right. But the project acts like it is the center of the universe and all should act to meet its needs. I have seen such project/modules at work, they usually makes everyone happy to start with and down the road when they finally get stuck and I am forced to consult we are already long way down the slope.
- e12e 11y ago
  As mentioned above, that doesn't seem needed [ed: nm, it is - if journald is set to log somewhere other than /run that's usually mounted as a tmpfs]. [S]urely, something like: mount -t tmpfs /run && pkill journald [ed: that is, for logs on /var/log: "mount -t tmpfs /var/log && pkill journald"] would make more sense? (In my experience systemd itself does something funky on / -- so the above wouldn't be enough -- but still seem a little more sane than running a chmod on a file (presumably) residing in an fs you want to fsck).
- kbenson 11y ago
  Actually, the workaround is to reboot with kernel param "emergency". It's single user mode with the root helpfully already in read-only mode.
  - esaym 11y ago
    The OS is Debian 8. I seemed to think the same too, but as I mentioned, rebooting into single user mode did no such thing.
    - JdeBP 11y ago
      You mentioned booting into rescue mode. Rescue mode is not emergency mode. Have a nearly two decades old explanation that still holds true for systems like nosh and systemd today: > If you need to check the root file system, you can reboot into single-user mode with the root partition mounted read-only by issuing the -b switch at the LILO prompt. The -b switch will be passed through LILO to init and will cause an emergency boot ... -- David A. Bandel (1997-01-01). Disk Maintenance under Linux. Linux Journal. http://linuxjournal.com/article/193 http://linuxjournal.com/article/193
    - kbenson 11y ago
      That points towards either Debian implementing systemd poorly in this aspect, Debian poorly documenting well how to accomplish this after the change, or a failure on the user's part to find said documentation. Systemd on Fedora supports this perfectly fine, so I can't really call this a failure of systemd itself.
hga 11y ago
This is a job for a distro like System Rescue CD. Boot off CD, run in DRAM, figure out which partition or LV is root, and directly fsck it.
kbenson 11y ago
Maybe the problem is that you're carrying conventions from an obsolete system to a new one? A quick google search for how to force fsck with systemd comes up with results[1] mentioning options you haven't covered. Then again, I haven't had to do this, so maybe you did try those steps and it didn't work. 1: http://lmgtfy.com/?q=systemd+force+fsck http://lmgtfy.com/?q=systemd+force+fsck
- esaym 11y ago
  As far as I can see, they all mention "fsck.mode=force, as a kernel parameter", which is what I was complaining about.
  - kbenson 11y ago
    Here's the TL;DR 1) It's actually super easy to get to a read-only root mode. 2) It actually is a problem of carrying over obsolete conventions from prior init systems. 3) It's not your fault because systemd has extremely poor documentation on this AFAICT, to the point that people searching for this problem can't even get answers from others that know how to do it, or at least those answers don't rate very high. 4) Just boot with kernel param "emergency" instead of "single" (or "rescue" which is an alias for "single") and you will get a single user mode with root already mounted read-only for you. And now for the longer rambling version. Okay, so I spent a little time on this since I happen to have a virt for Fedora 20, and I finally found a couple of solutions that work. First, the systemd solution, which is emergency mode. It's correct that the journal will cause problems here, but that's because the journal was already running and apparently systemd doesn't want to, or is unable to stop it. The solution is to reboot into emergency mode. This can be accomplished by using the kernel param "systemd.unit=emergency.target", or just the param "emergency". This will not only boot you into a single user mode, but will default to having root mounted read-only. To clarify, this appears to be a new mode in addition to single user mode. Single user mode can still be reached with the kernel param "single" or "rescue" or "systemd.unit=rescue.target". I finally stumbled on this when I decided to look at the directions for what I wanted to accomplish for the distro I was using[1] and not for systemd specifically. Which makes sense, but I think we are all to stuck on "systemd is different, how do I do this in systemd" rather than "how does my distro say I should accomplish this now, given the changes they've instituted." 1: https://docs.fedoraproject.org/en-US/Fedora/18/html/Installation_Guide/ap-rescuemode.html#s1-rescuemode-booting-emergency https://docs.fedoraproject.org/en-US/Fedora/18/html/Installa...

by esaym 11y ago

lisivka 11y ago

I hate it too, but workaround is not too hard: chmod a-x journald && pkill journald of course, it is bad idea to touch damaged FS before fsck.

simoncion 11y ago
Wait. To run an unscheduled fsck of your root fs (and only your root fs), you have to remove the execute bit from a systemd component, kill that component, then remember to reset the bit after you're done? That's nuts.
- morganvachon 11y ago
  That kind of thing is why I stick with Slackware and OpenBSD for my production systems. systemd has some great ideas, but it has some down right silliness as well, and I personally don't feel it's stable enough for daily use yet.
- rdtsc 11y ago
  (Just to make sure everyone gets it) It will prevent it from running again. So it is basically a big FU! to systemd. "So you you think you gonna restart yourself? Oh, no you won't"
- pdkl95 11y ago
  That's nothing - systemd panics (pid1 crash) if you restart dbus, which requires either the Magic SysRq or the reset button.. https://forums.gentoo.org/viewtopic-p-7811550.html#7811550 https://forums.gentoo.org/viewtopic-p-7811550.html#7811550 Pottering was apparently told about this in 2013, when he simply declared it expected behavior because dbus was "rock solid". http://lists.freedesktop.org/archives/systemd-devel/2013-January/008456.html http://lists.freedesktop.org/archives/systemd-devel/2013-Jan... Incidentally, this might be the original reason for kdbus - it's harder to restart the daemon after it becomes a kernel module.
  - simoncion 11y ago
    > Incidentally, this might be the original reason for kdbus... Yeah, especially given that its stated reason (Moving into the kernel gives us a 2x speedup over userspace dbus! Dbus is slow and we NEEED the speedup!) is mooted by the 10x speedup that cleaning up the userspace dbus libraries has achieved. [0] Having said that, in the systemd-devel post Poettering says that they're working on... > ...improving this in two ways: even if it still brings down the system, at least allow logind to handle this case nicely, so that you get a sane getty. [The other way is to push dbus into the kernel.] Systemd is a sprawling project, so I could see why it has taken more than two-and-a-half years to fix a local DOS caused by the perfectly normal method of responding to a DBUS upgrade. ;) [0] Fun fact: Those cleaned up libraries were supposed to be released after kdbus was merged into the kernel in 4.1. The Systemd Cabal was so disappointed that they had to ship the performant userspace dbus libs before the dbus daemon got pushed into kernelspace.
    - tmd83 11y ago
      I am by no means knowledgable in the relevant domain yet this is scary at so many levels. The team that is taking charge of so many critical infrastructure not doing a minimal level of fact finding before pushing for a new kernel module for RPC is a bit obscene. I may have read things that are biased but I have a feeling it is a useful project and it does a lot of thing elegantly or right. But the project acts like it is the center of the universe and all should act to meet its needs. I have seen such project/modules at work, they usually makes everyone happy to start with and down the road when they finally get stuck and I am forced to consult we are already long way down the slope.
e12e 11y ago
As mentioned above, that doesn't seem needed [ed: nm, it is - if journald is set to log somewhere other than /run that's usually mounted as a tmpfs]. [S]urely, something like: mount -t tmpfs /run && pkill journald [ed: that is, for logs on /var/log: "mount -t tmpfs /var/log && pkill journald"] would make more sense? (In my experience systemd itself does something funky on / -- so the above wouldn't be enough -- but still seem a little more sane than running a chmod on a file (presumably) residing in an fs you want to fsck).
kbenson 11y ago
Actually, the workaround is to reboot with kernel param "emergency". It's single user mode with the root helpfully already in read-only mode.
- esaym 11y ago
  The OS is Debian 8. I seemed to think the same too, but as I mentioned, rebooting into single user mode did no such thing.
  - JdeBP 11y ago
    You mentioned booting into rescue mode. Rescue mode is not emergency mode. Have a nearly two decades old explanation that still holds true for systems like nosh and systemd today: > If you need to check the root file system, you can reboot into single-user mode with the root partition mounted read-only by issuing the -b switch at the LILO prompt. The -b switch will be passed through LILO to init and will cause an emergency boot ... -- David A. Bandel (1997-01-01). Disk Maintenance under Linux. Linux Journal. http://linuxjournal.com/article/193 http://linuxjournal.com/article/193
  - kbenson 11y ago
    That points towards either Debian implementing systemd poorly in this aspect, Debian poorly documenting well how to accomplish this after the change, or a failure on the user's part to find said documentation. Systemd on Fedora supports this perfectly fine, so I can't really call this a failure of systemd itself.

hga 11y ago

This is a job for a distro like System Rescue CD. Boot off CD, run in DRAM, figure out which partition or LV is root, and directly fsck it.

kbenson 11y ago

Maybe the problem is that you're carrying conventions from an obsolete system to a new one? A quick google search for how to force fsck with systemd comes up with results[1] mentioning options you haven't covered. Then again, I haven't had to do this, so maybe you did try those steps and it didn't work. 1: http://lmgtfy.com/?q=systemd+force+fsck http://lmgtfy.com/?q=systemd+force+fsck

esaym 11y ago
As far as I can see, they all mention "fsck.mode=force, as a kernel parameter", which is what I was complaining about.
- kbenson 11y ago
  Here's the TL;DR 1) It's actually super easy to get to a read-only root mode. 2) It actually is a problem of carrying over obsolete conventions from prior init systems. 3) It's not your fault because systemd has extremely poor documentation on this AFAICT, to the point that people searching for this problem can't even get answers from others that know how to do it, or at least those answers don't rate very high. 4) Just boot with kernel param "emergency" instead of "single" (or "rescue" which is an alias for "single") and you will get a single user mode with root already mounted read-only for you. And now for the longer rambling version. Okay, so I spent a little time on this since I happen to have a virt for Fedora 20, and I finally found a couple of solutions that work. First, the systemd solution, which is emergency mode. It's correct that the journal will cause problems here, but that's because the journal was already running and apparently systemd doesn't want to, or is unable to stop it. The solution is to reboot into emergency mode. This can be accomplished by using the kernel param "systemd.unit=emergency.target", or just the param "emergency". This will not only boot you into a single user mode, but will default to having root mounted read-only. To clarify, this appears to be a new mode in addition to single user mode. Single user mode can still be reached with the kernel param "single" or "rescue" or "systemd.unit=rescue.target". I finally stumbled on this when I decided to look at the directions for what I wanted to accomplish for the distro I was using[1] and not for systemd specifically. Which makes sense, but I think we are all to stuck on "systemd is different, how do I do this in systemd" rather than "how does my distro say I should accomplish this now, given the changes they've instituted." 1: https://docs.fedoraproject.org/en-US/Fedora/18/html/Installation_Guide/ap-rescuemode.html#s1-rescuemode-booting-emergency https://docs.fedoraproject.org/en-US/Fedora/18/html/Installa...