6 ms·

I wasn't born yesterday, I know that. In fact, I helped fan the flames by making uselessd, though not out of any sly intent in the least, but research curiosity

I wasn't born yesterday, I know that. In fact, I helped fan the flames by making uselessd, though not out of any sly intent in the least, but research curiosity. My question was more about the GP elaborating why a new system would be something "[no one] wants to go through this again." (I'd characterize systemd as being a sysvinit replacement to be a great misnomer. It's not. It's an entire framework for providing certain low-level userspace daemons, utilities and auxiliaries meant as a common middleware for GNU/Linux distributions. Even systemd-the-PID1 cannot be called a sysvinit replacement in any integrity.)

esaym 11y ago
>My question was more about the GP elaborating why a new system would be something "[no one] wants to go through this again." Because the migration was terrible that is why. And I still continue to find stuff I hate about systemd. The latest was just yesterday, while troubleshooting some disk issues I wanted to fsck root before rebooting. Which normally means just remount root as read-only. Tried that and it didn't work, "disk was busy". Jumped into runlevel 1, still disk was busy. Rebooted into rescue mode, remount ro, still disk busy. After googling and digging through lsof, I could see it was systemd processes and journal that was holding onto files and preventing a remount. Tried killing systemd, journald, even kill -9, it just restarts it self and holds the disk. Darn. Did the touch /forcefsck trick, rebooted, yet no fsck. Darn again. More googling later, I see you have pass kernel parameters at boot time to force a fsck, and no, you can't select which FS either, it just does all of them. Rebooted again, fsck is running of course. I just wanted root checked but now it is doing all my terabyte partitions as well. Gave up at this point and just left the machine and hoped for the best.
- lisivka 11y ago
  I hate it too, but workaround is not too hard: chmod a-x journald && pkill journald of course, it is bad idea to touch damaged FS before fsck.
  - simoncion 11y ago
    Wait. To run an unscheduled fsck of your root fs (and only your root fs), you have to remove the execute bit from a systemd component, kill that component, then remember to reset the bit after you're done? That's nuts.
    - morganvachon 11y ago
      That kind of thing is why I stick with Slackware and OpenBSD for my production systems. systemd has some great ideas, but it has some down right silliness as well, and I personally don't feel it's stable enough for daily use yet.
    - rdtsc 11y ago
      (Just to make sure everyone gets it) It will prevent it from running again. So it is basically a big FU! to systemd. "So you you think you gonna restart yourself? Oh, no you won't"
    - pdkl95 11y ago
      That's nothing - systemd panics (pid1 crash) if you restart dbus, which requires either the Magic SysRq or the reset button.. https://forums.gentoo.org/viewtopic-p-7811550.html#7811550 https://forums.gentoo.org/viewtopic-p-7811550.html#7811550 Pottering was apparently told about this in 2013, when he simply declared it expected behavior because dbus was "rock solid". http://lists.freedesktop.org/archives/systemd-devel/2013-January/008456.html http://lists.freedesktop.org/archives/systemd-devel/2013-Jan... Incidentally, this might be the original reason for kdbus - it's harder to restart the daemon after it becomes a kernel module.
      - simoncion 11y ago
        > Incidentally, this might be the original reason for kdbus... Yeah, especially given that its stated reason (Moving into the kernel gives us a 2x speedup over userspace dbus! Dbus is slow and we NEEED the speedup!) is mooted by the 10x speedup that cleaning up the userspace dbus libraries has achieved. [0] Having said that, in the systemd-devel post Poettering says that they're working on... > ...improving this in two ways: even if it still brings down the system, at least allow logind to handle this case nicely, so that you get a sane getty. [The other way is to push dbus into the kernel.] Systemd is a sprawling project, so I could see why it has taken more than two-and-a-half years to fix a local DOS caused by the perfectly normal method of responding to a DBUS upgrade. ;) [0] Fun fact: Those cleaned up libraries were supposed to be released after kdbus was merged into the kernel in 4.1. The Systemd Cabal was so disappointed that they had to ship the performant userspace dbus libs before the dbus daemon got pushed into kernelspace.
- hga 11y ago
  This is a job for a distro like System Rescue CD. Boot off CD, run in DRAM, figure out which partition or LV is root, and directly fsck it.
- kbenson 11y ago
  Maybe the problem is that you're carrying conventions from an obsolete system to a new one? A quick google search for how to force fsck with systemd comes up with results[1] mentioning options you haven't covered. Then again, I haven't had to do this, so maybe you did try those steps and it didn't work. 1: http://lmgtfy.com/?q=systemd+force+fsck http://lmgtfy.com/?q=systemd+force+fsck
  - esaym 11y ago
    As far as I can see, they all mention "fsck.mode=force, as a kernel parameter", which is what I was complaining about.
    - kbenson 11y ago
      Here's the TL;DR 1) It's actually super easy to get to a read-only root mode. 2) It actually is a problem of carrying over obsolete conventions from prior init systems. 3) It's not your fault because systemd has extremely poor documentation on this AFAICT, to the point that people searching for this problem can't even get answers from others that know how to do it, or at least those answers don't rate very high. 4) Just boot with kernel param "emergency" instead of "single" (or "rescue" which is an alias for "single") and you will get a single user mode with root already mounted read-only for you. And now for the longer rambling version. Okay, so I spent a little time on this since I happen to have a virt for Fedora 20, and I finally found a couple of solutions that work. First, the systemd solution, which is emergency mode. It's correct that the journal will cause problems here, but that's because the journal was already running and apparently systemd doesn't want to, or is unable to stop it. The solution is to reboot into emergency mode. This can be accomplished by using the kernel param "systemd.unit=emergency.target", or just the param "emergency". This will not only boot you into a single user mode, but will default to having root mounted read-only. To clarify, this appears to be a new mode in addition to single user mode. Single user mode can still be reached with the kernel param "single" or "rescue" or "systemd.unit=rescue.target". I finally stumbled on this when I decided to look at the directions for what I wanted to accomplish for the distro I was using[1] and not for systemd specifically. Which makes sense, but I think we are all to stuck on "systemd is different, how do I do this in systemd" rather than "how does my distro say I should accomplish this now, given the changes they've instituted." 1: https://docs.fedoraproject.org/en-US/Fedora/18/html/Installation_Guide/ap-rescuemode.html#s1-rescuemode-booting-emergency https://docs.fedoraproject.org/en-US/Fedora/18/html/Installa...
e12e 11y ago
[ed2: Never mind - the standard setup is to have /run mounted as a tmpfs -- I failed to check that before this song and dance] Which distribution was this? I just tried on my laptop (running Debian 8, fs on top of lvm on top of LUKS), and a simple: telinit 1 # log in on console mount -oro,remount / worked as expected. [ed: Actually just tried forcing an fsck as well, and worked without a hitch. Just remember to "mount -oremount /" before running a "telinit 5" (Nothing really bad happens if not, but the system(d) will be confused if rootfs is mounted read-only).]
- JdeBP 11y ago
  esaym probably doesn't need people to try and diagnose xyr problem, and this isn't really the best venue to do that. If you want to try this, try at least replicating esaym's setup as described rather than using your own and declaring "works for me!". One way that springs to mind is moving the journal into /var/log and having no separate /var/log volume; which reproduces the described setup of open journal files on the root filesystem. Do that and then try to force an explicit check of the root volume. Of course, knowing that does point to one obvious way to avoid the stated problem: move the journal back into /run again (per the journald.conf manual page) and simply restart journald. No mucking about with permissions required. But that's not the only thing that xe could have tried. emergency mode, for example, is documented as not mounting additional non-API filesystems at all, nor read-write remounting the root volume. See http://freedesktop.org/wiki/Software/systemd/Debugging/#bootingintorescueoremergencytargets http://freedesktop.org/wiki/Software/systemd/Debugging/#boot...
  - e12e 11y ago
    Hm, good catch. I thought I checked to see if /run was mounted before going through the whole exercise, but apparently I didn't (My first test was to "mount -t tmpfs /run" -- to shadow the /run mountpoint -- but I didn't realize this already existed as a tmpfs). [ed: Given that it's not wise to write to a fs of questionable state, editing /etc/systemd/journal.conf is out. So it would appear shadowing the sub-tree in question with a tmpfs mount would be the sanest approach?] [ed2: Just checked that if setting "Storage=persistent" in journald.conf, forcing logging under /var/log (and with /var/log as part of the rootfs) at least here, shadowing /var/log with a tmpfs works, in order to remount rootfs read-only in "runlevel 1" (actually systemctl rescue, when using systemd). Not sure if there's any elegant way to unshadow/unmount /var/log w/o a reboot though.]

esaym 11y ago

>My question was more about the GP elaborating why a new system would be something "[no one] wants to go through this again." Because the migration was terrible that is why. And I still continue to find stuff I hate about systemd. The latest was just yesterday, while troubleshooting some disk issues I wanted to fsck root before rebooting. Which normally means just remount root as read-only. Tried that and it didn't work, "disk was busy". Jumped into runlevel 1, still disk was busy. Rebooted into rescue mode, remount ro, still disk busy. After googling and digging through lsof, I could see it was systemd processes and journal that was holding onto files and preventing a remount. Tried killing systemd, journald, even kill -9, it just restarts it self and holds the disk. Darn. Did the touch /forcefsck trick, rebooted, yet no fsck. Darn again. More googling later, I see you have pass kernel parameters at boot time to force a fsck, and no, you can't select which FS either, it just does all of them. Rebooted again, fsck is running of course. I just wanted root checked but now it is doing all my terabyte partitions as well. Gave up at this point and just left the machine and hoped for the best.

lisivka 11y ago
I hate it too, but workaround is not too hard: chmod a-x journald && pkill journald of course, it is bad idea to touch damaged FS before fsck.
- simoncion 11y ago
  Wait. To run an unscheduled fsck of your root fs (and only your root fs), you have to remove the execute bit from a systemd component, kill that component, then remember to reset the bit after you're done? That's nuts.
  - morganvachon 11y ago
    That kind of thing is why I stick with Slackware and OpenBSD for my production systems. systemd has some great ideas, but it has some down right silliness as well, and I personally don't feel it's stable enough for daily use yet.
  - rdtsc 11y ago
    (Just to make sure everyone gets it) It will prevent it from running again. So it is basically a big FU! to systemd. "So you you think you gonna restart yourself? Oh, no you won't"
  - pdkl95 11y ago
    That's nothing - systemd panics (pid1 crash) if you restart dbus, which requires either the Magic SysRq or the reset button.. https://forums.gentoo.org/viewtopic-p-7811550.html#7811550 https://forums.gentoo.org/viewtopic-p-7811550.html#7811550 Pottering was apparently told about this in 2013, when he simply declared it expected behavior because dbus was "rock solid". http://lists.freedesktop.org/archives/systemd-devel/2013-January/008456.html http://lists.freedesktop.org/archives/systemd-devel/2013-Jan... Incidentally, this might be the original reason for kdbus - it's harder to restart the daemon after it becomes a kernel module.
    - simoncion 11y ago
      > Incidentally, this might be the original reason for kdbus... Yeah, especially given that its stated reason (Moving into the kernel gives us a 2x speedup over userspace dbus! Dbus is slow and we NEEED the speedup!) is mooted by the 10x speedup that cleaning up the userspace dbus libraries has achieved. [0] Having said that, in the systemd-devel post Poettering says that they're working on... > ...improving this in two ways: even if it still brings down the system, at least allow logind to handle this case nicely, so that you get a sane getty. [The other way is to push dbus into the kernel.] Systemd is a sprawling project, so I could see why it has taken more than two-and-a-half years to fix a local DOS caused by the perfectly normal method of responding to a DBUS upgrade. ;) [0] Fun fact: Those cleaned up libraries were supposed to be released after kdbus was merged into the kernel in 4.1. The Systemd Cabal was so disappointed that they had to ship the performant userspace dbus libs before the dbus daemon got pushed into kernelspace.
hga 11y ago
This is a job for a distro like System Rescue CD. Boot off CD, run in DRAM, figure out which partition or LV is root, and directly fsck it.
kbenson 11y ago
Maybe the problem is that you're carrying conventions from an obsolete system to a new one? A quick google search for how to force fsck with systemd comes up with results[1] mentioning options you haven't covered. Then again, I haven't had to do this, so maybe you did try those steps and it didn't work. 1: http://lmgtfy.com/?q=systemd+force+fsck http://lmgtfy.com/?q=systemd+force+fsck
- esaym 11y ago
  As far as I can see, they all mention "fsck.mode=force, as a kernel parameter", which is what I was complaining about.
  - kbenson 11y ago
    Here's the TL;DR 1) It's actually super easy to get to a read-only root mode. 2) It actually is a problem of carrying over obsolete conventions from prior init systems. 3) It's not your fault because systemd has extremely poor documentation on this AFAICT, to the point that people searching for this problem can't even get answers from others that know how to do it, or at least those answers don't rate very high. 4) Just boot with kernel param "emergency" instead of "single" (or "rescue" which is an alias for "single") and you will get a single user mode with root already mounted read-only for you. And now for the longer rambling version. Okay, so I spent a little time on this since I happen to have a virt for Fedora 20, and I finally found a couple of solutions that work. First, the systemd solution, which is emergency mode. It's correct that the journal will cause problems here, but that's because the journal was already running and apparently systemd doesn't want to, or is unable to stop it. The solution is to reboot into emergency mode. This can be accomplished by using the kernel param "systemd.unit=emergency.target", or just the param "emergency". This will not only boot you into a single user mode, but will default to having root mounted read-only. To clarify, this appears to be a new mode in addition to single user mode. Single user mode can still be reached with the kernel param "single" or "rescue" or "systemd.unit=rescue.target". I finally stumbled on this when I decided to look at the directions for what I wanted to accomplish for the distro I was using[1] and not for systemd specifically. Which makes sense, but I think we are all to stuck on "systemd is different, how do I do this in systemd" rather than "how does my distro say I should accomplish this now, given the changes they've instituted." 1: https://docs.fedoraproject.org/en-US/Fedora/18/html/Installation_Guide/ap-rescuemode.html#s1-rescuemode-booting-emergency https://docs.fedoraproject.org/en-US/Fedora/18/html/Installa...

e12e 11y ago

[ed2: Never mind - the standard setup is to have /run mounted as a tmpfs -- I failed to check that before this song and dance] Which distribution was this? I just tried on my laptop (running Debian 8, fs on top of lvm on top of LUKS), and a simple: telinit 1 # log in on console mount -oro,remount / worked as expected. [ed: Actually just tried forcing an fsck as well, and worked without a hitch. Just remember to "mount -oremount /" before running a "telinit 5" (Nothing really bad happens if not, but the system(d) will be confused if rootfs is mounted read-only).]

JdeBP 11y ago
esaym probably doesn't need people to try and diagnose xyr problem, and this isn't really the best venue to do that. If you want to try this, try at least replicating esaym's setup as described rather than using your own and declaring "works for me!". One way that springs to mind is moving the journal into /var/log and having no separate /var/log volume; which reproduces the described setup of open journal files on the root filesystem. Do that and then try to force an explicit check of the root volume. Of course, knowing that does point to one obvious way to avoid the stated problem: move the journal back into /run again (per the journald.conf manual page) and simply restart journald. No mucking about with permissions required. But that's not the only thing that xe could have tried. emergency mode, for example, is documented as not mounting additional non-API filesystems at all, nor read-write remounting the root volume. See http://freedesktop.org/wiki/Software/systemd/Debugging/#bootingintorescueoremergencytargets http://freedesktop.org/wiki/Software/systemd/Debugging/#boot...
- e12e 11y ago
  Hm, good catch. I thought I checked to see if /run was mounted before going through the whole exercise, but apparently I didn't (My first test was to "mount -t tmpfs /run" -- to shadow the /run mountpoint -- but I didn't realize this already existed as a tmpfs). [ed: Given that it's not wise to write to a fs of questionable state, editing /etc/systemd/journal.conf is out. So it would appear shadowing the sub-tree in question with a tmpfs mount would be the sanest approach?] [ed2: Just checked that if setting "Storage=persistent" in journald.conf, forcing logging under /var/log (and with /var/log as part of the rootfs) at least here, shadowing /var/log with a tmpfs works, in order to remount rootfs read-only in "runlevel 1" (actually systemctl rescue, when using systemd). Not sure if there's any elegant way to unshadow/unmount /var/log w/o a reboot though.]