6 ms·

I have a system where systemd decides, about half the times I boot it, to immediately unmount all of the filesystems listed in fstab literally right after havin

I have a system where systemd decides, about half the times I boot it, to immediately unmount all of the filesystems listed in fstab literally right after having finished mounted them as part of the regular boot process. All of this with other daemons starting in parallel. It will fail to unmount most of the filesystems, since they are of course busy, but often it will succeed in unmounting /var, /tmp, /home and others. Then it will continue starting on further services as if nothing happened and even proceed to gdm. But of course without /home I can't even login. No idea how to even start debugging it.

jraph 5y ago
I would run systemd-analyze critical-chain and then systemd-analyze blame to see if there is a service that could do this. And then I would look at the detailed boot logs. If nothing can be found, I'd look for some way to increase the log level. If you are not curious, reinstalling everything could be quite fast too, depending on what you have installed.
- AshamedCaptain 5y ago
  The logs are mostly useless and random. There is nothing that consistently appears between the point where it finishes mounting (lots of "Mouting...", then "Reached target Local File Systems") and where it starts unmounting ("Stopped target Local File Systems", then lots of "Unmouting ..."). Sometimes these two lines are less than one second apart. But I haven't found any message which is consistently between these two points that would point to a suspect. And this is not a problem that happens on every boot, so it doesn't show up on analysis... not to mention that most of the analysis is designed on trying to reduce boottime, which obviously is no help (thought it is rather fast already). It is rather easy to guess why a service is starting, but not why it decides to stop 'orderly', much less why a .target decides to stop...
  - jraph 5y ago
    Most certainly you can only analyze things on boots where the problem appears, unless logs are persisted across boots. Indeed systemd-analyze focuses on boot times, though I find it is useful to get a list of services that are run. If you are adventurous you could probably replace the umount (resp. the systemd-umount) binary by a script that runs umount (resp. systemd-umount), and also prints the process tree (and its PID) in some file so you can get insight on what ran it.
teddyfrozevelt 5y ago
Systemd generates mount units for the filesystems listed in fstab. Maybe check those with systemctl status? (Like `systemctl status -- -.mount` for my root partition)
- dijit 5y ago
  Candidly: that command looks like the least discoverable command I’ve seen in the last year.
  - heavyset_go 5y ago
    The command systemctl status *.mount Just shows the status of all mount units. It's the same command you'd use to check any other systemd unit, for example, if you want to check Docker's status: systemctl status docker.service Seems consistent and discoverable as long as you have a surface understanding of systemd units.
    - regularfry 5y ago
      That to me reads as "show me the status of all files ending in .mount in the current directory". Once you start hijacking shell metacharacters you're on dodgy ground.
      - heavyset_go 5y ago
        It's not using the shell glob syntax, it's just a regex compatible string. The find utility does the same thing: find ~/Code -type f -name *.js Where *.js isn't using the shell glob feature, it's just a regex compatible string. You could put quotes around it to make it clearer: systemctl status "*.mount" The string is matching on unit names, and the mount units have the .mount suffix.
        regularfry 5y ago
        > Where *.js isn't using the shell glob feature, it's just a regex compatible string. Except it isn't, because a `*` at the start of a regex is invalid. `-name` patterns given to `find` are explicitly shell glob patterns. You might be thinking of `-regex` patterns, which GNU find also has. It's a bit shonky when `find` does it because it's not saying "the shell pattern applies in this directory", and I regularly trip over it. It's something special you've got to know about how `find` works, but at least it's still anchored to the idea of being searched in a path that's also part of the command. There's a semantic link there. With `systemctl status *.mount` you've not even got that. What path is being searched for `*.mount` files? Is that path part of the `systemctl` interface? Or is this just another way that `systemctl` insists on being a special snowflake that someone's decided I now need to devote neurons to?
viraptor 5y ago
You can check in journal which service startup does the unmounting align with. Maybe generate a bootchart http://manpages.ubuntu.com/manpages/bionic/man1/systemd-bootchart.1.html http://manpages.ubuntu.com/manpages/bionic/man1/systemd-boot... You could also replace umount with a wrapper that logs the parent process tree on execution. And of course you can open an issue on GitHub.
- pengaru 5y ago
  > And of course you can open an issue on GitHub. Only do this if you've reproduced what seems to be a bug in one of the latest two releases, otherwise contact your distro. The systemd github issue tracker is not a support forum, you'll find this stated in the issue submission form.
  - kristjansson 5y ago
    Fair advice, and they’re probably right to redirect most users that would file a bug with them. But on a normative level, perhaps a project that doesn’t want to support users shouldn’t have subsumed a huge swath of user land things?
    - pengaru 5y ago
      > But on a normative level, perhaps a project that doesn’t want to support users shouldn’t have subsumed a huge swath of user land things? It's not that the project doesn't want to support users in some absolute sense, it just doesn't want it happening in github issues. The project provides the systemd-devel mailing list for that, but it's still preferred that users stuck on old versions determined by their distro's release cadence seek support from their distro vendor. It's likely such slow-moving distros are patching their LTS versions anyways, and are best positioned to support what they ship.
  - Dah00n 5y ago
    If this is a bug in systemd it belongs at systemd no matter which version it is. If systemd doesn't want to do support that is their choice but distros aren't the right place to post systemd bugs. It only belongs there if it is either a bug in the distro itself or a support request for distro specific problems.
stevenhuang 5y ago
If that happens again I'd try opening a virtual terminal, login as root and check logs.
- Twirrim 5y ago
  > If that happens again I'd try opening a virtual terminal, login as root and check logs. If the system has unmounted all the drives, there won't be any logs. This happens often enough with systemd / journald to be a big source of annoyance. The only option you get is to tell systemd to boot in debug mode but that fundamentally alters the timing and behaviour of so much that it often stops the weird behaviour from ever happening (similar fun can be had with `dracut` where enabling debug tends to stop any race condition from ever happening)
  - AshamedCaptain 5y ago
    Note systemd does dump its own log to the kernel ring buffer until journald starts (the last message I see on the kernel ring buffer is Started Journal Service). It's no help though when (as in my case) journald starts but then has /var unmounted below its feet. I can write a script that keeps /var busy though, preventing it from being unmounted.
    - tremon 5y ago
      > I can write a script There's some delicious irony in having to write a shell script to combat the init system that was designed to replace all the shell scripts.
isodude 5y ago
We had that kind of problem because we relied on /dev/disks/by-label
- heavyset_go 5y ago
  Can you tell us more about this? How did you determine that that was the problem? What was your solution? I ask because I use /dev/disks/by-label and have never had this problem, and if I ever had to experience it, it would drive me insane.
  - sigg3 5y ago
    I've solved a similarly sounding issue by using the UUID. Get it with blkid, then use UUID="<uuid goes here>" in fstab instead of LABEL="<label>". But this was a long time ago, on a much older systemd (like Fedora 17 or 18).
  - dharmab 5y ago
    This is a common error even back in the Sysvinit days. /dev/disks/by-label is not guaranteed to map the name labels to disks on every boot, and it can change with boot timing based on the storage controller behavior. Use /dev/disks/by-uuid instead. /dev/disks/by-label is a legacy from the old days of systems with static disks wired in a specific, unchanging configuration.
    - heftig 5y ago
      You've confused by-label with the kernel's device nodes (e.g. /dev/sda1). Kernel names are assigned in order. Labels are just like UUIDs and part of the filesystem. However, unlike UUIDs they don't necessarily exist, are not as likely to be unique (you should only get duplicate UUIDs by cloning filesystems) and are easier to change. This makes them less suitable for mount configuration.
  - isodude 5y ago
    Long time ago now, but we had disks just unmounting and mounting at free will. As someone said by-label is not stable and could be changed, causing problems. We still use it though but not in customer facing servers where these problems arise. Maybe related to lvm/snapshot etc.
corty 5y ago
Deactivate graphical boot screens (often just by pressing Esc), increase the log level[0] and try to read the logs somehow. Maybe you can still log in as root. If you cannot log in as root even, but networking works, configure rsyslog to send your logs to another host and read them there. [0] https://freedesktop.org/wiki/Software/systemd/Debugging/ https://freedesktop.org/wiki/Software/systemd/Debugging/
- jraph 5y ago
  if you can't login as root, you can try to put bash as init (with init=/bin/bash in the kernel boot command line) to inspect the system. Might be difficult on some systems (with SELinux?)
  - megous 5y ago
    if you have access to kernel command line to add init= you can also disable selinux there.
    - corty 5y ago
      I'd rather recommend booting a liveCD. grml, knoppix, whatever floats your boat. An init=/bin/bash shell needs quite a bit of knowledge to get working (mount rw, get rid of selinux, mount the real rootfs), is possibly just busybox and generally a pain. Even more so if you need the net to copy something over. It isn't impossible, and OK in a pinch, but why make it hard on yourself?
      - isodude 5y ago
        You can actually specify systemd.unit=rescue.target nowadays. https://ostechnix.com/how-to-boot-into-rescue-mode-or-emergency-mode-in-ubuntu-18-04/ https://ostechnix.com/how-to-boot-into-rescue-mode-or-emerge...
      - megous 5y ago
        You forgot mounting /proc and /sys :) Anyway, it's the quickest way to recover if all you need is to edit some config file or run passwd.
sigg3 5y ago
I'd start by looking at the systemd unit files in /etc/systemd/system. Beware of service wants. I had straight up errors in a network unit file that caused weird behavior (IP address disappearing). This was a long time ago though. Sometimes moving a unit from one target (graphical) to an earlier (multiuser) solves the issue. You might also have "defaults" in /usr/lib/systemd/ to compare with. systemctl and friends are really good tools for troubleshooting but takes a little while to get used to. Duck duck go is your friend. Finally, some volume technologies like VDO requires a x-systemd.requires statement in fstab.
pengaru 5y ago
That's super annoying. I've experienced similarly frustrating boot problems when I had some FS corruption causing /var fsck to fail @ boot. Despite dropping me to a rescue shell, systemd entered a continuous loop of retrying fscking /var while I was attempting to identify and fsck the problematic filesystem. Few things are more infuriating than having your system drop you to a rescue shell, while it continues to endlessly change the state of the system - and not just any state, but the very area you're in the rescue shell to investigate and resolve. It just kept making /var busy while I kept attempting to manually fsck it...
regularfry 5y ago
Complex systems fail in complex ways.
asgeir 5y ago
Have you tried increasing the log-level in /etc/systemd/system.conf? You can also set LogTarget=kmsg if you have issues with the journal. You can also control this with the kernel command line if you prefer https://www.freedesktop.org/software/systemd/man/systemd.html#systemd.log_color https://www.freedesktop.org/software/systemd/man/systemd.htm... You could try a kernel command line like this 'systemd.log_level=debug systemd.log_target=kmsg' for instance You can also look at systemctl list-dependencies local-fs.target to see if it has any failed dependencies You can also use systemctl show local-fs.target to sanity-check it to see if there are any local modifications to the target that are breaking it

jraph 5y ago

I would run systemd-analyze critical-chain and then systemd-analyze blame to see if there is a service that could do this. And then I would look at the detailed boot logs. If nothing can be found, I'd look for some way to increase the log level. If you are not curious, reinstalling everything could be quite fast too, depending on what you have installed.

AshamedCaptain 5y ago
The logs are mostly useless and random. There is nothing that consistently appears between the point where it finishes mounting (lots of "Mouting...", then "Reached target Local File Systems") and where it starts unmounting ("Stopped target Local File Systems", then lots of "Unmouting ..."). Sometimes these two lines are less than one second apart. But I haven't found any message which is consistently between these two points that would point to a suspect. And this is not a problem that happens on every boot, so it doesn't show up on analysis... not to mention that most of the analysis is designed on trying to reduce boottime, which obviously is no help (thought it is rather fast already). It is rather easy to guess why a service is starting, but not why it decides to stop 'orderly', much less why a .target decides to stop...
- jraph 5y ago
  Most certainly you can only analyze things on boots where the problem appears, unless logs are persisted across boots. Indeed systemd-analyze focuses on boot times, though I find it is useful to get a list of services that are run. If you are adventurous you could probably replace the umount (resp. the systemd-umount) binary by a script that runs umount (resp. systemd-umount), and also prints the process tree (and its PID) in some file so you can get insight on what ran it.

teddyfrozevelt 5y ago

Systemd generates mount units for the filesystems listed in fstab. Maybe check those with systemctl status? (Like `systemctl status -- -.mount` for my root partition)

dijit 5y ago
Candidly: that command looks like the least discoverable command I’ve seen in the last year.
- heavyset_go 5y ago
  The command systemctl status *.mount Just shows the status of all mount units. It's the same command you'd use to check any other systemd unit, for example, if you want to check Docker's status: systemctl status docker.service Seems consistent and discoverable as long as you have a surface understanding of systemd units.
  - regularfry 5y ago
    That to me reads as "show me the status of all files ending in .mount in the current directory". Once you start hijacking shell metacharacters you're on dodgy ground.
    - heavyset_go 5y ago
      It's not using the shell glob syntax, it's just a regex compatible string. The find utility does the same thing: find ~/Code -type f -name *.js Where *.js isn't using the shell glob feature, it's just a regex compatible string. You could put quotes around it to make it clearer: systemctl status "*.mount" The string is matching on unit names, and the mount units have the .mount suffix.
      - regularfry 5y ago
        > Where *.js isn't using the shell glob feature, it's just a regex compatible string. Except it isn't, because a `*` at the start of a regex is invalid. `-name` patterns given to `find` are explicitly shell glob patterns. You might be thinking of `-regex` patterns, which GNU find also has. It's a bit shonky when `find` does it because it's not saying "the shell pattern applies in this directory", and I regularly trip over it. It's something special you've got to know about how `find` works, but at least it's still anchored to the idea of being searched in a path that's also part of the command. There's a semantic link there. With `systemctl status *.mount` you've not even got that. What path is being searched for `*.mount` files? Is that path part of the `systemctl` interface? Or is this just another way that `systemctl` insists on being a special snowflake that someone's decided I now need to devote neurons to?

viraptor 5y ago

You can check in journal which service startup does the unmounting align with. Maybe generate a bootchart http://manpages.ubuntu.com/manpages/bionic/man1/systemd-bootchart.1.html http://manpages.ubuntu.com/manpages/bionic/man1/systemd-boot... You could also replace umount with a wrapper that logs the parent process tree on execution. And of course you can open an issue on GitHub.

pengaru 5y ago
> And of course you can open an issue on GitHub. Only do this if you've reproduced what seems to be a bug in one of the latest two releases, otherwise contact your distro. The systemd github issue tracker is not a support forum, you'll find this stated in the issue submission form.
- kristjansson 5y ago
  Fair advice, and they’re probably right to redirect most users that would file a bug with them. But on a normative level, perhaps a project that doesn’t want to support users shouldn’t have subsumed a huge swath of user land things?
  - pengaru 5y ago
    > But on a normative level, perhaps a project that doesn’t want to support users shouldn’t have subsumed a huge swath of user land things? It's not that the project doesn't want to support users in some absolute sense, it just doesn't want it happening in github issues. The project provides the systemd-devel mailing list for that, but it's still preferred that users stuck on old versions determined by their distro's release cadence seek support from their distro vendor. It's likely such slow-moving distros are patching their LTS versions anyways, and are best positioned to support what they ship.
- Dah00n 5y ago
  If this is a bug in systemd it belongs at systemd no matter which version it is. If systemd doesn't want to do support that is their choice but distros aren't the right place to post systemd bugs. It only belongs there if it is either a bug in the distro itself or a support request for distro specific problems.

stevenhuang 5y ago

If that happens again I'd try opening a virtual terminal, login as root and check logs.

Twirrim 5y ago
> If that happens again I'd try opening a virtual terminal, login as root and check logs. If the system has unmounted all the drives, there won't be any logs. This happens often enough with systemd / journald to be a big source of annoyance. The only option you get is to tell systemd to boot in debug mode but that fundamentally alters the timing and behaviour of so much that it often stops the weird behaviour from ever happening (similar fun can be had with `dracut` where enabling debug tends to stop any race condition from ever happening)
- AshamedCaptain 5y ago
  Note systemd does dump its own log to the kernel ring buffer until journald starts (the last message I see on the kernel ring buffer is Started Journal Service). It's no help though when (as in my case) journald starts but then has /var unmounted below its feet. I can write a script that keeps /var busy though, preventing it from being unmounted.
  - tremon 5y ago
    > I can write a script There's some delicious irony in having to write a shell script to combat the init system that was designed to replace all the shell scripts.

isodude 5y ago

We had that kind of problem because we relied on /dev/disks/by-label

heavyset_go 5y ago
Can you tell us more about this? How did you determine that that was the problem? What was your solution? I ask because I use /dev/disks/by-label and have never had this problem, and if I ever had to experience it, it would drive me insane.
- sigg3 5y ago
  I've solved a similarly sounding issue by using the UUID. Get it with blkid, then use UUID="<uuid goes here>" in fstab instead of LABEL="<label>". But this was a long time ago, on a much older systemd (like Fedora 17 or 18).
- dharmab 5y ago
  This is a common error even back in the Sysvinit days. /dev/disks/by-label is not guaranteed to map the name labels to disks on every boot, and it can change with boot timing based on the storage controller behavior. Use /dev/disks/by-uuid instead. /dev/disks/by-label is a legacy from the old days of systems with static disks wired in a specific, unchanging configuration.
  - heftig 5y ago
    You've confused by-label with the kernel's device nodes (e.g. /dev/sda1). Kernel names are assigned in order. Labels are just like UUIDs and part of the filesystem. However, unlike UUIDs they don't necessarily exist, are not as likely to be unique (you should only get duplicate UUIDs by cloning filesystems) and are easier to change. This makes them less suitable for mount configuration.
- isodude 5y ago
  Long time ago now, but we had disks just unmounting and mounting at free will. As someone said by-label is not stable and could be changed, causing problems. We still use it though but not in customer facing servers where these problems arise. Maybe related to lvm/snapshot etc.

corty 5y ago

Deactivate graphical boot screens (often just by pressing Esc), increase the log level[0] and try to read the logs somehow. Maybe you can still log in as root. If you cannot log in as root even, but networking works, configure rsyslog to send your logs to another host and read them there. [0] https://freedesktop.org/wiki/Software/systemd/Debugging/ https://freedesktop.org/wiki/Software/systemd/Debugging/

jraph 5y ago
if you can't login as root, you can try to put bash as init (with init=/bin/bash in the kernel boot command line) to inspect the system. Might be difficult on some systems (with SELinux?)
- megous 5y ago
  if you have access to kernel command line to add init= you can also disable selinux there.
  - corty 5y ago
    I'd rather recommend booting a liveCD. grml, knoppix, whatever floats your boat. An init=/bin/bash shell needs quite a bit of knowledge to get working (mount rw, get rid of selinux, mount the real rootfs), is possibly just busybox and generally a pain. Even more so if you need the net to copy something over. It isn't impossible, and OK in a pinch, but why make it hard on yourself?
    - isodude 5y ago
      You can actually specify systemd.unit=rescue.target nowadays. https://ostechnix.com/how-to-boot-into-rescue-mode-or-emergency-mode-in-ubuntu-18-04/ https://ostechnix.com/how-to-boot-into-rescue-mode-or-emerge...
    - megous 5y ago
      You forgot mounting /proc and /sys :) Anyway, it's the quickest way to recover if all you need is to edit some config file or run passwd.

sigg3 5y ago

I'd start by looking at the systemd unit files in /etc/systemd/system. Beware of service wants. I had straight up errors in a network unit file that caused weird behavior (IP address disappearing). This was a long time ago though. Sometimes moving a unit from one target (graphical) to an earlier (multiuser) solves the issue. You might also have "defaults" in /usr/lib/systemd/ to compare with. systemctl and friends are really good tools for troubleshooting but takes a little while to get used to. Duck duck go is your friend. Finally, some volume technologies like VDO requires a x-systemd.requires statement in fstab.

pengaru 5y ago

That's super annoying. I've experienced similarly frustrating boot problems when I had some FS corruption causing /var fsck to fail @ boot. Despite dropping me to a rescue shell, systemd entered a continuous loop of retrying fscking /var while I was attempting to identify and fsck the problematic filesystem. Few things are more infuriating than having your system drop you to a rescue shell, while it continues to endlessly change the state of the system - and not just any state, but the very area you're in the rescue shell to investigate and resolve. It just kept making /var busy while I kept attempting to manually fsck it...

regularfry 5y ago

Complex systems fail in complex ways.

asgeir 5y ago

Have you tried increasing the log-level in /etc/systemd/system.conf? You can also set LogTarget=kmsg if you have issues with the journal. You can also control this with the kernel command line if you prefer https://www.freedesktop.org/software/systemd/man/systemd.html#systemd.log_color https://www.freedesktop.org/software/systemd/man/systemd.htm... You could try a kernel command line like this 'systemd.log_level=debug systemd.log_target=kmsg' for instance You can also look at systemctl list-dependencies local-fs.target to see if it has any failed dependencies You can also use systemctl show local-fs.target to sanity-check it to see if there are any local modifications to the target that are breaking it