4 ms·

"Myth #11: Instead of merging / into /usr it would make a lot more sense to merge /usr into /. Fact: This would make the separation between vendor-supplied OS

"Myth #11: Instead of merging / into /usr it would make a lot more sense to merge /usr into /. Fact: This would make the separation between vendor-supplied OS resources and machine-specific even worse, thus making OS snapshots and network/container sharing of it much harder and non-atomic, and clutter the root file system with a multitude of new directories."

icedchai 4y ago
Call me skeptical. I haven’t seen a Linux system that put /usr on its own partition in at least 15 years.
- rektide 4y ago
  I dont think that was quite the right argumentation. Or at least not clear enough. In general, the desire is to put the system image into /usr. A single partition, that holds literally everything you need to launch an os. If you can mount a /usr, everything else will populate. If you want to atomically swap system images, just swap /usr. Imagine the counter: what would happen if we just swapped /bin? Shit would be a mess. /lib and /usr/lib wouldnt get updated in sync. /include and /share wouldnt be in sync. The idea is that the Unified System Resources ought unifiedly be swapped out. And the system specific resources(/bin, /lib, /include, et ceteta) ought not exist, provide no real contemprary value, and making image swapping a much much harder to accomplish well.
  - icedchai 4y ago
    I can buy that argument, but I’ve never seen /usr get swapped out with any modern package manager. How would that work? Or is this more theoretical? FYI “usr” really means “user”. Somebody made up “unified system resources” as a backronym.
    - db48x 4y ago
      Yea, it’s pretty funny. It used to hold the home directories, but the disk holding the OS ran out of space, so they started putting new utilities in /usr/bin. The distribution tapes they sent out to everyone who ordered a copy were made directly from the live system, so we all inherited the same quirk. We’ve moved the home directories elsewhere, and now the whole OS is climbing into /usr in order to be atomically mountable. I keep thinking of that Gilbert and Sullivan song with the line “it gradually got so!”.
      - rtontic 4y ago
        Even funnier, now a lot of software pollutes your home directory with shared files and dotfiles, so that space is invaded too. I resort to creating a separate files directory, either in /files, my home directory or in /mnt/files, and store all my local files there.
        db48x 4y ago
        I’m not as bothered about that. I do have a git repository called dotfiles with some of the more important ones. I then use GNU Stow to install them from there to their customary locations. That helps me keep them in sync across computers.
    - rektide 4y ago
      Generally the idea is more for atomic system upgrades, and less for package managers. Why have package managers when you can just mmdebstrap a new image & try start it, but keep all your /etc and /home stuff? That's the dream. Package managers become semi-obsolete. I think there are a variety of ideas for how package managers might come into compliance & make use of this though! A lot of possibility. But the notion of OSes that can swap images, that transcend the idea of one special bucket of bits that we slowly move forward, is kind of new. Containers are 100% there, are making it clear. But the conventional OS is still lagging, learning, behind. I recommend checking out systems like Wyrcan to see a little deeper how we can eliminate some of the historical OS layers & build a more modular, more secure, more atomic, more application centric OS[1]. A lot of it relies on better filesystems like ZFS or Btrfs which have snapshots. I myself twiddled around with writing some Debian-Live boot tools that used btrfs snapshots, copied onto ram-drives[2], that would potentially work well with package managers (boot either your starting image, or your "new" snapshot image). These ideas predate (the deeply reviled by some haters) Lennart's post, "Revisiting How We Put Together Linux Systems"[3]. But Lennart here, in mid 2014, understood what containers were doing, understood how atomic system upgrades were trending, understood what was happening. This isn't a Lennart idea, but Lennart as usual semi got the gist of what made sense, what would help, where we were going. He captured the good idea of what was occuring already into a post. And calling out /usr as "the system image" was a clear & obvious win for that day, back in 2014, and it's still a clear & obvious win today to consolidate the OS into one directory today. May it be so. This is the way. So say we all. It is known. The "usr" meaning user is seemingly in fact true. But it's also, imho, wrong? The backronym is more truthful, better describes what role /usr plays & has played for multiple decades now. "user" doesn't really mean anything reasonable, clear, or cogent. [1] https://gitlab.com/wyrcan/wyrcan https://gitlab.com/wyrcan/wyrcan [2] https://github.com/rektide/debian-live-boot https://github.com/rektide/debian-live-boot [3] https://0pointer.net/blog/revisiting-how-we-put-together-linux-systems.html https://0pointer.net/blog/revisiting-how-we-put-together-lin...
      - icedchai 4y ago
        You could achieve the same thing with sym links, which is basically what Apple does with newer versions of macOS. The main OS partition (/) is read only, essentially immutable (unless you disable SIP.) There is a writeable volume mounted on /private (in a funky way, using "firm links".) /etc, /var, /home, and a couple others are symlinks to /private/etc, /private/var, /private/home ... They do some funky stuff with /usr/local and a few other directories, too. This person seems to explain it: http://www.swiftforensics.com/2019/10/macos-1015-volumes-firmlink-magic.html http://www.swiftforensics.com/2019/10/macos-1015-volumes-fir...
    - p_l 4y ago
      It's the standard upgrade method on most deployed desktop/laptop linux, a certain Gentoo derivative known as ChromeOS, and from there it's also common update mechanism of Google COS (default for GKE nowadays) and Flatcar (not sure if Fedora mangling of CoreOS uses it anymore as they reportedly use rpm-ostree)
      - deleted 4y ago
        [deleted]
      - kazinator 4y ago
        > standard upgrade method on most deployed desktop/laptop linux Deployed desktop Linux has /bin and /lib, so it's somehow working.
        p_l 4y ago
        ChromeOS has them as symlinks to /usr/bin and /usr/lib, so does Fedora and all Fedora-derived distros, so does Arch and its derivatives.
    - obscurette 4y ago
      CoreOS Container Linux (haven't checked out Fedora CoreOS) and its' fork Flatcar Linux do that. I couldn't find any good doc about this, but the idea is that: - RO /usr contains all the system, the rest is populated with tmpfilesd - There is two /usr partitions, the active one is selected with GPT priority attribute https://github.com/endocode/coreos-docs/blob/master/os/sdk-disk-partitions.md https://github.com/endocode/coreos-docs/blob/master/os/sdk-d...
    - kazinator 4y ago
      It's another made-up problem whose purpose is to enable the mockery of a valid idea in Myth #11.
  - cowtools 4y ago
    >In general, the desire is to put the system image into /usr. A single partition, that holds literally everything you need to launch an os Hasn't this use case been replaced by initramfs?
    - bonzini 4y ago
      Yeah, he (Poettering) intends /usr to contain everything needed to _run_ the OS, with the launch handled by initramfs.
    - p_l 4y ago
      Not really. A common spiel of the time was that having to mount / and /usr due to packager laziness started requiring it b handled in initramfs making it more complex. Previously / was supposed to be able to boot into minimal standalone system that could mount the rest of itself. So the common argument was that one should accept that readonly separate /usr and snapshots and whatsoever you wanted weren't going to be a thing because fedora didn't want to maintain support for mounting /usr in initramfs.
  - kazinator 4y ago
    Another idea: what if we don't ever swap images like this? It looks like a bad idea that I would never want to do. This is what your chroots and jails are for. But if I wanted to do it in style, I'd hack up the cool kernel support to atomically mount multiple directories as part of one atomic swap. How can you call it "atomic" if you have executables still running from the old /usr/bin? Say there is some script running that is executing external programs. At some unspecified point in its execution, /usr/bin/whatever changes meaning; all newly run programs are different versions on a different libc and so on. If you have enough control of the system that there are no such processes, then you can easily do the swap directory by directory: just do it with system calls out of a single process, rather than by invoking tools, so that nothing trips up at the point in time when you have a mismatched /bin and /lib.
    - rektide 4y ago
      Nothing you'be talked about sounds like a remotely normal thing that is documented or discussed anywhere and I doubt there's more than a fistful of orgs on the planet that have jury rigged that up. Meanwhile there are at least a dozen distros doing atomic image upgrades. Fedora SilverBlue for one example. These upgrades often involve a kexec restart. For fleets of servers- the target audience- it's expected that machines will go down & rolling restarts should be fine. That's how true atomic behavior is gotten. But I think you are nitpicking & being fussy with your complaining about old processes being left around; even if you do an in-place swap, but have some old processes around, at least the whole system is together & consistent. More broadly, thjs set of problem in general doesnt change much whether you are using a merged /usr or not. I genuinely dont get any of your disgust at having a /usr image that contains the functioning system image. You talk about chroots, as though it's an alternative somehow, but this practice of having a self-contained image seems like it would greatly assist in quickly cloning new images to chroot into.
      - kazinator 4y ago
        If "atomic" means "stop the world running one system, and reboot into a different system", we can easily arrange to reshuffle two or more directories. The program which does that can be easily designed not to be affected by the intermediate states, like where you have a new /lib, but still the old /bin.
db48x 4y ago
Yea, the real goal here is to be able to mount the whole OS in one operation. To that end, /dev is now autopopulated by udev and you can even leave /etc completely empty. The goal is for everything to have good enough defaults that having a config file in /etc is unimportant unless the admin wants to override something. Probably the only things in there when you install a new system would be the root password in /etc/shadow and your user and group information in /etc/passwd and /etc/group. It should make a lot of tasks a lot easier. The one I like most is trivially setting up containers or VMs that have your base OS in them plus some new software that you are testing. For example, at work we use Debian with a bunch of extra packages installed, plus our own package containing all manner of software and configuration that we’ve written. Although we do a fair amount of unit testing, and the Rust compiler really has our back, it is very difficult to really test things outside of production. Currently we have to painstakingly set up a test environment on a production machine, using tricks like altering our PATH so that the test version of the tools we are working on will be called, or editing things so that they call the version under test instead. It’s doable but laborious. If we were using Fedora instead of Debian, we would be able to use systemd-nspawn to spawn a container with the existing OS, read–only mounts containing our databases and other large datasets, and an overlayfs to redirect any writes to those file systems to an alternate location. Installing and running our software inside that container would work exactly as if we were in production, and testing everything would be so much simpler. systemd-nspawn is available on Debian, but running it requires keeping extra copies of the OS for each container you want to spawn, so we haven’t actually gotten around to doing it yet. Oh, and each container needs a properly set up /etc directory. We can’t just leave it empty, and we can’t just blindly copy the host’s either.
p_l 4y ago
This myth however was inverted by people pushing for /usr merge on various media platforms at the time. A common example of "what's wrong with split /usr" that was pushed was things like dropping pci.ids database in /usr/share and it being required by code mounting /usr, and that mounting / and /usr separately in initrd was a wasteful complexity. In fact, today is the first time I have heard of reasoning that supported /usr on separate partition and I thought it was just an inversion done later by people who noticed possible benefits for some extra initrd complexity (btw, dracut and systemd oriented initrd scripts are stupidly complex. Including generating systemd units by text templating in udev rules as part of dracut boot process. Wtf)

icedchai 4y ago

Call me skeptical. I haven’t seen a Linux system that put /usr on its own partition in at least 15 years.

rektide 4y ago
I dont think that was quite the right argumentation. Or at least not clear enough. In general, the desire is to put the system image into /usr. A single partition, that holds literally everything you need to launch an os. If you can mount a /usr, everything else will populate. If you want to atomically swap system images, just swap /usr. Imagine the counter: what would happen if we just swapped /bin? Shit would be a mess. /lib and /usr/lib wouldnt get updated in sync. /include and /share wouldnt be in sync. The idea is that the Unified System Resources ought unifiedly be swapped out. And the system specific resources(/bin, /lib, /include, et ceteta) ought not exist, provide no real contemprary value, and making image swapping a much much harder to accomplish well.
- icedchai 4y ago
  I can buy that argument, but I’ve never seen /usr get swapped out with any modern package manager. How would that work? Or is this more theoretical? FYI “usr” really means “user”. Somebody made up “unified system resources” as a backronym.
  - db48x 4y ago
    Yea, it’s pretty funny. It used to hold the home directories, but the disk holding the OS ran out of space, so they started putting new utilities in /usr/bin. The distribution tapes they sent out to everyone who ordered a copy were made directly from the live system, so we all inherited the same quirk. We’ve moved the home directories elsewhere, and now the whole OS is climbing into /usr in order to be atomically mountable. I keep thinking of that Gilbert and Sullivan song with the line “it gradually got so!”.
    - rtontic 4y ago
      Even funnier, now a lot of software pollutes your home directory with shared files and dotfiles, so that space is invaded too. I resort to creating a separate files directory, either in /files, my home directory or in /mnt/files, and store all my local files there.
      - db48x 4y ago
        I’m not as bothered about that. I do have a git repository called dotfiles with some of the more important ones. I then use GNU Stow to install them from there to their customary locations. That helps me keep them in sync across computers.
  - rektide 4y ago
    Generally the idea is more for atomic system upgrades, and less for package managers. Why have package managers when you can just mmdebstrap a new image & try start it, but keep all your /etc and /home stuff? That's the dream. Package managers become semi-obsolete. I think there are a variety of ideas for how package managers might come into compliance & make use of this though! A lot of possibility. But the notion of OSes that can swap images, that transcend the idea of one special bucket of bits that we slowly move forward, is kind of new. Containers are 100% there, are making it clear. But the conventional OS is still lagging, learning, behind. I recommend checking out systems like Wyrcan to see a little deeper how we can eliminate some of the historical OS layers & build a more modular, more secure, more atomic, more application centric OS[1]. A lot of it relies on better filesystems like ZFS or Btrfs which have snapshots. I myself twiddled around with writing some Debian-Live boot tools that used btrfs snapshots, copied onto ram-drives[2], that would potentially work well with package managers (boot either your starting image, or your "new" snapshot image). These ideas predate (the deeply reviled by some haters) Lennart's post, "Revisiting How We Put Together Linux Systems"[3]. But Lennart here, in mid 2014, understood what containers were doing, understood how atomic system upgrades were trending, understood what was happening. This isn't a Lennart idea, but Lennart as usual semi got the gist of what made sense, what would help, where we were going. He captured the good idea of what was occuring already into a post. And calling out /usr as "the system image" was a clear & obvious win for that day, back in 2014, and it's still a clear & obvious win today to consolidate the OS into one directory today. May it be so. This is the way. So say we all. It is known. The "usr" meaning user is seemingly in fact true. But it's also, imho, wrong? The backronym is more truthful, better describes what role /usr plays & has played for multiple decades now. "user" doesn't really mean anything reasonable, clear, or cogent. [1] https://gitlab.com/wyrcan/wyrcan https://gitlab.com/wyrcan/wyrcan [2] https://github.com/rektide/debian-live-boot https://github.com/rektide/debian-live-boot [3] https://0pointer.net/blog/revisiting-how-we-put-together-linux-systems.html https://0pointer.net/blog/revisiting-how-we-put-together-lin...
    - icedchai 4y ago
      You could achieve the same thing with sym links, which is basically what Apple does with newer versions of macOS. The main OS partition (/) is read only, essentially immutable (unless you disable SIP.) There is a writeable volume mounted on /private (in a funky way, using "firm links".) /etc, /var, /home, and a couple others are symlinks to /private/etc, /private/var, /private/home ... They do some funky stuff with /usr/local and a few other directories, too. This person seems to explain it: http://www.swiftforensics.com/2019/10/macos-1015-volumes-firmlink-magic.html http://www.swiftforensics.com/2019/10/macos-1015-volumes-fir...
  - p_l 4y ago
    It's the standard upgrade method on most deployed desktop/laptop linux, a certain Gentoo derivative known as ChromeOS, and from there it's also common update mechanism of Google COS (default for GKE nowadays) and Flatcar (not sure if Fedora mangling of CoreOS uses it anymore as they reportedly use rpm-ostree)
    - deleted 4y ago
      [deleted]
    - kazinator 4y ago
      > standard upgrade method on most deployed desktop/laptop linux Deployed desktop Linux has /bin and /lib, so it's somehow working.
      - p_l 4y ago
        ChromeOS has them as symlinks to /usr/bin and /usr/lib, so does Fedora and all Fedora-derived distros, so does Arch and its derivatives.
  - obscurette 4y ago
    CoreOS Container Linux (haven't checked out Fedora CoreOS) and its' fork Flatcar Linux do that. I couldn't find any good doc about this, but the idea is that: - RO /usr contains all the system, the rest is populated with tmpfilesd - There is two /usr partitions, the active one is selected with GPT priority attribute https://github.com/endocode/coreos-docs/blob/master/os/sdk-disk-partitions.md https://github.com/endocode/coreos-docs/blob/master/os/sdk-d...
  - kazinator 4y ago
    It's another made-up problem whose purpose is to enable the mockery of a valid idea in Myth #11.
- cowtools 4y ago
  >In general, the desire is to put the system image into /usr. A single partition, that holds literally everything you need to launch an os Hasn't this use case been replaced by initramfs?
  - bonzini 4y ago
    Yeah, he (Poettering) intends /usr to contain everything needed to _run_ the OS, with the launch handled by initramfs.
  - p_l 4y ago
    Not really. A common spiel of the time was that having to mount / and /usr due to packager laziness started requiring it b handled in initramfs making it more complex. Previously / was supposed to be able to boot into minimal standalone system that could mount the rest of itself. So the common argument was that one should accept that readonly separate /usr and snapshots and whatsoever you wanted weren't going to be a thing because fedora didn't want to maintain support for mounting /usr in initramfs.
- kazinator 4y ago
  Another idea: what if we don't ever swap images like this? It looks like a bad idea that I would never want to do. This is what your chroots and jails are for. But if I wanted to do it in style, I'd hack up the cool kernel support to atomically mount multiple directories as part of one atomic swap. How can you call it "atomic" if you have executables still running from the old /usr/bin? Say there is some script running that is executing external programs. At some unspecified point in its execution, /usr/bin/whatever changes meaning; all newly run programs are different versions on a different libc and so on. If you have enough control of the system that there are no such processes, then you can easily do the swap directory by directory: just do it with system calls out of a single process, rather than by invoking tools, so that nothing trips up at the point in time when you have a mismatched /bin and /lib.
  - rektide 4y ago
    Nothing you'be talked about sounds like a remotely normal thing that is documented or discussed anywhere and I doubt there's more than a fistful of orgs on the planet that have jury rigged that up. Meanwhile there are at least a dozen distros doing atomic image upgrades. Fedora SilverBlue for one example. These upgrades often involve a kexec restart. For fleets of servers- the target audience- it's expected that machines will go down & rolling restarts should be fine. That's how true atomic behavior is gotten. But I think you are nitpicking & being fussy with your complaining about old processes being left around; even if you do an in-place swap, but have some old processes around, at least the whole system is together & consistent. More broadly, thjs set of problem in general doesnt change much whether you are using a merged /usr or not. I genuinely dont get any of your disgust at having a /usr image that contains the functioning system image. You talk about chroots, as though it's an alternative somehow, but this practice of having a self-contained image seems like it would greatly assist in quickly cloning new images to chroot into.
    - kazinator 4y ago
      If "atomic" means "stop the world running one system, and reboot into a different system", we can easily arrange to reshuffle two or more directories. The program which does that can be easily designed not to be affected by the intermediate states, like where you have a new /lib, but still the old /bin.

db48x 4y ago

Yea, the real goal here is to be able to mount the whole OS in one operation. To that end, /dev is now autopopulated by udev and you can even leave /etc completely empty. The goal is for everything to have good enough defaults that having a config file in /etc is unimportant unless the admin wants to override something. Probably the only things in there when you install a new system would be the root password in /etc/shadow and your user and group information in /etc/passwd and /etc/group. It should make a lot of tasks a lot easier. The one I like most is trivially setting up containers or VMs that have your base OS in them plus some new software that you are testing. For example, at work we use Debian with a bunch of extra packages installed, plus our own package containing all manner of software and configuration that we’ve written. Although we do a fair amount of unit testing, and the Rust compiler really has our back, it is very difficult to really test things outside of production. Currently we have to painstakingly set up a test environment on a production machine, using tricks like altering our PATH so that the test version of the tools we are working on will be called, or editing things so that they call the version under test instead. It’s doable but laborious. If we were using Fedora instead of Debian, we would be able to use systemd-nspawn to spawn a container with the existing OS, read–only mounts containing our databases and other large datasets, and an overlayfs to redirect any writes to those file systems to an alternate location. Installing and running our software inside that container would work exactly as if we were in production, and testing everything would be so much simpler. systemd-nspawn is available on Debian, but running it requires keeping extra copies of the OS for each container you want to spawn, so we haven’t actually gotten around to doing it yet. Oh, and each container needs a properly set up /etc directory. We can’t just leave it empty, and we can’t just blindly copy the host’s either.

p_l 4y ago

This myth however was inverted by people pushing for /usr merge on various media platforms at the time. A common example of "what's wrong with split /usr" that was pushed was things like dropping pci.ids database in /usr/share and it being required by code mounting /usr, and that mounting / and /usr separately in initrd was a wasteful complexity. In fact, today is the first time I have heard of reasoning that supported /usr on separate partition and I thought it was just an inversion done later by people who noticed possible benefits for some extra initrd complexity (btw, dracut and systemd oriented initrd scripts are stupidly complex. Including generating systemd units by text templating in udev rules as part of dracut boot process. Wtf)