6 ms·

Bootc and OSTree: Modernizing Linux System Deployment

123 points by mrtedbear 7mo ago

Borealid 7mo ago
I like the idea of using the same format for kernel-included VMs as I use for containers. Next up, backups stored as layers in the same OCI registries. I am not, however, sure ostree is going to be the final image format. Last time I looked work was in progress to replace that.
- mroche 7mo ago
  It is not, the future is currently pointing to composefs: https://github.com/bootc-dev/bootc/issues/1190 https://github.com/bootc-dev/bootc/issues/1190 There's a GitHub org that builds bootc-ready images for non-Red Hat family distributions using this backend. https://github.com/bootcrew https://github.com/bootcrew
selfhosting_sh 7mo ago
[flagged]
pojntfx 7mo ago
bootc and OSTree are both very neat, but the leading edge of immutable Linux distros (GNOME OS, KDE Linux) is currently converging on a different proposal by systemd developers that's standardized by the UAPI Group (https://uapi-group.org/specifications/ https://uapi-group.org/specifications/). It fixes quite a few of the complexities with OSTree (updates are handled by `systemd-sysupdate`/`updatectl` and are just files served via HTTP) and is quite a bit easier to extend with things like an immutable version of the Nvidia drivers or codecs thanks to system extensions handled by `systemd-sysext` (which in turn are just simple squashfs files overlayed over `/usr`) and configuration via `systemd-confext`. `mkosi`, also by systemd, is quickly becoming _the_ way to build custom images too, and is somewhat tied to these new standards.
- smashed 7mo ago
  > the bleeding edge of immutable Linux distros (GNOME OS, KDE Linux) These are words but they don't make sense.
  - n42 7mo ago
    "some of the newer ideas happening in this space are in the GNOME OS project and the KDE Linux project"
    - hparadiz 7mo ago
      My Gentoo box is immutable. Right up until I run emerge.
  - pojntfx 7mo ago
    Corrected - I meant leading edge. Context re:distros mentioned: GNOME OS: https://os.gnome.org/ https://os.gnome.org/ KDE Linux: https://kde.org/linux/ https://kde.org/linux/
    - 7777777phil 7mo ago
      Doubling progression-free survival (17.6 vs 7.4 months) is a large effect size for recurrent prostate cancer.
      - Scipio_Afri 7mo ago
        I’m pretty sure Linux doesn’t have a prostate, even with all the changes in the leading edge distros, and you’re commenting in the wrong post.
        7777777phil 7mo ago
        Pretty sure I didn’t want to post that here. But then I got rate limited and upon coming out of rate limit jail blindly pasted this comment where my page reloaded - my bad should have been here: https://news.ycombinator.com/item?id=47193047 https://news.ycombinator.com/item?id=47193047
deleted 7mo ago
[deleted]
azibi 7mo ago
We use TorizonOS, which is also based on OSTree: https://www.torizon.io/blog/ota-best-linux-os-image-update-model https://www.torizon.io/blog/ota-best-linux-os-image-update-m.... It works quite well for our edge devices. It’s tightly integrated with Toradex hardware, but not limited to it. It may seems litte a niche, but it has strong potential for long‑term supported edge products. Any additional experiences to share?
- tuananh 7mo ago
  bootc is kind of perfect for edge. delivering OS update as a whole. ease of update/rollback.
lproven 7mo ago
It is very odd to me to watch OStree-based distros starting to take off and win recruits. The only reason Red Hat needed to invent this very complex mechanism was because RH does not officially have a COW-snapshot capable filesystem in its enterprise distro. A filesystem with snapshots makes software installation transactional. You take a snapshot, install some software, and if it doesn't work right, you can revert to the snapshot. (With very slightly more flexible snapshots, you can limit the snapshot to just some part of the directory tree, but this is not essential; it merely permits more flexibility.) In other words, you are a long way toward what in database language is called ACID: https://en.wikipedia.org/wiki/ACID https://en.wikipedia.org/wiki/ACID Atomicity, consistency, isolation, durability. It makes your software inastallation transactional: an update either happens completely (A), you can check it is valid (C) and works (I), or it can be totally reverted, and the system restored to the earlier state (D). That's a good thing. It means you can safely automate software deployment knowing that if it goes wrong you have an Undo mechanism. Databases got this 50+ years ago; in the 21st century it's making its way to FOSS OSes. Do this in the filesystem and it's easy. SUSE's implementation is so simple, it's basically a bunch of shell scripts, and it can be turned on and off. You can run an immutable OS, reboot for updates, and if you need, disable it, go in and fix the system, and then turn it back on again. This is because SUSE leans very heavily on Btrfs and that is the critical weakness -- Btrfs is only half finished and is not robust. But RH removed Btrfs from RHEL and Btrfs was the only GPL COW filesystem, so core infrastructure in the distro means no COW on RH. Oracle Linux has Btrfs -- the FS was developed at Oracle, after all -- and so does Alma. (Yes I know, Fedora put it back, but the key thing is, it only uses Btrfs only for compression so that Flatpak looks less horrendously inefficient. Fedora doesn't use snapshots.) With no COW FS, RH had to invent a way to do transactional updates without filesystem support. Result, OStree. Git, but for binaries. And yes, everyone developing FOSS uses Git, but almost nobody understands Git: https://xkcd.com/1597/ https://xkcd.com/1597/ You know that if there's an Xkcd about it, it must be true. Embedding something you don't understand in your OS design is a VERY BAD PLAN. With OStree your FS is a virtual one, it's not real, it's synthesized on the fly from a local repository. The real FS is hidden and can't be hand-edited or anything. It generates the OS filesystem tree on the fly, you see. OS-tree. Use it just for GUI apps, that's Flatpak. Use it for the whole OS, that's OStree. It is so mind-shreddingly complicated that you can't do package management any more, you can't touch the underlying FS. So you need a whole new set of layers on top: virtual directories on top of the main virtual directory, and some bits with extra pseudo-filesystems layered on top of that to make some bits read-write. It's like the scene in the Wasp Factory where under the skull plate it's just writhing maggots. I recall in horror and revulsion when I see it. So it's deeply bizarre to read blog posts praising all the cool stuff you can do with it.
YorickPeterse 7mo ago
For those looking for a more extensive article about bootc, I recently wrote about using it in https://yorickpeterse.com/articles/self-hosting-my-websites-using-bootable-containers/ https://yorickpeterse.com/articles/self-hosting-my-websites-..., including a comparison to some other existing tools.
nicman23 7mo ago
developers will do anything but to use a cow fs
- bogwog 7mo ago
  Can you blame them? https://www.theregister.com/2026/02/25/bcachefs_creator_ai/ https://www.theregister.com/2026/02/25/bcachefs_creator_ai/
  - nicman23 7mo ago
    my man openzfs and btrfs is a thing
- okanat 7mo ago
  You cannot replicate the full filesystem among multiple hosts with their own drives though, can you? With OSTree my team can deploy a commit over an existing ones to a field of 60k embedded Linux devices. Similarly bootc is a great alternative for deploying images for dedicated or virtualized servers.
  - nicman23 7mo ago
    yeah you can? what you on about
    - okanat 7mo ago
      How do you tag / mark a commit a snapahot of all files in an OS image that's independent from each device's local state and then move them from one commit to another atomically? Basically a `git pull` on a branch but binary and atomic. CoW is strictly local. You can make a snapshot of local state sure and even atomically change it. But that's not we want.
      - nicman23 7mo ago
        if i know all devices are on the same root? you just ship the snapshot diff and if it does not sync to the fs you just resync it all
iamcalledrob 7mo ago
I'd love to have my system be declared in code, so I can replicate the same environment across a laptop and a desktop with minimal drift. So same OS, users, packages, flatpaks etc. And a mostly synced home dir too. Is NixOS the only viable way to do this? I don't like the path mangling that Nix introduces. It seems like an immutable distro customized via a Containerfile could work too? Except rebooting/reimagine for every change sounds tedious as hell.
- jcastro 7mo ago
  > customized via a Containerfile could work too? Except rebooting/reimagine for every change sounds tedious as hell. You can do this today with Aurora, Bazzite, Bluefin, and other bootc systems. The system updates by default are weekly and require a reboot but when you move most of the stuff into the userspace most of that stuff updates independently anyway.
  - aaravchen 7mo ago
    In fact, if you want to use something like Nix on a UniversalBlue system, you have to spin your own. The "hotfix" and chattr solutions of pre-composefs don't work anymore. Anything that needs to go into a read only location and isn't package as an RPM requires you to "spin your own". Luckily UniversalBlue makes it incredibly easy, they have a template repo you can use that has all the GitHub action setup included to auto-bills on every change, and directions for how to set it up. It took me about 10 minutes
- aaravchen 7mo ago
  All the immutable system solutions out there pretty much all make your rootfs immutable, but leave your home folder and system config folders (i.e. /var and /etc) as mutable. It's pretty obvious that if you make the config folders and/or home folder immutable it starts causing most people problems, since in the vast majority of cases people just want to be able to persistently change the desktop background color or spaces vs tabs setting in their IDE without having to locate the setting in a full system config, set it, and regenerate. This does cause some interesting tension in the immutability though. /etc in particular is really a mix of things that a sysadmin should really only be setting, and things a regular user may set indirectly. This usage has grown organically over time with the tools involved in the implementation, so it's not at all consistent which are which. The immutable system solutions recognize this by usually handling the whole /etc folder the same way package managers handle package installs that include /etc file: by doing a 3-way merge between the old provided files, the new provided files, and the current existing files to see if the existing are unchanged from the old provided and can just be directly replaced by the new provided or if a merge conflict needs resolving. Additionally, a separate copy of /etc is maintained associated with each available bootable system version so when you roll back you get the old /etc files you had before. Though this does introduce a system-unique variation since you now have new /etc being affected by the state of /etc when it was forked. If you want all your home folder and system config to be identical, nix or guix really are your primary way to go, that extra lockdown of the user and system config is exactly what most people don't want for usability reasons. I personally use nix home-manager on top of Aurora DX from Universal Blue. I have my nix home-manager config setup to manage only the things I want to be locked down in my home config, and to provide some extra tools that are easier to manage/supply via Nix than a system package manager (where I would need to do a whole system update to get the new version). My IDE for example is installed on a specific version via Nix, but I don't have Nix manage the settings of it so I can separately tweak as needed without need a home-manager rebuild. EDIT: typo
samhclark 7mo ago
Personally, I've really enjoyed using bootc for both my personal laptop and my NAS. I really like the NAS use case because I can build the ZFS kmods for that specific version of Fedora CoreOS in CI/CD. If there's any compatibility failure, then my NAS doesn't get an update and I get the CI/CD failed email. No downtime because of some kernel incompatibility. For the laptop though, I feel like there's a better way that I haven't found. Some way not to require CI/CD, to build the next image and switch to it all locally. I haven't gone down that path yet, but it looks kinda like that Option 2 the author described. Maybe it's really just that easy. I've really been enjoying this space.
kj4ips 7mo ago
I like the idea behind ostree and bootc, but I feel that OCI (with tarlayers) is not a good fit. `repack` makes an absolute hash of things, and since the layers are logically packaged, they will have to be composed somehow, and then ostree becomes only slightly more useful than coreos's A/B usr. OCI roughly assumes that layers will be laid out in some logical way, and that a given host will see opportunities to reuse across different instances, but with bootc, there will only ever be one instance. OCI also assumes that individual layers are small enough that it is always worth pulling and unpacking a layer instead of some kind of authentication delta, which is great for a k8s cluster in a center, but not great for devices out on the edge, where you might want this kind of pseudo-immutable system even more. I really want some standardized way for a manifest in OCI to say that "this content is also available in other format X here".
Levitating 7mo ago
Am I the only one that isn't interested in building OCI images just to install a webserver? rpm-ostree worked fine for me

123 points by mrtedbear 7mo ago

I like the idea of using the same format for kernel-included VMs as I use for containers. Next up, backups stored as layers in the same OCI registries. I am not, however, sure ostree is going to be the final image format. Last time I looked work was in progress to replace that.

mroche 7mo ago
It is not, the future is currently pointing to composefs: https://github.com/bootc-dev/bootc/issues/1190 https://github.com/bootc-dev/bootc/issues/1190 There's a GitHub org that builds bootc-ready images for non-Red Hat family distributions using this backend. https://github.com/bootcrew https://github.com/bootcrew

selfhosting_sh 7mo ago

[flagged]

pojntfx 7mo ago

bootc and OSTree are both very neat, but the leading edge of immutable Linux distros (GNOME OS, KDE Linux) is currently converging on a different proposal by systemd developers that's standardized by the UAPI Group (https://uapi-group.org/specifications/ https://uapi-group.org/specifications/). It fixes quite a few of the complexities with OSTree (updates are handled by `systemd-sysupdate`/`updatectl` and are just files served via HTTP) and is quite a bit easier to extend with things like an immutable version of the Nvidia drivers or codecs thanks to system extensions handled by `systemd-sysext` (which in turn are just simple squashfs files overlayed over `/usr`) and configuration via `systemd-confext`. `mkosi`, also by systemd, is quickly becoming _the_ way to build custom images too, and is somewhat tied to these new standards.

smashed 7mo ago
> the bleeding edge of immutable Linux distros (GNOME OS, KDE Linux) These are words but they don't make sense.
- n42 7mo ago
  "some of the newer ideas happening in this space are in the GNOME OS project and the KDE Linux project"
  - hparadiz 7mo ago
    My Gentoo box is immutable. Right up until I run emerge.
- pojntfx 7mo ago
  Corrected - I meant leading edge. Context re:distros mentioned: GNOME OS: https://os.gnome.org/ https://os.gnome.org/ KDE Linux: https://kde.org/linux/ https://kde.org/linux/
  - 7777777phil 7mo ago
    Doubling progression-free survival (17.6 vs 7.4 months) is a large effect size for recurrent prostate cancer.
    - Scipio_Afri 7mo ago
      I’m pretty sure Linux doesn’t have a prostate, even with all the changes in the leading edge distros, and you’re commenting in the wrong post.
      - 7777777phil 7mo ago
        Pretty sure I didn’t want to post that here. But then I got rate limited and upon coming out of rate limit jail blindly pasted this comment where my page reloaded - my bad should have been here: https://news.ycombinator.com/item?id=47193047 https://news.ycombinator.com/item?id=47193047

deleted 7mo ago

[deleted]

azibi 7mo ago

We use TorizonOS, which is also based on OSTree: https://www.torizon.io/blog/ota-best-linux-os-image-update-model https://www.torizon.io/blog/ota-best-linux-os-image-update-m.... It works quite well for our edge devices. It’s tightly integrated with Toradex hardware, but not limited to it. It may seems litte a niche, but it has strong potential for long‑term supported edge products. Any additional experiences to share?

tuananh 7mo ago
bootc is kind of perfect for edge. delivering OS update as a whole. ease of update/rollback.

lproven 7mo ago

It is very odd to me to watch OStree-based distros starting to take off and win recruits. The only reason Red Hat needed to invent this very complex mechanism was because RH does not officially have a COW-snapshot capable filesystem in its enterprise distro. A filesystem with snapshots makes software installation transactional. You take a snapshot, install some software, and if it doesn't work right, you can revert to the snapshot. (With very slightly more flexible snapshots, you can limit the snapshot to just some part of the directory tree, but this is not essential; it merely permits more flexibility.) In other words, you are a long way toward what in database language is called ACID: https://en.wikipedia.org/wiki/ACID https://en.wikipedia.org/wiki/ACID Atomicity, consistency, isolation, durability. It makes your software inastallation transactional: an update either happens completely (A), you can check it is valid (C) and works (I), or it can be totally reverted, and the system restored to the earlier state (D). That's a good thing. It means you can safely automate software deployment knowing that if it goes wrong you have an Undo mechanism. Databases got this 50+ years ago; in the 21st century it's making its way to FOSS OSes. Do this in the filesystem and it's easy. SUSE's implementation is so simple, it's basically a bunch of shell scripts, and it can be turned on and off. You can run an immutable OS, reboot for updates, and if you need, disable it, go in and fix the system, and then turn it back on again. This is because SUSE leans very heavily on Btrfs and that is the critical weakness -- Btrfs is only half finished and is not robust. But RH removed Btrfs from RHEL and Btrfs was the only GPL COW filesystem, so core infrastructure in the distro means no COW on RH. Oracle Linux has Btrfs -- the FS was developed at Oracle, after all -- and so does Alma. (Yes I know, Fedora put it back, but the key thing is, it only uses Btrfs only for compression so that Flatpak looks less horrendously inefficient. Fedora doesn't use snapshots.) With no COW FS, RH had to invent a way to do transactional updates without filesystem support. Result, OStree. Git, but for binaries. And yes, everyone developing FOSS uses Git, but almost nobody understands Git: https://xkcd.com/1597/ https://xkcd.com/1597/ You know that if there's an Xkcd about it, it must be true. Embedding something you don't understand in your OS design is a VERY BAD PLAN. With OStree your FS is a virtual one, it's not real, it's synthesized on the fly from a local repository. The real FS is hidden and can't be hand-edited or anything. It generates the OS filesystem tree on the fly, you see. OS-tree. Use it just for GUI apps, that's Flatpak. Use it for the whole OS, that's OStree. It is so mind-shreddingly complicated that you can't do package management any more, you can't touch the underlying FS. So you need a whole new set of layers on top: virtual directories on top of the main virtual directory, and some bits with extra pseudo-filesystems layered on top of that to make some bits read-write. It's like the scene in the Wasp Factory where under the skull plate it's just writhing maggots. I recall in horror and revulsion when I see it. So it's deeply bizarre to read blog posts praising all the cool stuff you can do with it.

YorickPeterse 7mo ago

For those looking for a more extensive article about bootc, I recently wrote about using it in https://yorickpeterse.com/articles/self-hosting-my-websites-using-bootable-containers/ https://yorickpeterse.com/articles/self-hosting-my-websites-..., including a comparison to some other existing tools.

nicman23 7mo ago

developers will do anything but to use a cow fs

bogwog 7mo ago
Can you blame them? https://www.theregister.com/2026/02/25/bcachefs_creator_ai/ https://www.theregister.com/2026/02/25/bcachefs_creator_ai/
- nicman23 7mo ago
  my man openzfs and btrfs is a thing
okanat 7mo ago
You cannot replicate the full filesystem among multiple hosts with their own drives though, can you? With OSTree my team can deploy a commit over an existing ones to a field of 60k embedded Linux devices. Similarly bootc is a great alternative for deploying images for dedicated or virtualized servers.
- nicman23 7mo ago
  yeah you can? what you on about
  - okanat 7mo ago
    How do you tag / mark a commit a snapahot of all files in an OS image that's independent from each device's local state and then move them from one commit to another atomically? Basically a `git pull` on a branch but binary and atomic. CoW is strictly local. You can make a snapshot of local state sure and even atomically change it. But that's not we want.
    - nicman23 7mo ago
      if i know all devices are on the same root? you just ship the snapshot diff and if it does not sync to the fs you just resync it all

iamcalledrob 7mo ago

I'd love to have my system be declared in code, so I can replicate the same environment across a laptop and a desktop with minimal drift. So same OS, users, packages, flatpaks etc. And a mostly synced home dir too. Is NixOS the only viable way to do this? I don't like the path mangling that Nix introduces. It seems like an immutable distro customized via a Containerfile could work too? Except rebooting/reimagine for every change sounds tedious as hell.

jcastro 7mo ago
> customized via a Containerfile could work too? Except rebooting/reimagine for every change sounds tedious as hell. You can do this today with Aurora, Bazzite, Bluefin, and other bootc systems. The system updates by default are weekly and require a reboot but when you move most of the stuff into the userspace most of that stuff updates independently anyway.
- aaravchen 7mo ago
  In fact, if you want to use something like Nix on a UniversalBlue system, you have to spin your own. The "hotfix" and chattr solutions of pre-composefs don't work anymore. Anything that needs to go into a read only location and isn't package as an RPM requires you to "spin your own". Luckily UniversalBlue makes it incredibly easy, they have a template repo you can use that has all the GitHub action setup included to auto-bills on every change, and directions for how to set it up. It took me about 10 minutes
aaravchen 7mo ago
All the immutable system solutions out there pretty much all make your rootfs immutable, but leave your home folder and system config folders (i.e. /var and /etc) as mutable. It's pretty obvious that if you make the config folders and/or home folder immutable it starts causing most people problems, since in the vast majority of cases people just want to be able to persistently change the desktop background color or spaces vs tabs setting in their IDE without having to locate the setting in a full system config, set it, and regenerate. This does cause some interesting tension in the immutability though. /etc in particular is really a mix of things that a sysadmin should really only be setting, and things a regular user may set indirectly. This usage has grown organically over time with the tools involved in the implementation, so it's not at all consistent which are which. The immutable system solutions recognize this by usually handling the whole /etc folder the same way package managers handle package installs that include /etc file: by doing a 3-way merge between the old provided files, the new provided files, and the current existing files to see if the existing are unchanged from the old provided and can just be directly replaced by the new provided or if a merge conflict needs resolving. Additionally, a separate copy of /etc is maintained associated with each available bootable system version so when you roll back you get the old /etc files you had before. Though this does introduce a system-unique variation since you now have new /etc being affected by the state of /etc when it was forked. If you want all your home folder and system config to be identical, nix or guix really are your primary way to go, that extra lockdown of the user and system config is exactly what most people don't want for usability reasons. I personally use nix home-manager on top of Aurora DX from Universal Blue. I have my nix home-manager config setup to manage only the things I want to be locked down in my home config, and to provide some extra tools that are easier to manage/supply via Nix than a system package manager (where I would need to do a whole system update to get the new version). My IDE for example is installed on a specific version via Nix, but I don't have Nix manage the settings of it so I can separately tweak as needed without need a home-manager rebuild. EDIT: typo

samhclark 7mo ago

Personally, I've really enjoyed using bootc for both my personal laptop and my NAS. I really like the NAS use case because I can build the ZFS kmods for that specific version of Fedora CoreOS in CI/CD. If there's any compatibility failure, then my NAS doesn't get an update and I get the CI/CD failed email. No downtime because of some kernel incompatibility. For the laptop though, I feel like there's a better way that I haven't found. Some way not to require CI/CD, to build the next image and switch to it all locally. I haven't gone down that path yet, but it looks kinda like that Option 2 the author described. Maybe it's really just that easy. I've really been enjoying this space.

kj4ips 7mo ago

I like the idea behind ostree and bootc, but I feel that OCI (with tarlayers) is not a good fit. `repack` makes an absolute hash of things, and since the layers are logically packaged, they will have to be composed somehow, and then ostree becomes only slightly more useful than coreos's A/B usr. OCI roughly assumes that layers will be laid out in some logical way, and that a given host will see opportunities to reuse across different instances, but with bootc, there will only ever be one instance. OCI also assumes that individual layers are small enough that it is always worth pulling and unpacking a layer instead of some kind of authentication delta, which is great for a k8s cluster in a center, but not great for devices out on the edge, where you might want this kind of pseudo-immutable system even more. I really want some standardized way for a manifest in OCI to say that "this content is also available in other format X here".

Levitating 7mo ago

Am I the only one that isn't interested in building OCI images just to install a webserver? rpm-ostree worked fine for me