13 ms·

Immutable Linux distributions

170 points by unripe_syntax 4y ago

Zurrrrr 4y ago
Rather than use a dedicated OS for something like this, using an SELinux (or similar) policy makes more sense and grants more flexibility.
- maksimur 4y ago
  Could you elaborate?
  - Zurrrrr 4y ago
    Something like SELinux would let you make any distro just as immutable as any of those distros in the list, to a finer level of granularity, and without having to use a specialized distro.
    - TheDong 4y ago
      Sure, and on any distro you can "sudo mount -o remount,ro /" and also have a useless distro, just like using selinux to make the whole thing immutable. The difference is you won't have to spent 2 years learning a custom policy language.
      - Zurrrrr 4y ago
        > also have a useless distro, just like using selinux to make the whole thing immutable. Absolute nonsense. > The difference is you won't have to spent 2 years learning a custom policy language. If you had bothered to spend maybe 2 months bothering to learn this tech that's been around for almost 20 years, you wouldn't have to use specialty distros to accomplish what your distro can already do natively. I've never been a fan of willful ignorance and fearmongering, and I see a lot of that when it comes to SELinux.
        toyg 4y ago
        If it's been around 20 years and most people still loathe it, it probably isn't nice to work with.
        Zurrrrr 4y ago
        It could certainly be a lot nicer and more straightforward (and there are competing solutions that are), but people vastly exaggerate the difficulty of learning it, and dismiss the advantages that come from having it properly configured.
        TheDong 4y ago
        If your goal is a usable immutable distro, selinux is the wrong tool. It just doesn't operate at the layer of creating a linux distro, or declaring what containers are running, or declaring a list of flatpak apps, or whatever. My point, by comparing selinux to "mount" and calling them equally useless, was not that they are not useful tools in their niches. My point was that they are useless if your goal is to build a usable immutable distro. SELinux is a hammer that cannot be used to turn a default debian installation into a usable immutable linux distro. The initial claim of "using SELinux rather than a dedicated distro can work" was nonsense, so of course the thread of replies off this is nonsense. GIGO as they say.
        Zurrrrr 4y ago
        > SELinux is a hammer that cannot be used to turn a default debian installation into a usable immutable linux distro. It's only a hammer if you use it as one. It's a very fine grained tool and it can be as precise as you want it to be. > The initial claim of "using SELinux rather than a dedicated distro can work" was nonsense It wasn't at all, and it's baffling to me why anyone would claim so.
    - t-3 4y ago
      People that want an entirely containerized distro probably don't want to maintain a bunch of SELinux rules or deal with the inevitable breakages that occur due to old/poorly designed software that is needed for one reason or another. It's entirely opposite of the goal of making everything a drop-in no-maintenance package.
    - tremon 4y ago
      SELinux isn't about immutability, it's about program confinement. Running each daemon in its own fs/user/net namespace comes a lot closer to mimicking the value of SELinux than making the OS immutable.
      - Zurrrrr 4y ago
        > SELinux isn't about immutability, it's about program confinement. The mechanisms that allow for program confinement also allow you to construct an immutable OS. > Running each daemon in its own fs/user/net namespace comes a lot closer to mimicking the value of SELinux than making the OS immutable. Maybe, but it's still pretty far off. SELinux is about removing DAC/an all powerful user much more than it is about sandboxing.
    - throwaway894345 4y ago
      How? How do you get immutability from SELinux policies (specifically without crafting detailed policies for everything running in the entire userland--at which point you're just begging for a distro, right?)? Honestly, if you really want something immutable and distro-agnostic, you probably want something like btrfs snapshots; however, converting the root filesystem to btrfs is probably a pain, and there will be some initial configuration to prepare some cloud-init, ignition, etc before you take your initial snapshot and you'll also probably want to configure some "boot from snapshot" type functionality, at which point it would be pretty nice to have all of this packaged up in a distro so each user doesn't have to figure all of that out every time.
- FireInsight 4y ago
  Well, with a dedicated Linux distro, you get it ootb and it's supported by people other than you. Also with Silverblue, there's a mechanism for getting the base OS from a OCI container image (check out https://ublue.it/ https://ublue.it/)
  - Zurrrrr 4y ago
    > Well, with a dedicated Linux distro, you get it ootb and it's supported by people other than you. For sure, I can see there are use cases and a market for it.
fsflover 4y ago
Qubes OS should have been mentioned. It's not a Linux distro and it's not immutable, but it allows to create disposable virtual machines from customized templates. For me, it is much easier to manage.
- FireInsight 4y ago
  > It's not a linux distro and it's not immutable. Then it wouldn't fit with the lists theme?
  - fsflover 4y ago
    It depends on the reason why you need it. For me, it's security, and Qubes allows to do it more conveniently.
  - palata 4y ago
    Well, there are distros in the list that are optimized for containers. If you use Linux distros in Qubes OS, then I would say that it counts as a Linux distro (by default you get Fedora). And by default you run in VMs, where some parts are persistent (e.g. your home) and some are not. Just like running a container with some persistent storage. It depends on your use-case whether you like it or not, but I would definitely put Qubes OS in the list.
- maksimur 4y ago
  Qubes OS is a hassle to set up and it's too heavy in my experience. Also I couldn't suspend to RAM or hibernate because my hardware wasn't fully supported.
  - fsflover 4y ago
    It depends on the hardware, see https://forum.qubes-os.org/t/community-recommended-computers/5560 https://forum.qubes-os.org/t/community-recommended-computers.... On my laptop suspend to RAM works flawlessly. Hibernation is not implemented in Qubes yet.
zerof1l 4y ago
I see this being very useful for containerized use cases. Usually, you have a Docker file that installs and sets everything once at container creation time. After that, you don't want anything to change in the OS. Great for security.
- eru 4y ago
  Yes. If you want to change anything, you rebuild from source. Exactly what we are doing when we are eg working on a C program: we don't monkey patch the binary, we run the build system again.
- throwaway894345 4y ago
  I agree, but at least in many cases, containers aren't the immutability mechanism. E.g., MicroOS seems to rely on btrfs snapshots--all changes are dropped on reboot and the system starts from a fresh factory state. Your application could be containerized or it could be a package pulled from the distro repository and managed by systemd.
plaguepilled 4y ago
No NixOS? No Guix? C'mon those are the best ones!
- rgoulter 4y ago
  Right. NixOS' absence from the list is notable. c.f. Google trends for "nixos, fedora silverblue, guix" https://trends.google.com/trends/explore?q=nixos,fedora%20silverblue,guix&hl=en https://trends.google.com/trends/explore?q=nixos,fedora%20si... which suggests NixOS is relatively more searched for than silverblue.
  - eloisant 4y ago
    I guess you can use NixOS for your desktop but that's not the main focus of this distribution. It's more to create images to deploy server-side applications.
    - rgoulter 4y ago
      c.f. Nix survey results from last year, which indicates many of the respondents use NixOS as their daily Linux distribution. (e.g. number of responses for just "uses gnome" and "uses plasma" are about half of the number of responses for "use NixOS daily"). https://discourse.nixos.org/t/2022-nix-survey-results/18983 https://discourse.nixos.org/t/2022-nix-survey-results/18983 Many of those who do use NixOS as a desktop Linux are enthusiastically happy about it. Indeed, this user described the combination of nice features & rough edges as "cursed". (It's so good you won't want to use anything else, but it's too rough to recommend generally). https://blog.wesleyac.com/posts/the-curse-of-nixos https://blog.wesleyac.com/posts/the-curse-of-nixos
    - plaguepilled 4y ago
      This comment threw me off hard because of how loaded it is. 1. How does that address the fact that NixOS is immutable? 2. If discussing server side applications was the intention of the author, why didn't they make that the explicit focus of the article? As the article currently stands, that's not the stated topic, but it is a point included in the article about why immutable OSes are cool. That's a noticeably different reality. 3. Even if we look past the above, NixOS can be used for server side deployment and in many cases is favourable to, say, Ansible deployments, so that STILL doesn't explain why it was left off!
hendry 4y ago
My OS for Web kiosks was immutable since 2007. It's what people wanted and needed.
heisenzombie 4y ago
It’s much less plug-and-play than the options in the article… but I’ve had fun with a related idea, which is running Nixos on storage where root is erased on every boot. My home server is set up this way. At the time, I more or less followed this guide, which explains things nicely: https://grahamc.com/blog/erase-your-darlings/ https://grahamc.com/blog/erase-your-darlings/ Been erasing root for almost 2 years I think now. Works great!
- LanternLight83 4y ago
  I've cloned this setup for BTRFS on Guix, which took some questionable hacks because they don't have all the hooks I needed throughout the boot process... still meaning to migrate it to ZFS on Root, which will entail yet more hacks. Going for it though c:
  - tripleo1 4y ago
    > I've cloned this setup for BTRFS on Guix Please share :)
- ochoseis 4y ago
  One of the hurdles with this approach is that _everything_ has to live in configuration.nix. How are you handling secrets like your host keys? I recently got sops-nix working as a prototype, but curious if there are better ways. Side note: everything living in configuration.nix makes flake-ifying your system a lot easier, which is a win in my book.
  - anotherhue 4y ago
    There's a nix way to read a secret from a file. I'm not sure if it still ends up in the nix store that way but at least it's not in the config file so your VCS is clean.
    - piperswe 4y ago
      Typically with NixOS, you'll have systemd or a wrapper script read the secret from a file into an environment variable for the service that needs the secret. This secret file would be stored in `/var/lib` or similar, outside the Nix store. There isn't really a "Nix" way, just a pattern used in various places in NixOS.
charcircuit 4y ago
This is missing the most poplar immutable Linux distribution. Android.
- cleanchit 4y ago
  Yeah it's immutable but the build process is a pain in the ass. For nixos, the input is a couple text files and the process is a single command.
  - juliosueiras 4y ago
    why not both ;) https://github.com/danielfullmer/robotnix https://github.com/danielfullmer/robotnix
- otabdeveloper4 4y ago
  Android is not a Linux distribution. Whatever runs in your phone is only slightly based on stock kernel.org tarballs.
  - zamnos 4y ago
    But they're still based on that kernel that's hosted at that Linux website? Do you consider ChromeOS a Linux distro? it's based on Gentoo.
    - palata 4y ago
      I don't know about ChromeOS, but I would definitely consider that Android is not a Linux distribution. To me, a Linux distribution is an OS where, given that I am very comfortable with another Linux distribution, I will quickly be able to find my way around. Does Android have services? As root, can I start/stop them? Can I install libraries somewhere and use them somehow? How do I package an app (other than by running "build" in Android-Studio and letting the magic happen)? Could I install my own window manager? => I don't know any of that, so it's not a Linux distro. As a comparison, SteamOS is based on Arch Linux and has a particular setup (with a read-only "core" partition I guess?), but I am pretty sure that if you give me a shell, I will feel like I am in a Linux distro.
boris 4y ago
Most of these focus on running containers. What I would like is a distribution that focuses on the reliability of upgrades, both configuration and data. To give a concrete example, I have a Debian server running Apache (with auto-certificate renewal), PostgreSQL, and Postfix. When it's time to upgrade to the next release, it's basically the "it should work, but who really knows, depends on your specific setup you may have to tweak a few things" kind of guarantee. What I want is a rigorous guarantee like we have for relational database schema/data migration.
- znpy 4y ago
  I’ve recently switched to fedora and I’m having a great time in this sense.
- ajdude 4y ago
  You would probably get a lot out of NixOS. Sans that, I wonder how Fedora Silverblue would do as a server.
  - itsmartapuntocm 4y ago
    Fedora IoT is essentially the server variant of Silverblue.
  - bonzini 4y ago
    Fedora Silverblue is to workstation what Fedora CoreOS is to servers, so the underlying technology would fare very well.
- evol262 4y ago
  This _is_ Silverblue and other ostree-based repos. This article is focused on containers because when the root OS is immutable, then mutable developer workflows/etc necessarily happen in containers (distrobox, toolbox, whatever). OSTree is/was commonly described as "git for filesystems". Every update is an entirely new system image which applies a 3-way diff. It's possibly to state in a single command "show me the drift in my config files/data versus the ref I'm running" and/or "show me the diff between my running system and an update I may apply".
eloisant 4y ago
While it's not yet easy to install on anything else than a Steam Deck, SteamOS is also a great example of an immutable distribution.
nathants 4y ago
i want a pc game that i reboot into. an immutable linux iso and usb boot.
tmitchel2 4y ago
Is it not possible to have a client / desktop OS where each app runs in its own container by default with its own writeable filesystem... I would have thought that's beneficial from a security perspective as well as making things easier to separate...
- ElectricalUnion 4y ago
  > Is it not possible to have a client / desktop OS where each app runs in its own container by default with its own writeable filesystem... We have several "container as app" solutions around: * Microsoft Universal Windows Platform; * Canonical snaps; * Flatpak; * containertoolbx/Distrobox (this one is very DIY and what I use); It's just that (as of now) most apps rely (and are allowed to be deployed in "stores") on very leaky container isolation (like full filesystem access) so might as well not be deployed inside containers in the first place.
- qbasic_forever 4y ago
  Flatpaks do exactly that.
  - palata 4y ago
    https://ludocode.com/blog/flatpak-is-not-the-future https://ludocode.com/blog/flatpak-is-not-the-future
jacooper 4y ago
Do any of the immutable server oses like coreOS and MicroOS make any sense for a bare-metal one-instance installation? It seems they are more optimized for temporary k8s nodes, not something that's long term.
- jcastro 4y ago
  I've been running Flatcar Linux as my home server OS since it was CoreOS (which was later acquired by redhat and rebased onto ostree -- Flatcar kept the A/B partition setup so it was nice to be able to directly upgrade to it.) You set a maintenance window for the reboots and then basically you don't need to touch it. I keep all my docker-compose files in github and just have them all set to launch on boot. linuxserver.io has well maintained images for just about everything you'd want to run at home. Both Fedora CoreOS (ostree based) and openSUSE's MicroOS are the same way, they give you a kernel, systemd, and a container runtime and then the workload is decoupled into containers, it's a nice setup.
  - throwaway894345 4y ago
    I've been looking for something very similar to this for a while. Do you have a writeup for your setup? Also, how does Flatcar's immutability work? TFA mentioned that MicroOS is based on btrfs snapshots, but it didn't give any indication about how Flatcar worked (nor did the Flatcar GitHub readme). Do you know of any comparisons between Flatcar, CoreOS, and MicroOS? Also, any idea which of these work with Raspberry Pis?
    - jcastro 4y ago
      Flatcar uses 2 partitions, A and B, you boot into one and then updates update the one that you're not booted into, when you reboot it it boots into the updated one. It's like Android: https://source.android.com/docs/core/ota/ab https://source.android.com/docs/core/ota/ab I maintain an awesome-list of immutable resources with a collection of talks and presentations from the people making the stuff. They do a better job explaining it than I can in an hn comment: https://github.com/castrojo/awesome-immutable https://github.com/castrojo/awesome-immutable However the list is currently focused on desktop stuff since this is a fairly common pattern in cloud already, I should probably write it up. Semi-related, a few of us have started a community around composable OCI fedora images, and one of our images is intended to be used as a home server built on CoreOS with ZFS, cockpit, and all the goodies you'd need. It's still fresh and we're looking for help if anyone's interested: https://github.com/ublue-os/ucore https://github.com/ublue-os/ucore (Disclaimer: I helped start this project)
xena 4y ago
Surprised they didn't mention NixOS.
sylware 4y ago
Elephant in the room: computer language syntaxes/compilers, and open source system software are plagued with planned obsolescence.
- palata 4y ago
  I would say "premature obsolescence" (risking to sound pedantic). To me, "planned obsolescence" says "we engineer the product such that it does not last". Premature obsolescence says "our product is not good enough to last longer". The tendency is to build cheap crap, so of course it doesn't last. Doesn't mean that the engineers spent resources making sure it would not last.
  - sylware 4y ago
    It is not malicious all the time: It could be a sincere lack of perspective or brain-washing from the real toxic and malicious ppl. Whatever the reasons, it should be fought against. The real scammers use the word "innovation" to hide their planned obsolescence. And we reach a point where the real innovation is often removing and simplifying instead of adding/complexifying.
1MachineElf 4y ago
Linux distros designed to run "Live" (formerly off a a CD, but nowadays a USB or) are approximately immutable. A couple good examples are Tails OS and the venerable TinyCore Linux distribution. I also agree with many other commenters about inclusion of NixOS and GuixSD.
0xbadcafebee 4y ago
I like immutability, but I don't find it super useful in an OS. For file updates, sure, I like the idea of system-level snapshots. But generally speaking it's the entire system state that needs immutability, not just one application. In terms of server workloads, I run immutable containers, so doesn't matter what the base OS is. But I want the OS to be something very popular and stable with paid support.
indigodaddy 4y ago
I think Barry K’s new(ish) EasyOS would be considerate immutable https://easyos.org/about/how-and-why-easyos-is-different.html https://easyos.org/about/how-and-why-easyos-is-different.htm...
EntrePrescott 4y ago
Immutability is an important step in the right direction. What I'd like to see come up is a distribution that in addition to immutability provides the following: 1. separate install prefix directory for each currently installed package version (except for a minimal immutable core), thus allowing the coexistance of difference versions … and making access control policies practicable without the mess they are on a filesystem where there is a wild and hard to separate mix of stuff from lots of packages and other origins. 2. a modern and secure binding/mapping mechanism between the package install prefixes and what each of those and the user are supposed to see (dependig on profiles that can vary not only between packages but also e.g. between different users, different use cases etc). The "modern and secure" part meaning: not an old symlink based thing but something that uses the modern isolation techniques, cgroup/namespace, bind mounts, and the other mechanisms the kernel provides today. 3. a nice frontend for making the administration of those package-versions/mappings and the setting of access control policies based on them easy, with a sensible set of distribution curated profiles for the mappings, isolation settings and access control policies. Parts of that already exist, e.g: 1. (the per-package version install prefix) has been spearheaded by Nix/NixOS … (but it's missing 2. and 3.) 2. the secure isolation, cgroups, namespaces, bind mounts etc part of 2. has been spearheaded by Containers… but their simple layering structure doesn't allow to map the more complex depndency graph of a package management system.
- waynesonfire 4y ago
  I really like these ideas. The OS, the interface that I would like to work with, that I have worked with my entire career, the interface that I'm an expert in, begins to recover some territory from the abomination represented by Kubernetes and Docker.
  - DJBunnies 4y ago
    Hey, some people prefer cattle to pets.
- 0x69420 4y ago
  nixos missing 2/3 is what's behind my repeated insistence that nix is in sore need of a plan 9-esque host system. perhaps with fuchsia there's hope yet. cgroups and linux namespaces are great and all but pervasive namespace abstraction really really needs a fluid, low-friction interface for constructing those namespaces and subsequently examining precisely how they're put together. this is why typing ns in plan 9 is a huge eye-opener for a lot of newcomers. suid and friends ensure a perpetual lower bound on red tape and drudgery where traditional unix-likes implement namespaces.
thinkmassive 4y ago
EndlessOS (started in 2017) is a distro based on debian that uses OSTree for an immutable root partition. It’s targeted at non-technical users and particularly suitable for not-always-online environments. https://endlessos.com/ https://endlessos.com/
- thinkmassive 4y ago
  I should have mentioned their approach to user apps too… Flatpak is the preferred app packaging format, and Endless maintains its own collection of curated apps in the App Center. I think it’s straightforward to install anything from Flathub as well, but it’s been a while since I last checked.
tpoacher 4y ago
Can someone explain 'immutability' (in this context) more clearly? The article doesn't really explain anything. The filesystem is read-only (from the perspective of the user) in all linux distributions. How is this different?
Dwedit 4y ago
How did the list leave out MX Linux Frugal Install?
anotherhue 4y ago
Here's a snippet of my nixos config that enables immutability . It mounts root on a tmpfs which is erased on reboot. tmpfs is kind of a ramfs but will page to swap in most cases. Add additional bind mounts as necessary. https://en.wikipedia.org/wiki/Tmpfs https://en.wikipedia.org/wiki/Tmpfs fileSystems = { "/" = { device = "none"; fsType = "tmpfs"; options = [ "defaults" "size=8G" "mode=755" ]; }; "/boot" = { device = v.bootDev; fsType = "vfat"; }; "/nix" = { device = "tank/nix"; fsType = "zfs"; }; "/persist" = { device = "tank/persist"; fsType = "zfs"; options = [ "zfsutil" ]; }; "/home" = { device = "tank/home"; fsType = "zfs"; options = [ "zfsutil" ]; }; "/cache" = { device = "tank/cache"; fsType = "zfs"; options = [ "zfsutil" ]; }; "/etc/nixos" = { device = "/persist/nixos"; fsType = "none"; options = [ "bind" ]; }; "/var/lib/bluetooth" = { device = "/persist/bluetooth"; fsType = "none"; options = [ "bind" ]; };
amelius 4y ago
Isn't this what the embedded folks always (should) use?
1827162 4y ago
I've been doing this since ~2008 or so using Debian Linux. My custom setup is also capable of booting other computers on the network over PXE. It uses AUFS and tmpfs - yes, it needs upgrading to a more modern union filesystem. The neat thing about it is that you can turn off the power to the computer and any changes you make are completely gone (depending on the user account). Thus whatever I do on my computer leaves absolutely no trace whatsoever. I use it to write my temporary personal diaries/notebooks, and I can write whatever I want, and once the power's cut, it's gone, because it was all in RAM only. A very good adaptation for living in an authoritarian society (a "police state") with draconian laws, where they are actively trying to clamp down on dissent, including suppression of certain political views. Down with Big Brother and the Thought Police!!!
bfrog 4y ago
Immutable? No I want nixos with better docs.
MuffinFlavored 4y ago
I don't really understand what immutable means here. If you install Debian and then never touch anything other than your /home folder (therefore never touching the bootloader, any configuration files in /etc, not running apt for any changes/upgrades, leaving the kernel alone, etc.), isn't that "immutable"?

170 points by unripe_syntax 4y ago

Rather than use a dedicated OS for something like this, using an SELinux (or similar) policy makes more sense and grants more flexibility.

maksimur 4y ago
Could you elaborate?
- Zurrrrr 4y ago
  Something like SELinux would let you make any distro just as immutable as any of those distros in the list, to a finer level of granularity, and without having to use a specialized distro.
  - TheDong 4y ago
    Sure, and on any distro you can "sudo mount -o remount,ro /" and also have a useless distro, just like using selinux to make the whole thing immutable. The difference is you won't have to spent 2 years learning a custom policy language.
    - Zurrrrr 4y ago
      > also have a useless distro, just like using selinux to make the whole thing immutable. Absolute nonsense. > The difference is you won't have to spent 2 years learning a custom policy language. If you had bothered to spend maybe 2 months bothering to learn this tech that's been around for almost 20 years, you wouldn't have to use specialty distros to accomplish what your distro can already do natively. I've never been a fan of willful ignorance and fearmongering, and I see a lot of that when it comes to SELinux.
      - toyg 4y ago
        If it's been around 20 years and most people still loathe it, it probably isn't nice to work with.
        Zurrrrr 4y ago
        It could certainly be a lot nicer and more straightforward (and there are competing solutions that are), but people vastly exaggerate the difficulty of learning it, and dismiss the advantages that come from having it properly configured.
      - TheDong 4y ago
        If your goal is a usable immutable distro, selinux is the wrong tool. It just doesn't operate at the layer of creating a linux distro, or declaring what containers are running, or declaring a list of flatpak apps, or whatever. My point, by comparing selinux to "mount" and calling them equally useless, was not that they are not useful tools in their niches. My point was that they are useless if your goal is to build a usable immutable distro. SELinux is a hammer that cannot be used to turn a default debian installation into a usable immutable linux distro. The initial claim of "using SELinux rather than a dedicated distro can work" was nonsense, so of course the thread of replies off this is nonsense. GIGO as they say.
        Zurrrrr 4y ago
        > SELinux is a hammer that cannot be used to turn a default debian installation into a usable immutable linux distro. It's only a hammer if you use it as one. It's a very fine grained tool and it can be as precise as you want it to be. > The initial claim of "using SELinux rather than a dedicated distro can work" was nonsense It wasn't at all, and it's baffling to me why anyone would claim so.
  - t-3 4y ago
    People that want an entirely containerized distro probably don't want to maintain a bunch of SELinux rules or deal with the inevitable breakages that occur due to old/poorly designed software that is needed for one reason or another. It's entirely opposite of the goal of making everything a drop-in no-maintenance package.
  - tremon 4y ago
    SELinux isn't about immutability, it's about program confinement. Running each daemon in its own fs/user/net namespace comes a lot closer to mimicking the value of SELinux than making the OS immutable.
    - Zurrrrr 4y ago
      > SELinux isn't about immutability, it's about program confinement. The mechanisms that allow for program confinement also allow you to construct an immutable OS. > Running each daemon in its own fs/user/net namespace comes a lot closer to mimicking the value of SELinux than making the OS immutable. Maybe, but it's still pretty far off. SELinux is about removing DAC/an all powerful user much more than it is about sandboxing.
  - throwaway894345 4y ago
    How? How do you get immutability from SELinux policies (specifically without crafting detailed policies for everything running in the entire userland--at which point you're just begging for a distro, right?)? Honestly, if you really want something immutable and distro-agnostic, you probably want something like btrfs snapshots; however, converting the root filesystem to btrfs is probably a pain, and there will be some initial configuration to prepare some cloud-init, ignition, etc before you take your initial snapshot and you'll also probably want to configure some "boot from snapshot" type functionality, at which point it would be pretty nice to have all of this packaged up in a distro so each user doesn't have to figure all of that out every time.
FireInsight 4y ago
Well, with a dedicated Linux distro, you get it ootb and it's supported by people other than you. Also with Silverblue, there's a mechanism for getting the base OS from a OCI container image (check out https://ublue.it/ https://ublue.it/)
- Zurrrrr 4y ago
  > Well, with a dedicated Linux distro, you get it ootb and it's supported by people other than you. For sure, I can see there are use cases and a market for it.

fsflover 4y ago

Qubes OS should have been mentioned. It's not a Linux distro and it's not immutable, but it allows to create disposable virtual machines from customized templates. For me, it is much easier to manage.

FireInsight 4y ago
> It's not a linux distro and it's not immutable. Then it wouldn't fit with the lists theme?
- fsflover 4y ago
  It depends on the reason why you need it. For me, it's security, and Qubes allows to do it more conveniently.
- palata 4y ago
  Well, there are distros in the list that are optimized for containers. If you use Linux distros in Qubes OS, then I would say that it counts as a Linux distro (by default you get Fedora). And by default you run in VMs, where some parts are persistent (e.g. your home) and some are not. Just like running a container with some persistent storage. It depends on your use-case whether you like it or not, but I would definitely put Qubes OS in the list.
maksimur 4y ago
Qubes OS is a hassle to set up and it's too heavy in my experience. Also I couldn't suspend to RAM or hibernate because my hardware wasn't fully supported.
- fsflover 4y ago
  It depends on the hardware, see https://forum.qubes-os.org/t/community-recommended-computers/5560 https://forum.qubes-os.org/t/community-recommended-computers.... On my laptop suspend to RAM works flawlessly. Hibernation is not implemented in Qubes yet.

zerof1l 4y ago

I see this being very useful for containerized use cases. Usually, you have a Docker file that installs and sets everything once at container creation time. After that, you don't want anything to change in the OS. Great for security.

eru 4y ago
Yes. If you want to change anything, you rebuild from source. Exactly what we are doing when we are eg working on a C program: we don't monkey patch the binary, we run the build system again.
throwaway894345 4y ago
I agree, but at least in many cases, containers aren't the immutability mechanism. E.g., MicroOS seems to rely on btrfs snapshots--all changes are dropped on reboot and the system starts from a fresh factory state. Your application could be containerized or it could be a package pulled from the distro repository and managed by systemd.

plaguepilled 4y ago

No NixOS? No Guix? C'mon those are the best ones!

rgoulter 4y ago
Right. NixOS' absence from the list is notable. c.f. Google trends for "nixos, fedora silverblue, guix" https://trends.google.com/trends/explore?q=nixos,fedora%20silverblue,guix&hl=en https://trends.google.com/trends/explore?q=nixos,fedora%20si... which suggests NixOS is relatively more searched for than silverblue.
- eloisant 4y ago
  I guess you can use NixOS for your desktop but that's not the main focus of this distribution. It's more to create images to deploy server-side applications.
  - rgoulter 4y ago
    c.f. Nix survey results from last year, which indicates many of the respondents use NixOS as their daily Linux distribution. (e.g. number of responses for just "uses gnome" and "uses plasma" are about half of the number of responses for "use NixOS daily"). https://discourse.nixos.org/t/2022-nix-survey-results/18983 https://discourse.nixos.org/t/2022-nix-survey-results/18983 Many of those who do use NixOS as a desktop Linux are enthusiastically happy about it. Indeed, this user described the combination of nice features & rough edges as "cursed". (It's so good you won't want to use anything else, but it's too rough to recommend generally). https://blog.wesleyac.com/posts/the-curse-of-nixos https://blog.wesleyac.com/posts/the-curse-of-nixos
  - plaguepilled 4y ago
    This comment threw me off hard because of how loaded it is. 1. How does that address the fact that NixOS is immutable? 2. If discussing server side applications was the intention of the author, why didn't they make that the explicit focus of the article? As the article currently stands, that's not the stated topic, but it is a point included in the article about why immutable OSes are cool. That's a noticeably different reality. 3. Even if we look past the above, NixOS can be used for server side deployment and in many cases is favourable to, say, Ansible deployments, so that STILL doesn't explain why it was left off!

hendry 4y ago

My OS for Web kiosks was immutable since 2007. It's what people wanted and needed.

heisenzombie 4y ago

It’s much less plug-and-play than the options in the article… but I’ve had fun with a related idea, which is running Nixos on storage where root is erased on every boot. My home server is set up this way. At the time, I more or less followed this guide, which explains things nicely: https://grahamc.com/blog/erase-your-darlings/ https://grahamc.com/blog/erase-your-darlings/ Been erasing root for almost 2 years I think now. Works great!

LanternLight83 4y ago
I've cloned this setup for BTRFS on Guix, which took some questionable hacks because they don't have all the hooks I needed throughout the boot process... still meaning to migrate it to ZFS on Root, which will entail yet more hacks. Going for it though c:
- tripleo1 4y ago
  > I've cloned this setup for BTRFS on Guix Please share :)
ochoseis 4y ago
One of the hurdles with this approach is that _everything_ has to live in configuration.nix. How are you handling secrets like your host keys? I recently got sops-nix working as a prototype, but curious if there are better ways. Side note: everything living in configuration.nix makes flake-ifying your system a lot easier, which is a win in my book.
- anotherhue 4y ago
  There's a nix way to read a secret from a file. I'm not sure if it still ends up in the nix store that way but at least it's not in the config file so your VCS is clean.
  - piperswe 4y ago
    Typically with NixOS, you'll have systemd or a wrapper script read the secret from a file into an environment variable for the service that needs the secret. This secret file would be stored in `/var/lib` or similar, outside the Nix store. There isn't really a "Nix" way, just a pattern used in various places in NixOS.

charcircuit 4y ago

This is missing the most poplar immutable Linux distribution. Android.

cleanchit 4y ago
Yeah it's immutable but the build process is a pain in the ass. For nixos, the input is a couple text files and the process is a single command.
- juliosueiras 4y ago
  why not both ;) https://github.com/danielfullmer/robotnix https://github.com/danielfullmer/robotnix
otabdeveloper4 4y ago
Android is not a Linux distribution. Whatever runs in your phone is only slightly based on stock kernel.org tarballs.
- zamnos 4y ago
  But they're still based on that kernel that's hosted at that Linux website? Do you consider ChromeOS a Linux distro? it's based on Gentoo.
  - palata 4y ago
    I don't know about ChromeOS, but I would definitely consider that Android is not a Linux distribution. To me, a Linux distribution is an OS where, given that I am very comfortable with another Linux distribution, I will quickly be able to find my way around. Does Android have services? As root, can I start/stop them? Can I install libraries somewhere and use them somehow? How do I package an app (other than by running "build" in Android-Studio and letting the magic happen)? Could I install my own window manager? => I don't know any of that, so it's not a Linux distro. As a comparison, SteamOS is based on Arch Linux and has a particular setup (with a read-only "core" partition I guess?), but I am pretty sure that if you give me a shell, I will feel like I am in a Linux distro.

boris 4y ago

Most of these focus on running containers. What I would like is a distribution that focuses on the reliability of upgrades, both configuration and data. To give a concrete example, I have a Debian server running Apache (with auto-certificate renewal), PostgreSQL, and Postfix. When it's time to upgrade to the next release, it's basically the "it should work, but who really knows, depends on your specific setup you may have to tweak a few things" kind of guarantee. What I want is a rigorous guarantee like we have for relational database schema/data migration.

znpy 4y ago
I’ve recently switched to fedora and I’m having a great time in this sense.
ajdude 4y ago
You would probably get a lot out of NixOS. Sans that, I wonder how Fedora Silverblue would do as a server.
- itsmartapuntocm 4y ago
  Fedora IoT is essentially the server variant of Silverblue.
- bonzini 4y ago
  Fedora Silverblue is to workstation what Fedora CoreOS is to servers, so the underlying technology would fare very well.
evol262 4y ago
This _is_ Silverblue and other ostree-based repos. This article is focused on containers because when the root OS is immutable, then mutable developer workflows/etc necessarily happen in containers (distrobox, toolbox, whatever). OSTree is/was commonly described as "git for filesystems". Every update is an entirely new system image which applies a 3-way diff. It's possibly to state in a single command "show me the drift in my config files/data versus the ref I'm running" and/or "show me the diff between my running system and an update I may apply".

eloisant 4y ago

While it's not yet easy to install on anything else than a Steam Deck, SteamOS is also a great example of an immutable distribution.

nathants 4y ago

i want a pc game that i reboot into. an immutable linux iso and usb boot.

tmitchel2 4y ago

Is it not possible to have a client / desktop OS where each app runs in its own container by default with its own writeable filesystem... I would have thought that's beneficial from a security perspective as well as making things easier to separate...

ElectricalUnion 4y ago
> Is it not possible to have a client / desktop OS where each app runs in its own container by default with its own writeable filesystem... We have several "container as app" solutions around: * Microsoft Universal Windows Platform; * Canonical snaps; * Flatpak; * containertoolbx/Distrobox (this one is very DIY and what I use); It's just that (as of now) most apps rely (and are allowed to be deployed in "stores") on very leaky container isolation (like full filesystem access) so might as well not be deployed inside containers in the first place.
qbasic_forever 4y ago
Flatpaks do exactly that.
- palata 4y ago
  https://ludocode.com/blog/flatpak-is-not-the-future https://ludocode.com/blog/flatpak-is-not-the-future

jacooper 4y ago

Do any of the immutable server oses like coreOS and MicroOS make any sense for a bare-metal one-instance installation? It seems they are more optimized for temporary k8s nodes, not something that's long term.

jcastro 4y ago
I've been running Flatcar Linux as my home server OS since it was CoreOS (which was later acquired by redhat and rebased onto ostree -- Flatcar kept the A/B partition setup so it was nice to be able to directly upgrade to it.) You set a maintenance window for the reboots and then basically you don't need to touch it. I keep all my docker-compose files in github and just have them all set to launch on boot. linuxserver.io has well maintained images for just about everything you'd want to run at home. Both Fedora CoreOS (ostree based) and openSUSE's MicroOS are the same way, they give you a kernel, systemd, and a container runtime and then the workload is decoupled into containers, it's a nice setup.
- throwaway894345 4y ago
  I've been looking for something very similar to this for a while. Do you have a writeup for your setup? Also, how does Flatcar's immutability work? TFA mentioned that MicroOS is based on btrfs snapshots, but it didn't give any indication about how Flatcar worked (nor did the Flatcar GitHub readme). Do you know of any comparisons between Flatcar, CoreOS, and MicroOS? Also, any idea which of these work with Raspberry Pis?
  - jcastro 4y ago
    Flatcar uses 2 partitions, A and B, you boot into one and then updates update the one that you're not booted into, when you reboot it it boots into the updated one. It's like Android: https://source.android.com/docs/core/ota/ab https://source.android.com/docs/core/ota/ab I maintain an awesome-list of immutable resources with a collection of talks and presentations from the people making the stuff. They do a better job explaining it than I can in an hn comment: https://github.com/castrojo/awesome-immutable https://github.com/castrojo/awesome-immutable However the list is currently focused on desktop stuff since this is a fairly common pattern in cloud already, I should probably write it up. Semi-related, a few of us have started a community around composable OCI fedora images, and one of our images is intended to be used as a home server built on CoreOS with ZFS, cockpit, and all the goodies you'd need. It's still fresh and we're looking for help if anyone's interested: https://github.com/ublue-os/ucore https://github.com/ublue-os/ucore (Disclaimer: I helped start this project)

xena 4y ago

Surprised they didn't mention NixOS.

sylware 4y ago

Elephant in the room: computer language syntaxes/compilers, and open source system software are plagued with planned obsolescence.

palata 4y ago
I would say "premature obsolescence" (risking to sound pedantic). To me, "planned obsolescence" says "we engineer the product such that it does not last". Premature obsolescence says "our product is not good enough to last longer". The tendency is to build cheap crap, so of course it doesn't last. Doesn't mean that the engineers spent resources making sure it would not last.
- sylware 4y ago
  It is not malicious all the time: It could be a sincere lack of perspective or brain-washing from the real toxic and malicious ppl. Whatever the reasons, it should be fought against. The real scammers use the word "innovation" to hide their planned obsolescence. And we reach a point where the real innovation is often removing and simplifying instead of adding/complexifying.

1MachineElf 4y ago

Linux distros designed to run "Live" (formerly off a a CD, but nowadays a USB or) are approximately immutable. A couple good examples are Tails OS and the venerable TinyCore Linux distribution. I also agree with many other commenters about inclusion of NixOS and GuixSD.

0xbadcafebee 4y ago

I like immutability, but I don't find it super useful in an OS. For file updates, sure, I like the idea of system-level snapshots. But generally speaking it's the entire system state that needs immutability, not just one application. In terms of server workloads, I run immutable containers, so doesn't matter what the base OS is. But I want the OS to be something very popular and stable with paid support.

indigodaddy 4y ago

I think Barry K’s new(ish) EasyOS would be considerate immutable https://easyos.org/about/how-and-why-easyos-is-different.html https://easyos.org/about/how-and-why-easyos-is-different.htm...

EntrePrescott 4y ago

Immutability is an important step in the right direction. What I'd like to see come up is a distribution that in addition to immutability provides the following: 1. separate install prefix directory for each currently installed package version (except for a minimal immutable core), thus allowing the coexistance of difference versions … and making access control policies practicable without the mess they are on a filesystem where there is a wild and hard to separate mix of stuff from lots of packages and other origins. 2. a modern and secure binding/mapping mechanism between the package install prefixes and what each of those and the user are supposed to see (dependig on profiles that can vary not only between packages but also e.g. between different users, different use cases etc). The "modern and secure" part meaning: not an old symlink based thing but something that uses the modern isolation techniques, cgroup/namespace, bind mounts, and the other mechanisms the kernel provides today. 3. a nice frontend for making the administration of those package-versions/mappings and the setting of access control policies based on them easy, with a sensible set of distribution curated profiles for the mappings, isolation settings and access control policies. Parts of that already exist, e.g: 1. (the per-package version install prefix) has been spearheaded by Nix/NixOS … (but it's missing 2. and 3.) 2. the secure isolation, cgroups, namespaces, bind mounts etc part of 2. has been spearheaded by Containers… but their simple layering structure doesn't allow to map the more complex depndency graph of a package management system.

waynesonfire 4y ago
I really like these ideas. The OS, the interface that I would like to work with, that I have worked with my entire career, the interface that I'm an expert in, begins to recover some territory from the abomination represented by Kubernetes and Docker.
- DJBunnies 4y ago
  Hey, some people prefer cattle to pets.
0x69420 4y ago
nixos missing 2/3 is what's behind my repeated insistence that nix is in sore need of a plan 9-esque host system. perhaps with fuchsia there's hope yet. cgroups and linux namespaces are great and all but pervasive namespace abstraction really really needs a fluid, low-friction interface for constructing those namespaces and subsequently examining precisely how they're put together. this is why typing ns in plan 9 is a huge eye-opener for a lot of newcomers. suid and friends ensure a perpetual lower bound on red tape and drudgery where traditional unix-likes implement namespaces.

thinkmassive 4y ago

EndlessOS (started in 2017) is a distro based on debian that uses OSTree for an immutable root partition. It’s targeted at non-technical users and particularly suitable for not-always-online environments. https://endlessos.com/ https://endlessos.com/

thinkmassive 4y ago
I should have mentioned their approach to user apps too… Flatpak is the preferred app packaging format, and Endless maintains its own collection of curated apps in the App Center. I think it’s straightforward to install anything from Flathub as well, but it’s been a while since I last checked.

tpoacher 4y ago

Can someone explain 'immutability' (in this context) more clearly? The article doesn't really explain anything. The filesystem is read-only (from the perspective of the user) in all linux distributions. How is this different?

Dwedit 4y ago

How did the list leave out MX Linux Frugal Install?

anotherhue 4y ago

Here's a snippet of my nixos config that enables immutability . It mounts root on a tmpfs which is erased on reboot. tmpfs is kind of a ramfs but will page to swap in most cases. Add additional bind mounts as necessary. https://en.wikipedia.org/wiki/Tmpfs https://en.wikipedia.org/wiki/Tmpfs fileSystems = { "/" = { device = "none"; fsType = "tmpfs"; options = [ "defaults" "size=8G" "mode=755" ]; }; "/boot" = { device = v.bootDev; fsType = "vfat"; }; "/nix" = { device = "tank/nix"; fsType = "zfs"; }; "/persist" = { device = "tank/persist"; fsType = "zfs"; options = [ "zfsutil" ]; }; "/home" = { device = "tank/home"; fsType = "zfs"; options = [ "zfsutil" ]; }; "/cache" = { device = "tank/cache"; fsType = "zfs"; options = [ "zfsutil" ]; }; "/etc/nixos" = { device = "/persist/nixos"; fsType = "none"; options = [ "bind" ]; }; "/var/lib/bluetooth" = { device = "/persist/bluetooth"; fsType = "none"; options = [ "bind" ]; };

amelius 4y ago

Isn't this what the embedded folks always (should) use?

1827162 4y ago

I've been doing this since ~2008 or so using Debian Linux. My custom setup is also capable of booting other computers on the network over PXE. It uses AUFS and tmpfs - yes, it needs upgrading to a more modern union filesystem. The neat thing about it is that you can turn off the power to the computer and any changes you make are completely gone (depending on the user account). Thus whatever I do on my computer leaves absolutely no trace whatsoever. I use it to write my temporary personal diaries/notebooks, and I can write whatever I want, and once the power's cut, it's gone, because it was all in RAM only. A very good adaptation for living in an authoritarian society (a "police state") with draconian laws, where they are actively trying to clamp down on dissent, including suppression of certain political views. Down with Big Brother and the Thought Police!!!

bfrog 4y ago

Immutable? No I want nixos with better docs.

MuffinFlavored 4y ago

I don't really understand what immutable means here. If you install Debian and then never touch anything other than your /home folder (therefore never touching the bootloader, any configuration files in /etc, not running apt for any changes/upgrades, leaving the kernel alone, etc.), isn't that "immutable"?