7 ms·

Impermanence

119 points by anotherhue 3y ago

jbott 3y ago
This is linked in the footer of this page, but Graham Christensen has an excellent blog post “Erase your darlings” [1] that explains why you might want to do this. There is a script floating around on github [2] that does the install automatically, but I needs to be modified slightly to work on newer nixos versions. I have a forked version [3] that I used this morning, but my decisions might not make sense for everyone, but it’s provided as-is for now :) 1. https://grahamc.com/blog/erase-your-darlings/ https://grahamc.com/blog/erase-your-darlings/ 2. https://gist.github.com/mx00s/ea2462a3fe6fdaa65692fe7ee824de3e https://gist.github.com/mx00s/ea2462a3fe6fdaa65692fe7ee824de... 3. https://gist.github.com/jbott/531b9d555dae7f197f25326ef251f164/ae1478e90e7efc8c98f2eb070d02ef3d738a633d https://gist.github.com/jbott/531b9d555dae7f197f25326ef251f1...
- Tainnor 3y ago
  Sounds interesting in theory, but modern development setups are often full with tools, dependencies, IDE indices, intermediate build results, etc. pp. that all may take a very long time to download and build from scratch. Sure, you could try to keep track of all such things and persist them but it's going to be a lot of effort trying to figure out where all your tools are dumping their state.
  - n42 3y ago
    it's not so difficult, really. most of this is in your home folder, which is just a separate persistent mount. you can also use impermanence with home-manager if you want https://github.com/nix-community/impermanence#home-manager https://github.com/nix-community/impermanence#home-manager
    - Tainnor 3y ago
      Sure, but then you're throwing away the purported benefits (setup is always reproducible) for your entire home folder. It seems like there are two contradictory goals here, each with their own benefits. Impermanence gives you reproducibility, but permanence gives you performance. Maybe the right tradeoff is to just persist the entire home folder and nothing else (or few other things... I'm not sure I'd want to always download some programs that are quite huge), but the tradeoff is essentially still there.
      - n42 3y ago
        you're talking about three different things: NixOS, Home Manager, and Impermanence. or put another way: reproducible root filesystem, reproducible home folder, automatically reproducing things on boot. if you want a reproducible root filesystem, use NixOS. if you want to reproduce your root filesystem on boot, use NixOS with impermanence. if you want a reproducible home folder, use home manager. if you want to reproduce your home folder on boot, use home manager with impermanence. I only use impermanence on servers. I don't think there's any point using it on a desktop, or with Home Manager, other than street cred. it just complicates things. I don't really even recommend Home Manager for first time NixOS users for the same reason. edit: and I want to address this comment: > I'm not sure I'd want to always download some programs that are quite huge this isn't how it works; the binaries, as well as the entire initial state of the filesystem, are cached in the Nix cache, a read-only filesystem. you're not downloading everything on every boot.
  - yjftsjthsd-h 3y ago
    > often full with tools, dependencies, IDE indices, intermediate build results Tools and dependencies go in nix. Indexes and temporary build stuff is just cache, which you can still have (I'd lean towards regenerating per reboot, but YMMV). > but it's going to be a lot of effort trying to figure out where all your tools are dumping their state. Fair. Kind of an indictment of the current state of the ecosystem, but yes.
  - solatic 3y ago
    I don't think of impermanence as a tool for development setups, but rather a tool to improve production security. When a server gets compromised, it's common for an attacker to leverage their initial access to set up backdoor access for themselves, e.g. an additional privileged user or privileged service which phones home, so that they're no longer reliant on the original vulnerability to gain access again. This is important to ensure that they can launch a more damaging attack at a more opportune time (e.g. at the beginning of a long weekend). Now consider a stateful server which you need to host (e.g. Kubernetes control plane / etcd) where you ordinarily cannot practice immutable infrastructure due to the stateful nature of the server. Modules like impermanence allow you to guard against this kind of compromise by simply wiping out everything but the actual state as a result of rebooting. Any privileged users or malicious processes (which, of course, are not part of the system configuration used at boot) get wiped out at every reboot. It's not a silver bullet - an attacker could simply releverage the original vulnerability and set up access again - but doing the reboots frequently would force the vulnerability to be re-exploited each time, making it a pattern of access more likely to be detected in a SIEM.
  - Cloudef 3y ago
    With NixOS, you typically use flake.nix or shell.nix to "shell into" development environment so this is kinda non issue. You can also use `nix run` or `nix-env` to mix traditional package management and this. > but it's going to be a lot of effort trying to figure out where all your tools are dumping their state. This is true though, if you dont want to manage all of their configs from nix, I would make their dump locations, or $XDG_DIRS/$HOME (if you are lazy) permanent in this case.
- SkyMarshal 3y ago
  There’s an even better setup than Erase your Darlings. Instead of putting / on a zfs pool and erasing and rewriting it every reboot, just put tmpfs on / instead. With / all in memory it automatically gets wiped and recreated every reboot, without needing to actively erase a disk, and thus with much less drive wear (depending on how frequently you reboot). It’s also faster on some things when loading from RAM instead of the disk. And it’s overall a cleaner, simpler setup. https://elis.nu/blog/2020/05/nixos-tmpfs-as-root/ https://elis.nu/blog/2020/05/nixos-tmpfs-as-root/ You can also put tmpfs on home as well, using Impermanence and Home Manager to persist things like ~/.config and whatever other files or folders need to persist between reboots: https://elis.nu/blog/2020/06/nixos-tmpfs-as-home/ https://elis.nu/blog/2020/06/nixos-tmpfs-as-home/ tmpfs on / is enabled by NixOS’s unique design, in which it keeps the entire system in /nix/store and then softlinks all the paths into their appropriate place in /. With tmpfs on /, NixOS automatically recreates those softlinks in tmpfs on reboot. Very little setup effort is required to make this work.
  - jbott 3y ago
    I don’t think the point about disk wear is true. ZFS is a block-based CoW filesystem already, so the “erase” is actually something more like “make a new metadata entry that points to an earlier state snapshot”. Plus, I’d expect normal disk usage (ie. chrome disk cache) probably far outweighs any disk wear you’d get from the (relatively) small size of files you get written into /. Unless you’re working on a server with a ton of ram, I also think using tmpfs is more likely to shoot yourself in the foot with excess memory pressure. I don’t know of a way for the kernel to free memory if you write a huge file to the tmpfs partition by mistake, unless you use swap, and then you have the problems that come with that. Tmpfs might be faster though!
    - yjftsjthsd-h 3y ago
      > tmpfs is more likely to shoot yourself in the foot with excess memory pressure It defaults to 50% of physical RAM, but you can set the size option to whatever you want: https://www.kernel.org/doc/html/latest/filesystems/tmpfs.html https://www.kernel.org/doc/html/latest/filesystems/tmpfs.htm... (So yes, something to consider, but totally possible to mitigate)
sigmonsays 3y ago
it's really interesting the things that become possible with a system like nix in use. I don't think i'd ever use this but having a configuration persisted make for a interesting platform when reboot restores it to the original state.
analognoise 3y ago
I've been getting really into Nix and NixOs recently, and I love when stuff gets put here about it.
acka 3y ago
Regarding the example hardwareconfiguration.nix file: why would anyone want to mount a filesystem on a user's home directory as world writable? Maybe it shows my age, I know that most people don't run multi-user systems anymore (I do though). Even so, I would still be quite worried about some runaway process, perhaps running as user nobody for a reason, messing with my stuff.
- n42 3y ago
  I don't think this is common or acceptable; I think it is just a bad user contribution
- winternewt 3y ago
  I installed NixOS for the first time a couple of weeks ago. My biggest annoyance has been that most documentation and wiki contributions seem to assume that you're running as a single-user system where it's OK to sudo as root every time you want to change some user settings. For example, I have still not been able to run Sway as a window manager because I can't find how to set it up properly using Home Manager (doing the equivalent with an .xsession and i3 was no problem). The wiki article about it (https://nixos.wiki/wiki/Sway https://nixos.wiki/wiki/Sway) has a lot of configuration at the system level configuration.nix, but most of that is missing from the ostensibly equivalent Home Manager config. It seems like a design error that the Home Manager configuration options do not map 1:1 to the NixOS configuration options for users, and I'm guessing that error is caused by a lack of interest in properly configured multi-user systems.
  - e3bc54b2 3y ago
    For better or worse, home-manager is developed outside nixpkgs, where rest of NixOS lies. While I cannot speak to the reasons behind the schism, home-manager is better thought of as independent project that builds off of abstractions in nixpkgs, without taking into account what NixOS does.
  - drakerossman 3y ago
    You may want to read my blog post [1] on that one. It explains exactly how to install Sway with Home Manager. And no, it is not a design error to have Home Manager separated from nixpkgs, and unlike a child reply suggests, there's no "schism", and the reasons for separation are rather prosaic, as explained in another blog post of mine [2]. [1] https://drakerossman.com/blog/wayland-on-nixos-confusion-conquest-triumph https://drakerossman.com/blog/wayland-on-nixos-confusion-con... [2] https://drakerossman.com/blog/how-to-add-home-manager-to-nixos https://drakerossman.com/blog/how-to-add-home-manager-to-nix...
sph 3y ago
Even better than having / mounted as tmpfs, is having / mounted as read-only. So no misconfigured application can write and depend on files on tmpfs (which is stored in RAM), and breaking after reboot. On my Fedora Silverblue, only /etc and /var are mounted read-write. /home is a symlink to /var/home, /root is a symlink to /var/root.
fsflover 3y ago
This is the default behavior on Qubes OS: you run all software in VMs and VMs reset their /root partitions upon reboot, https://www.qubes-os.org/doc/templates https://www.qubes-os.org/doc/templates.
0x69420 3y ago
one of the nicest things about nixos is illustrated here, where a concrete few tens of lines of code implement a feature that some consider to define an entire genre of distro as a teenager i ran gentoo & passively marvelled in the back of my head that this self-described “meta-distribution” was flexible enough to serve as the basis of chromeos or whatever it's no coincidence that “gentoo but Good Actually” was one of the descriptions i heard that convinced me to give nixos a spin, paralleled further by the fact that i unironically ran gentoo prefix on macos in lieu of homebrew/macports for a while, and would later go on to run nix-darwin this particular instance of flexibility, though, feeds back nicely into one of the bullet points on the nixos elevator pitch (one little config directory is all you need to care about outside /var or /home). in a sense, running this setup is putting your money where your mouth is as far as that claim is concerned. and hey whoa would you look at that, it works. neat!
deleted 3y ago
[deleted]
autumn-antlers 3y ago
I've got a similar setup on GNU Guix, though I'm still refactoring the code (for this and a billion other features) and figuring out how my /home is going to work. I've gone ahead and copied the important parts out here[1] in case anyone is interested, but it's not great code, hasn't been tested, and isn't complete. Could be backed by BTRFS with a few tweaks (in which case the missing kernel bits wouldn't need worried about), and you could make root a TMPFS by just setting the `#:volatile-root?` flag of the `initramfs` procedure (in which case you wouldn't need rollback in the initramfs at all, just mounts for persistent data). 1: https://www.illucid.net/static/unpublished/erasing-darlings-on-guix.html https://www.illucid.net/static/unpublished/erasing-darlings-... edit: oh, lol, didn't escape the heredoc at all, import zfs / other guile modules, or put parted in the shebang; ah well that's what i mean by informational
Cu3PO42 3y ago
I'm now running impermanence on NixOS servers. Setting it up was slightly more work, obviously, but it's been... freeing in a sense. My confidence that I can rebuild all servers at any time and that all the peculiarities of my setup are documented have never been higher. Rather than using a script for setup, I am using disko [0] with nixos-anywhere [1] and a small home-grown script that integrates with 1Password for deployments and it's wonderful, because everything including disk formatting is now purely declarative. [0] https://github.com/nix-community/disko https://github.com/nix-community/disko [1] https://github.com/numtide/nixos-anywhere https://github.com/numtide/nixos-anywhere
1MachineElf 3y ago
One of my favorite NixOS features. Hope to see it more in GuixSD.

119 points by anotherhue 3y ago

This is linked in the footer of this page, but Graham Christensen has an excellent blog post “Erase your darlings” [1] that explains why you might want to do this. There is a script floating around on github [2] that does the install automatically, but I needs to be modified slightly to work on newer nixos versions. I have a forked version [3] that I used this morning, but my decisions might not make sense for everyone, but it’s provided as-is for now :) 1. https://grahamc.com/blog/erase-your-darlings/ https://grahamc.com/blog/erase-your-darlings/ 2. https://gist.github.com/mx00s/ea2462a3fe6fdaa65692fe7ee824de3e https://gist.github.com/mx00s/ea2462a3fe6fdaa65692fe7ee824de... 3. https://gist.github.com/jbott/531b9d555dae7f197f25326ef251f164/ae1478e90e7efc8c98f2eb070d02ef3d738a633d https://gist.github.com/jbott/531b9d555dae7f197f25326ef251f1...

Tainnor 3y ago
Sounds interesting in theory, but modern development setups are often full with tools, dependencies, IDE indices, intermediate build results, etc. pp. that all may take a very long time to download and build from scratch. Sure, you could try to keep track of all such things and persist them but it's going to be a lot of effort trying to figure out where all your tools are dumping their state.
- n42 3y ago
  it's not so difficult, really. most of this is in your home folder, which is just a separate persistent mount. you can also use impermanence with home-manager if you want https://github.com/nix-community/impermanence#home-manager https://github.com/nix-community/impermanence#home-manager
  - Tainnor 3y ago
    Sure, but then you're throwing away the purported benefits (setup is always reproducible) for your entire home folder. It seems like there are two contradictory goals here, each with their own benefits. Impermanence gives you reproducibility, but permanence gives you performance. Maybe the right tradeoff is to just persist the entire home folder and nothing else (or few other things... I'm not sure I'd want to always download some programs that are quite huge), but the tradeoff is essentially still there.
    - n42 3y ago
      you're talking about three different things: NixOS, Home Manager, and Impermanence. or put another way: reproducible root filesystem, reproducible home folder, automatically reproducing things on boot. if you want a reproducible root filesystem, use NixOS. if you want to reproduce your root filesystem on boot, use NixOS with impermanence. if you want a reproducible home folder, use home manager. if you want to reproduce your home folder on boot, use home manager with impermanence. I only use impermanence on servers. I don't think there's any point using it on a desktop, or with Home Manager, other than street cred. it just complicates things. I don't really even recommend Home Manager for first time NixOS users for the same reason. edit: and I want to address this comment: > I'm not sure I'd want to always download some programs that are quite huge this isn't how it works; the binaries, as well as the entire initial state of the filesystem, are cached in the Nix cache, a read-only filesystem. you're not downloading everything on every boot.
- yjftsjthsd-h 3y ago
  > often full with tools, dependencies, IDE indices, intermediate build results Tools and dependencies go in nix. Indexes and temporary build stuff is just cache, which you can still have (I'd lean towards regenerating per reboot, but YMMV). > but it's going to be a lot of effort trying to figure out where all your tools are dumping their state. Fair. Kind of an indictment of the current state of the ecosystem, but yes.
- solatic 3y ago
  I don't think of impermanence as a tool for development setups, but rather a tool to improve production security. When a server gets compromised, it's common for an attacker to leverage their initial access to set up backdoor access for themselves, e.g. an additional privileged user or privileged service which phones home, so that they're no longer reliant on the original vulnerability to gain access again. This is important to ensure that they can launch a more damaging attack at a more opportune time (e.g. at the beginning of a long weekend). Now consider a stateful server which you need to host (e.g. Kubernetes control plane / etcd) where you ordinarily cannot practice immutable infrastructure due to the stateful nature of the server. Modules like impermanence allow you to guard against this kind of compromise by simply wiping out everything but the actual state as a result of rebooting. Any privileged users or malicious processes (which, of course, are not part of the system configuration used at boot) get wiped out at every reboot. It's not a silver bullet - an attacker could simply releverage the original vulnerability and set up access again - but doing the reboots frequently would force the vulnerability to be re-exploited each time, making it a pattern of access more likely to be detected in a SIEM.
- Cloudef 3y ago
  With NixOS, you typically use flake.nix or shell.nix to "shell into" development environment so this is kinda non issue. You can also use `nix run` or `nix-env` to mix traditional package management and this. > but it's going to be a lot of effort trying to figure out where all your tools are dumping their state. This is true though, if you dont want to manage all of their configs from nix, I would make their dump locations, or $XDG_DIRS/$HOME (if you are lazy) permanent in this case.
SkyMarshal 3y ago
There’s an even better setup than Erase your Darlings. Instead of putting / on a zfs pool and erasing and rewriting it every reboot, just put tmpfs on / instead. With / all in memory it automatically gets wiped and recreated every reboot, without needing to actively erase a disk, and thus with much less drive wear (depending on how frequently you reboot). It’s also faster on some things when loading from RAM instead of the disk. And it’s overall a cleaner, simpler setup. https://elis.nu/blog/2020/05/nixos-tmpfs-as-root/ https://elis.nu/blog/2020/05/nixos-tmpfs-as-root/ You can also put tmpfs on home as well, using Impermanence and Home Manager to persist things like ~/.config and whatever other files or folders need to persist between reboots: https://elis.nu/blog/2020/06/nixos-tmpfs-as-home/ https://elis.nu/blog/2020/06/nixos-tmpfs-as-home/ tmpfs on / is enabled by NixOS’s unique design, in which it keeps the entire system in /nix/store and then softlinks all the paths into their appropriate place in /. With tmpfs on /, NixOS automatically recreates those softlinks in tmpfs on reboot. Very little setup effort is required to make this work.
- jbott 3y ago
  I don’t think the point about disk wear is true. ZFS is a block-based CoW filesystem already, so the “erase” is actually something more like “make a new metadata entry that points to an earlier state snapshot”. Plus, I’d expect normal disk usage (ie. chrome disk cache) probably far outweighs any disk wear you’d get from the (relatively) small size of files you get written into /. Unless you’re working on a server with a ton of ram, I also think using tmpfs is more likely to shoot yourself in the foot with excess memory pressure. I don’t know of a way for the kernel to free memory if you write a huge file to the tmpfs partition by mistake, unless you use swap, and then you have the problems that come with that. Tmpfs might be faster though!
  - yjftsjthsd-h 3y ago
    > tmpfs is more likely to shoot yourself in the foot with excess memory pressure It defaults to 50% of physical RAM, but you can set the size option to whatever you want: https://www.kernel.org/doc/html/latest/filesystems/tmpfs.html https://www.kernel.org/doc/html/latest/filesystems/tmpfs.htm... (So yes, something to consider, but totally possible to mitigate)

sigmonsays 3y ago

it's really interesting the things that become possible with a system like nix in use. I don't think i'd ever use this but having a configuration persisted make for a interesting platform when reboot restores it to the original state.

analognoise 3y ago

I've been getting really into Nix and NixOs recently, and I love when stuff gets put here about it.

acka 3y ago

Regarding the example hardwareconfiguration.nix file: why would anyone want to mount a filesystem on a user's home directory as world writable? Maybe it shows my age, I know that most people don't run multi-user systems anymore (I do though). Even so, I would still be quite worried about some runaway process, perhaps running as user nobody for a reason, messing with my stuff.

n42 3y ago
I don't think this is common or acceptable; I think it is just a bad user contribution
winternewt 3y ago
I installed NixOS for the first time a couple of weeks ago. My biggest annoyance has been that most documentation and wiki contributions seem to assume that you're running as a single-user system where it's OK to sudo as root every time you want to change some user settings. For example, I have still not been able to run Sway as a window manager because I can't find how to set it up properly using Home Manager (doing the equivalent with an .xsession and i3 was no problem). The wiki article about it (https://nixos.wiki/wiki/Sway https://nixos.wiki/wiki/Sway) has a lot of configuration at the system level configuration.nix, but most of that is missing from the ostensibly equivalent Home Manager config. It seems like a design error that the Home Manager configuration options do not map 1:1 to the NixOS configuration options for users, and I'm guessing that error is caused by a lack of interest in properly configured multi-user systems.
- e3bc54b2 3y ago
  For better or worse, home-manager is developed outside nixpkgs, where rest of NixOS lies. While I cannot speak to the reasons behind the schism, home-manager is better thought of as independent project that builds off of abstractions in nixpkgs, without taking into account what NixOS does.
- drakerossman 3y ago
  You may want to read my blog post [1] on that one. It explains exactly how to install Sway with Home Manager. And no, it is not a design error to have Home Manager separated from nixpkgs, and unlike a child reply suggests, there's no "schism", and the reasons for separation are rather prosaic, as explained in another blog post of mine [2]. [1] https://drakerossman.com/blog/wayland-on-nixos-confusion-conquest-triumph https://drakerossman.com/blog/wayland-on-nixos-confusion-con... [2] https://drakerossman.com/blog/how-to-add-home-manager-to-nixos https://drakerossman.com/blog/how-to-add-home-manager-to-nix...

sph 3y ago

Even better than having / mounted as tmpfs, is having / mounted as read-only. So no misconfigured application can write and depend on files on tmpfs (which is stored in RAM), and breaking after reboot. On my Fedora Silverblue, only /etc and /var are mounted read-write. /home is a symlink to /var/home, /root is a symlink to /var/root.

fsflover 3y ago

This is the default behavior on Qubes OS: you run all software in VMs and VMs reset their /root partitions upon reboot, https://www.qubes-os.org/doc/templates https://www.qubes-os.org/doc/templates.

0x69420 3y ago

one of the nicest things about nixos is illustrated here, where a concrete few tens of lines of code implement a feature that some consider to define an entire genre of distro as a teenager i ran gentoo & passively marvelled in the back of my head that this self-described “meta-distribution” was flexible enough to serve as the basis of chromeos or whatever it's no coincidence that “gentoo but Good Actually” was one of the descriptions i heard that convinced me to give nixos a spin, paralleled further by the fact that i unironically ran gentoo prefix on macos in lieu of homebrew/macports for a while, and would later go on to run nix-darwin this particular instance of flexibility, though, feeds back nicely into one of the bullet points on the nixos elevator pitch (one little config directory is all you need to care about outside /var or /home). in a sense, running this setup is putting your money where your mouth is as far as that claim is concerned. and hey whoa would you look at that, it works. neat!

deleted 3y ago

[deleted]

autumn-antlers 3y ago

I've got a similar setup on GNU Guix, though I'm still refactoring the code (for this and a billion other features) and figuring out how my /home is going to work. I've gone ahead and copied the important parts out here[1] in case anyone is interested, but it's not great code, hasn't been tested, and isn't complete. Could be backed by BTRFS with a few tweaks (in which case the missing kernel bits wouldn't need worried about), and you could make root a TMPFS by just setting the `#:volatile-root?` flag of the `initramfs` procedure (in which case you wouldn't need rollback in the initramfs at all, just mounts for persistent data). 1: https://www.illucid.net/static/unpublished/erasing-darlings-on-guix.html https://www.illucid.net/static/unpublished/erasing-darlings-... edit: oh, lol, didn't escape the heredoc at all, import zfs / other guile modules, or put parted in the shebang; ah well that's what i mean by informational

Cu3PO42 3y ago

I'm now running impermanence on NixOS servers. Setting it up was slightly more work, obviously, but it's been... freeing in a sense. My confidence that I can rebuild all servers at any time and that all the peculiarities of my setup are documented have never been higher. Rather than using a script for setup, I am using disko [0] with nixos-anywhere [1] and a small home-grown script that integrates with 1Password for deployments and it's wonderful, because everything including disk formatting is now purely declarative. [0] https://github.com/nix-community/disko https://github.com/nix-community/disko [1] https://github.com/numtide/nixos-anywhere https://github.com/numtide/nixos-anywhere

1MachineElf 3y ago

One of my favorite NixOS features. Hope to see it more in GuixSD.