6 ms·

Erase your darlings: immutable infrastructure for mutable systems

77 points by ducktective 5y ago

trepanne 5y ago
I followed the tenets laid out in this article to provision a home router on NixOS... only difference is I used BTRFS snapshots instead of ZFS (which would have been pretty heavyweight for a packet filtering appliance). It's a pretty great system for such an application. All the details of all the specialized, rarely-touched, hard-to-remember moving parts (nftables syntax, dual-stack DNS resolution, RS-232 serial connection parameters, etc. etc.) are all neatly collected under /etc/nixos for future me to puzzle out... and under version control, backed up offsite. It would be pretty easy for me to swap out failed hardware or upgrade it. I wouldn't mind getting more infrastructure set up on these lines, and then maybe figure out a good setup for NixOps. It wasn't too bad learning a little Nix to keep the configs DRY, modularized and parametrized. I find the results clean & readable, even though I'm hardly a grade-A FP propellerhead. My main complaint was that nftables rules need to be expressed as dead strings instead of proper objects in the Nix language, which limits their composability. This would be a nice thing for the wish list.
- dom_hutton 5y ago
  I'm sure nixpkgs will accept your PR (: truly though I've found the community pretty great and if a pkg is already in place it's not a quantum leap to add additional options to it.
goodpoint 5y ago
No, thanks. Any person with experience on large, critical infrastructure knows the importance of being able to make changes locally to investigate an incident.
- aequitas 5y ago
  It's better to think of these kind of systems as Disposable infrastructure instead of Immutable infrastructure. The difference being is that you occasionally, temporary can accrue some state on a machine (eg: debugging, hotfix), but you are accustomed to the idea that you can indiscriminately dispose of any system without worrying about state present on the system. So after you're done debugging, you just throw away the system and have a new replacement build automatically (by autoscaling or similar systems).
  - hinkley 5y ago
    Your theory is that the third thing you tried and only the third thing you tried fixed the problem. To test that theory you have to recreate it, which you do by modifying the baseline. You're gonna find out real fast if #2 was also required to fix the problem. Or you can find out in four months when the servers need a fix for an advisory.
  - goodpoint 5y ago
    Yes. Even better if something tracks and shows you the changes. Etckeeper does that for /etc using git. It can even push/pull changes. And something else can update or rollback software deployments. Apt, yum, zypper do that just fine.
- bo0tzz 5y ago
  I don't think there's anything that stops you from doing that? You can edit the running system the usual way AFAIK. Even better, afterwards you can just reset the box and all your changes are automatically cleaned up.
  - joepie91_ 5y ago
    Not to mention that since systems are deterministically built, you can usually reliably reproduce the failure mode on a separate system and investigate it there. "Hey, what's the store hash of the thing that's failing, so I can grab it and reproduce it locally" is a thing you can just do with Nix.
- unicornmama 5y ago
  Managing NixOS system is an organizational pain in the ass. Virtually everyone knows Ubuntu. Virtually all vendors provide some sort of standard supported deployment like Docker or Ansible. Now anyone who’s affected by a NixOS deployment either has to spend months learning Nix or escalate to the a handful of gurus.
  - whateveracct 5y ago
    It's not hard to become a NixOS guru if you work somewhere that uses it..only thing blocking most people in my experience is a stick up their butt.
hinkley 5y ago
I don't do a whole lot in this space, but when I do I create base docker images to handle cross-cutting concerns like this. I end up with a little stack of base images depending on which ones need some large or complex install (python, nginx, node, etc), and occasionally you end up with a couple vestigial things to avoid the worst combinatorics, but it mostly seems to work. You have to get past runbooks to actual scripts to do this, but that's not a bad goal to have. Change history in wikis leaves a lot to be desired, and when you're trying to go from beginner to mastery, the change history can help a lot.
raffraffraff 5y ago
Isn't a lot of this wisdom encapsulated in the (old, uncool) configuration management stuff like chef and puppet? The main difference being that you still need to rebuild your systems regularly to keep yourself honest (no lazy one-off changes, everything goes into the CM codebase). I mean, I get that NixOS can pin versions of everything, and we've all been bitten by a new server build on identical CM code failing because of an upstream version change. But it's an eternal problem: pinning all the versions means that you a now micromanaging a zillion different versions of shit that you don't want to care about.
- PterodactylsRUs 5y ago
  The problem with Chef and Puppet is that you tell it what to _do_ to a machine, but what you care about is what the machine _is_ or _has_. If you install a package on a machine using Chef, and then delete the line to install the package, it won't go ahead and uninstall the package. Even if you want to keep all your machines up-to-date with your configuration, and tell Chef to do an uninstall, uninstalls don't necessarily restore the state of the machine to what it was before the install. Data frequently gets left behind.
  - snicker7 5y ago
    It’s worse than that. “Installing a package” is not meaningful without a reference to its version. Nix/Guix are supposed to give you byte-indentical systems.
    - raffraffraff 5y ago
      CM systems don't just install any old package. You can absolutely specify a version.
      - bfrog 5y ago
        Only what has been packaged, nix let’s you roll your own package if you like, letting you overlay ontop of existing expressions to produce your own. Want to rebuild a specific OpenSSL with a specific tool chain and compile time flags? Easily expressed in nix if you don’t mind the time to build it all
joshuak 5y ago
I'm unclear on the benefit of NixOS vs container optimized os (aka coreos/flatcar). These systems have read only root partitions, no package manager, minimal services. The package manager is simply docker.
- scns 5y ago
  In case of Fedora coreOS and openSUSE microOS you can you use an ignition file to declare your complete OS before the installation. From partions over packages to file contents.
- fragileone 5y ago
  See spectrum-os for a combo of both
zerotolerance 5y ago
Disposable != immutable
itomato 5y ago
I’ve always done this with overlay filesystems and repackaged configs. SystemImager and rsync modules did what Kickstart and CFEngine could not: take me from COTS to Custom in this lab or one in another country. Kiwi did a pretty good job for a bit, but locked you to SUSE. Nix seems like a runtime environment rather than an OS, but I haven’t spent the sweaty hours with it that I have the others. Ansible and Debian work equally well on a laptop VM, Cloud Provider or colo rack.
Diti 5y ago
How does one edit the configuration.nix file on such an installation? /etc/ has no nixos/ directory.

77 points by ducktective 5y ago

I followed the tenets laid out in this article to provision a home router on NixOS... only difference is I used BTRFS snapshots instead of ZFS (which would have been pretty heavyweight for a packet filtering appliance). It's a pretty great system for such an application. All the details of all the specialized, rarely-touched, hard-to-remember moving parts (nftables syntax, dual-stack DNS resolution, RS-232 serial connection parameters, etc. etc.) are all neatly collected under /etc/nixos for future me to puzzle out... and under version control, backed up offsite. It would be pretty easy for me to swap out failed hardware or upgrade it. I wouldn't mind getting more infrastructure set up on these lines, and then maybe figure out a good setup for NixOps. It wasn't too bad learning a little Nix to keep the configs DRY, modularized and parametrized. I find the results clean & readable, even though I'm hardly a grade-A FP propellerhead. My main complaint was that nftables rules need to be expressed as dead strings instead of proper objects in the Nix language, which limits their composability. This would be a nice thing for the wish list.

dom_hutton 5y ago
I'm sure nixpkgs will accept your PR (: truly though I've found the community pretty great and if a pkg is already in place it's not a quantum leap to add additional options to it.

goodpoint 5y ago

No, thanks. Any person with experience on large, critical infrastructure knows the importance of being able to make changes locally to investigate an incident.

aequitas 5y ago
It's better to think of these kind of systems as Disposable infrastructure instead of Immutable infrastructure. The difference being is that you occasionally, temporary can accrue some state on a machine (eg: debugging, hotfix), but you are accustomed to the idea that you can indiscriminately dispose of any system without worrying about state present on the system. So after you're done debugging, you just throw away the system and have a new replacement build automatically (by autoscaling or similar systems).
- hinkley 5y ago
  Your theory is that the third thing you tried and only the third thing you tried fixed the problem. To test that theory you have to recreate it, which you do by modifying the baseline. You're gonna find out real fast if #2 was also required to fix the problem. Or you can find out in four months when the servers need a fix for an advisory.
- goodpoint 5y ago
  Yes. Even better if something tracks and shows you the changes. Etckeeper does that for /etc using git. It can even push/pull changes. And something else can update or rollback software deployments. Apt, yum, zypper do that just fine.
bo0tzz 5y ago
I don't think there's anything that stops you from doing that? You can edit the running system the usual way AFAIK. Even better, afterwards you can just reset the box and all your changes are automatically cleaned up.
- joepie91_ 5y ago
  Not to mention that since systems are deterministically built, you can usually reliably reproduce the failure mode on a separate system and investigate it there. "Hey, what's the store hash of the thing that's failing, so I can grab it and reproduce it locally" is a thing you can just do with Nix.
unicornmama 5y ago
Managing NixOS system is an organizational pain in the ass. Virtually everyone knows Ubuntu. Virtually all vendors provide some sort of standard supported deployment like Docker or Ansible. Now anyone who’s affected by a NixOS deployment either has to spend months learning Nix or escalate to the a handful of gurus.
- whateveracct 5y ago
  It's not hard to become a NixOS guru if you work somewhere that uses it..only thing blocking most people in my experience is a stick up their butt.

hinkley 5y ago

I don't do a whole lot in this space, but when I do I create base docker images to handle cross-cutting concerns like this. I end up with a little stack of base images depending on which ones need some large or complex install (python, nginx, node, etc), and occasionally you end up with a couple vestigial things to avoid the worst combinatorics, but it mostly seems to work. You have to get past runbooks to actual scripts to do this, but that's not a bad goal to have. Change history in wikis leaves a lot to be desired, and when you're trying to go from beginner to mastery, the change history can help a lot.

raffraffraff 5y ago

Isn't a lot of this wisdom encapsulated in the (old, uncool) configuration management stuff like chef and puppet? The main difference being that you still need to rebuild your systems regularly to keep yourself honest (no lazy one-off changes, everything goes into the CM codebase). I mean, I get that NixOS can pin versions of everything, and we've all been bitten by a new server build on identical CM code failing because of an upstream version change. But it's an eternal problem: pinning all the versions means that you a now micromanaging a zillion different versions of shit that you don't want to care about.

PterodactylsRUs 5y ago
The problem with Chef and Puppet is that you tell it what to _do_ to a machine, but what you care about is what the machine _is_ or _has_. If you install a package on a machine using Chef, and then delete the line to install the package, it won't go ahead and uninstall the package. Even if you want to keep all your machines up-to-date with your configuration, and tell Chef to do an uninstall, uninstalls don't necessarily restore the state of the machine to what it was before the install. Data frequently gets left behind.
- snicker7 5y ago
  It’s worse than that. “Installing a package” is not meaningful without a reference to its version. Nix/Guix are supposed to give you byte-indentical systems.
  - raffraffraff 5y ago
    CM systems don't just install any old package. You can absolutely specify a version.
    - bfrog 5y ago
      Only what has been packaged, nix let’s you roll your own package if you like, letting you overlay ontop of existing expressions to produce your own. Want to rebuild a specific OpenSSL with a specific tool chain and compile time flags? Easily expressed in nix if you don’t mind the time to build it all

joshuak 5y ago

I'm unclear on the benefit of NixOS vs container optimized os (aka coreos/flatcar). These systems have read only root partitions, no package manager, minimal services. The package manager is simply docker.

scns 5y ago
In case of Fedora coreOS and openSUSE microOS you can you use an ignition file to declare your complete OS before the installation. From partions over packages to file contents.
fragileone 5y ago
See spectrum-os for a combo of both

zerotolerance 5y ago

Disposable != immutable

itomato 5y ago

I’ve always done this with overlay filesystems and repackaged configs. SystemImager and rsync modules did what Kickstart and CFEngine could not: take me from COTS to Custom in this lab or one in another country. Kiwi did a pretty good job for a bit, but locked you to SUSE. Nix seems like a runtime environment rather than an OS, but I haven’t spent the sweaty hours with it that I have the others. Ansible and Debian work equally well on a laptop VM, Cloud Provider or colo rack.

Diti 5y ago

How does one edit the configuration.nix file on such an installation? /etc/ has no nixos/ directory.