5 ms·

Yeah I used to use pfSense but hated having a load of config and firewall rules created manually, it all felt quite brittle. Moved to NixOS almost 5 years ago

Yeah I used to use pfSense but hated having a load of config and firewall rules created manually, it all felt quite brittle. Moved to NixOS almost 5 years ago now and haven’t looked back, my router config is in Git and I’m able to make changes with confidence. Best part is making a change/installing an update and being able to rollback if I mess something up. I also use Nixus to push new configs which has a nifty feature where it will autorollback if you e.g make a config change that locks you out of SSH. I also moved from ESXI to NixOS for a virtualisation host more recently and that has been great as these machines are inherently long lived so you can’t apply the “cattle not pets” mindset here, but I feel like I can make changes and install updates with confidence. It also helped when I migrated to a Threadripper build where there were some strange quirks which needed Linux kernel patches, it’s nice being able to do this declaratively and to store everything in a versioned Git repo.

Asdrubalini 4y ago
Wow, didn’t know about Nixus, thanks!
colordrops 4y ago
Can you share your router configs? I'd love to have a NixOS based router.
- astro1 4y ago
  https://gitea.c3d2.de/zentralwerk/network https://gitea.c3d2.de/zentralwerk/network
- notemaker 4y ago
  https://skogsbrus.xyz/blog/2022/06/12/router https://skogsbrus.xyz/blog/2022/06/12/router
- ochoseis 4y ago
  One drawback I realized about NixOS as a router for home use is that if you need to share administration of the network, then it has to be easy for non-experts. In that case something like OpenWRT might be a better choice.
  - pmarreck 4y ago
    Well, at least the config file options are relatively straightforward, but you're right in that a GUI makes more sense for genpop
ochoseis 4y ago
How are you managing the VM’s on NixOS? A couple years ago I was searching for a good way to declaratively manage them, and the best I came up with was libvirt + terraform + nixos-generators for images. It feels like you should just be able to set them up as systemd units, but I couldn’t figure it out. microvm has caught my eye recently but I haven’t played with it yet: https://github.com/astro/microvm.nix https://github.com/astro/microvm.nix
- amarshall 4y ago
  Not OP (and I suspect they don’t have what you’re looking for since they have stated they treat the VMs as pets, not cattle). I currently use libvirt, but I eschew persistent VMs and instead build the libvirt domain XML with Nix, and then start VMs with `virsh create /path/to/myvm.xml`. Using an absolute path means that libvirt will not persist the domain, and makes it easier to declaratively manage the config with Nix. However, none of my VMs are always-on, so I don’t have to deal with restarts on config change. That said, when libvirt isn’t managing the lifecycle of the VMs as much, writing a systemd unit should be quite a bit easier.
- madjam002 4y ago
  The VMs themselves are a mix of some manually created ones (the VM configs are still NixOS though), and some IaC VMs which I just use Terraform and the libvirt provider. It's a bit clunky but I wouldn't want to embed the VM config in the hypervisor configuration as the VMs can be live migrated to other hypervisors. MicroVM looks awesome though, thanks for the pointer!
  - ochoseis 4y ago
    Did you manage to get live migration working with the terraform provider? As of a while ago it wasn’t implemented but idk if that’s changed: https://github.com/dmacvicar/terraform-provider-libvirt/issues/490 https://github.com/dmacvicar/terraform-provider-libvirt/issu...
1MachineElf 4y ago
I appreciate your comment here, and the parent comment. I've been using OPNSense[0] as my router for the past several months. So far, I am quite happy with it, but I've also thought that NixOS would be the next step. My OPNSense router has 30+ VLANs and many layered firewall rules (my take on "zEr0 tRuSt") and so the task of converting it all to NixOS seems a little daunting. I appreciate the utility of OPNSense's web GUI when configuring and troubleshooting my router config. It would be awesome if something like that could be integrated with NixOS. Additionally, something like nsh[1] to provide a traditional router/switch style CLI would be a dream come true. [0] https://opnsense.org/ https://opnsense.org/ [1] https://www.nmedia.net/nsh/ https://www.nmedia.net/nsh/
- hamandcheese 4y ago
  NixOS is great for a basic home router (I use it for my home router) but it’s networking config is still pretty rudimentary, and some things I would expect to work just don’t - i.e. port forwarding only works from outside your network, not inside. I haven’t done much with vlans yet so I can’t comment on that.
  - SuperSandro2000 4y ago
    I would highly recommend to use systemd-networkd based networking which should be able to do almost everything if configured correct.
    - hamandcheese 4y ago
      Is there a nix config wrapper for systemd-networkd? NixOS still does everything I need it too, but the parent comment sounded like they had a bit more complex of a setup.
      - janfrode 4y ago
        Not sure what you mean by config wrapper, but yes, systems-networkd is quite nice. Here's my home router setup with vlan1 for uplink to ISP, a bunch of other vlans for internal networks, and IPv6 prefix delegation to internal networks, and of course fireguard. All configured using systemd-networkd: { pkgs, lib, ... }:{ networking = { useNetworkd = true; useDHCP = false; enableIPv6 = true; }; networking.wireguard.interfaces = { wireguard = { ips = [ "172.20.60.1/24" ]; listenPort = 61891; privateKeyFile = "/etc/nixos/secrets/wireguard-privateKey"; peers = [ { publicKey = "897mRPejuv9yVnmTvcUL7ckQkIiM0wnSgHmgR15Evyw="; allowedIPs = [ "172.20.60.10/32" ]; presharedKeyFile = "/etc/nixos/secrets/wireguard-presharedkey"; } ... systemd.network.networks = { "10-eno1" = { matchConfig.Name = "eno1"; networkConfig.LinkLocalAddressing = "no"; networkConfig.DHCP = "no"; extraConfig = '' VLAN=wan VLAN=vlan99 VLAN=vlan30 VLAN=vlan20 VLAN=vlan1 VLAN=podnet LLDP=no EmitLLDP=no IPv6AcceptRA=no IPv6SendRA=no ''; }; "11-vlan1" = { matchConfig.Name = "vlan1"; linkConfig.RequiredForOnline = false; networkConfig.DHCP = "no"; networkConfig.Address = "192.168.1.1/24"; networkConfig.Domains = "tanso.net"; networkConfig.ConfigureWithoutCarrier = "yes"; }; "11-podnet" = { matchConfig.Name = "podnet"; linkConfig.RequiredForOnline = false; networkConfig.DHCP = "no"; networkConfig.Address = "172.20.2.1/24"; networkConfig.Domains = "tanso.net"; networkConfig.ConfigureWithoutCarrier = "yes"; }; "11-vlan20" = { matchConfig.Name = "vlan20"; networkConfig.DHCP = "no"; networkConfig.Address = "172.20.20.1/24"; networkConfig.Domains = "tanso.net"; networkConfig.ConfigureWithoutCarrier = "yes"; extraConfig = '' IPv6SendRA=yes DHCPv6PrefixDelegation=yes ''; }; .... systemd.network.netdevs = { "11-vlan1" = { netdevConfig = { Name = "vlan1"; Kind = "vlan"; }; vlanConfig.Id = 1; }; "11-podnet" = { netdevConfig = { Name = "podnet"; Kind = "vlan"; }; vlanConfig.Id = 2; }; "11-vlan20" = { netdevConfig = { Name = "vlan20"; Kind = "vlan"; }; vlanConfig.Id = 20; }; "11-vlan30" = { netdevConfig = { Name = "vlan30"; Kind = "vlan"; }; vlanConfig.Id = 30; };

Asdrubalini 4y ago

Wow, didn’t know about Nixus, thanks!

colordrops 4y ago

Can you share your router configs? I'd love to have a NixOS based router.

astro1 4y ago
https://gitea.c3d2.de/zentralwerk/network https://gitea.c3d2.de/zentralwerk/network
notemaker 4y ago
https://skogsbrus.xyz/blog/2022/06/12/router https://skogsbrus.xyz/blog/2022/06/12/router
ochoseis 4y ago
One drawback I realized about NixOS as a router for home use is that if you need to share administration of the network, then it has to be easy for non-experts. In that case something like OpenWRT might be a better choice.
- pmarreck 4y ago
  Well, at least the config file options are relatively straightforward, but you're right in that a GUI makes more sense for genpop

ochoseis 4y ago

How are you managing the VM’s on NixOS? A couple years ago I was searching for a good way to declaratively manage them, and the best I came up with was libvirt + terraform + nixos-generators for images. It feels like you should just be able to set them up as systemd units, but I couldn’t figure it out. microvm has caught my eye recently but I haven’t played with it yet: https://github.com/astro/microvm.nix https://github.com/astro/microvm.nix

amarshall 4y ago
Not OP (and I suspect they don’t have what you’re looking for since they have stated they treat the VMs as pets, not cattle). I currently use libvirt, but I eschew persistent VMs and instead build the libvirt domain XML with Nix, and then start VMs with `virsh create /path/to/myvm.xml`. Using an absolute path means that libvirt will not persist the domain, and makes it easier to declaratively manage the config with Nix. However, none of my VMs are always-on, so I don’t have to deal with restarts on config change. That said, when libvirt isn’t managing the lifecycle of the VMs as much, writing a systemd unit should be quite a bit easier.
madjam002 4y ago
The VMs themselves are a mix of some manually created ones (the VM configs are still NixOS though), and some IaC VMs which I just use Terraform and the libvirt provider. It's a bit clunky but I wouldn't want to embed the VM config in the hypervisor configuration as the VMs can be live migrated to other hypervisors. MicroVM looks awesome though, thanks for the pointer!
- ochoseis 4y ago
  Did you manage to get live migration working with the terraform provider? As of a while ago it wasn’t implemented but idk if that’s changed: https://github.com/dmacvicar/terraform-provider-libvirt/issues/490 https://github.com/dmacvicar/terraform-provider-libvirt/issu...

1MachineElf 4y ago

I appreciate your comment here, and the parent comment. I've been using OPNSense[0] as my router for the past several months. So far, I am quite happy with it, but I've also thought that NixOS would be the next step. My OPNSense router has 30+ VLANs and many layered firewall rules (my take on "zEr0 tRuSt") and so the task of converting it all to NixOS seems a little daunting. I appreciate the utility of OPNSense's web GUI when configuring and troubleshooting my router config. It would be awesome if something like that could be integrated with NixOS. Additionally, something like nsh[1] to provide a traditional router/switch style CLI would be a dream come true. [0] https://opnsense.org/ https://opnsense.org/ [1] https://www.nmedia.net/nsh/ https://www.nmedia.net/nsh/

hamandcheese 4y ago
NixOS is great for a basic home router (I use it for my home router) but it’s networking config is still pretty rudimentary, and some things I would expect to work just don’t - i.e. port forwarding only works from outside your network, not inside. I haven’t done much with vlans yet so I can’t comment on that.
- SuperSandro2000 4y ago
  I would highly recommend to use systemd-networkd based networking which should be able to do almost everything if configured correct.
  - hamandcheese 4y ago
    Is there a nix config wrapper for systemd-networkd? NixOS still does everything I need it too, but the parent comment sounded like they had a bit more complex of a setup.
    - janfrode 4y ago
      Not sure what you mean by config wrapper, but yes, systems-networkd is quite nice. Here's my home router setup with vlan1 for uplink to ISP, a bunch of other vlans for internal networks, and IPv6 prefix delegation to internal networks, and of course fireguard. All configured using systemd-networkd: { pkgs, lib, ... }:{ networking = { useNetworkd = true; useDHCP = false; enableIPv6 = true; }; networking.wireguard.interfaces = { wireguard = { ips = [ "172.20.60.1/24" ]; listenPort = 61891; privateKeyFile = "/etc/nixos/secrets/wireguard-privateKey"; peers = [ { publicKey = "897mRPejuv9yVnmTvcUL7ckQkIiM0wnSgHmgR15Evyw="; allowedIPs = [ "172.20.60.10/32" ]; presharedKeyFile = "/etc/nixos/secrets/wireguard-presharedkey"; } ... systemd.network.networks = { "10-eno1" = { matchConfig.Name = "eno1"; networkConfig.LinkLocalAddressing = "no"; networkConfig.DHCP = "no"; extraConfig = '' VLAN=wan VLAN=vlan99 VLAN=vlan30 VLAN=vlan20 VLAN=vlan1 VLAN=podnet LLDP=no EmitLLDP=no IPv6AcceptRA=no IPv6SendRA=no ''; }; "11-vlan1" = { matchConfig.Name = "vlan1"; linkConfig.RequiredForOnline = false; networkConfig.DHCP = "no"; networkConfig.Address = "192.168.1.1/24"; networkConfig.Domains = "tanso.net"; networkConfig.ConfigureWithoutCarrier = "yes"; }; "11-podnet" = { matchConfig.Name = "podnet"; linkConfig.RequiredForOnline = false; networkConfig.DHCP = "no"; networkConfig.Address = "172.20.2.1/24"; networkConfig.Domains = "tanso.net"; networkConfig.ConfigureWithoutCarrier = "yes"; }; "11-vlan20" = { matchConfig.Name = "vlan20"; networkConfig.DHCP = "no"; networkConfig.Address = "172.20.20.1/24"; networkConfig.Domains = "tanso.net"; networkConfig.ConfigureWithoutCarrier = "yes"; extraConfig = '' IPv6SendRA=yes DHCPv6PrefixDelegation=yes ''; }; .... systemd.network.netdevs = { "11-vlan1" = { netdevConfig = { Name = "vlan1"; Kind = "vlan"; }; vlanConfig.Id = 1; }; "11-podnet" = { netdevConfig = { Name = "podnet"; Kind = "vlan"; }; vlanConfig.Id = 2; }; "11-vlan20" = { netdevConfig = { Name = "vlan20"; Kind = "vlan"; }; vlanConfig.Id = 20; }; "11-vlan30" = { netdevConfig = { Name = "vlan30"; Kind = "vlan"; }; vlanConfig.Id = 30; };