4 ms·

Hey y'all, I know getting a setup that feels "right" can be a process. We all have different goals, tech preferences, etc. I wanted to a share my blog post wal

Hey y'all, I know getting a setup that feels "right" can be a process. We all have different goals, tech preferences, etc. I wanted to a share my blog post walking through how I finally built a setup that I can just be happy with and use. It goes over my goals, requirements, tech choices, layout, and some specific problems I've resolved. Where I've landed of course isn't where everyone else will, but I hope it can serve as a good reference. I’ve really benefited from the content and software folks have freely shared, and hope I can continue that and help others.

redrove 1y ago
How are you finding Nix for the homelab to be? Every time I try it I just end up confused, maybe next time will be the charm. The reason I ask is I homelab “hardcore”; i.e. I have a 25U rack and I run a small Kubernetes cluster and ceph via Talos Linux. Due to various reasons, including me running k8s in the lab for about 7 years now, I’ve been itching to change and consolidate and simplify, and every time i think about my requirements I somehow end up where you did: Nix and ZFS. All those services and problems are very very familiar to me, feel free to ask me questions back btw.
- MisterKent 1y ago
  I've been trying to switch my home cluster from Debian + K3s to Talos but keep running into issues. What does your persistent storage layer look like on Talos? How have you found it's hardware stability over the long term?
  - esseph 1y ago
    Talos is the Linux kernel at heart, so.. just fine.
  - redrove 1y ago
    >What does your persistent storage layer look like on Talos? Well, for its own storage: it's an immutable OS that you can configure via a single YAML file, it automatically provisions appropriate partitions for you, or you can even install the ZFS extension and have it use ZFS (no zfs on root though). For application/data storage there's a myriad of options to choose from[0]; after going back and forth a few times years ago with Longhorn and other solutions, I ended up at rook-ceph for PVCs and I've been using it for many years without any issues. If you don't have 10gig networking you can even do iSCSI from another host (or nvmeof via democratic-csi but that's quite esoteric). >How have you found it's hardware stability over the long term? It's Linux so pretty good! No complaints and everything just works. If something is down it's always me misconfiguring or a hardware failure. [0] https://www.talos.dev/v1.11/kubernetes-guides/configuration/storage/#storage-clusters https://www.talos.dev/v1.11/kubernetes-guides/configuration/...
    - avtar 1y ago
      > after going back and forth a few times years ago with Longhorn and other solutions, I ended up at rook-ceph for PVCs Curious to know what issues you ran into with Longhorn.
      - redrove 1y ago
        It was years ago but I recall high CPU usage being an issue in particular. In general it's just not as battle tested as ceph and I needed something more bulletproof. However I will say this: I'm sure that issue with the CPU usage was fixed (I was watching the GitHub issue) and you might not need your distributed FS to be CERN ready for your lab; AND the UI and built-in backups Longhorn offers are great for beginners so I'd suggest giving it a try if you don't already know you want ceph or OpenEBS Mayastor for the performance and so on.
        avtar 1y ago
        Thanks! I haven’t had to implement replicated storage (still using EFS) but I was curious about Longhorn.
- udev4096 1y ago
  Honestly, I personally like the combination of keepalived+docker(swarm if needed)+rsync for syncing config files. keepalived uses VRRP, which creates a floating IP. It's extremely lightweight and works like a charm. You won't even notice the downtime, the switch to another server IP is instant
  - cess11 1y ago
    Keepalived is great. Learning about it was one of the best things I got from building HA-aiming infra at a job once.
- mirdaki 1y ago
  I certainly didn't take to Nix the first few times I looked at it. The language itself is unusual and the error messages leave much to be desired. And the split around Flakes just complicates things further (though I do recommend using them, once you set it up, it's simple and the added reproducibility gives nice peace of mind) But once I fully understood how it's features really make it easy for you to recover from mistakes and how useful the package options available from nixpkgs are, I decided it was time to sink in and figure it out. Looking at other folks nix config on GitHub (especially for specific services you're wanting to use) is incredibly helpful (mine is also linked in the post) I certainly don't consider myself to be a nix expert, but the nice thing is you can do most things by using other examples and modifying them till you feel good about it. Then overtime you just get more familiar with and just grow your skill Oh man, having a 25U rack sounds really fun. I have a moderate size cabinet I keep my server, desktop, a UPS, 10Gig switch, and my little fanless Home Assistant box. What's yours look like? I should add it to the article, but one of my anti-requirements was anything in the realm of high availability. It's neat tech to play with, but I can deal with downtime for most things if the trade off is everything being much simpler. I've played a little bit with Kubernetes at work, but that is a whole ecosystem I've yet to tackle
  - redrove 1y ago
    >The language itself is unusual and the error messages leave much to be desired. And the split around Flakes just complicates things further Those are my chief complaints as well, actually. I never quite got to the point where I grasped how all the bits fit together. I understand the DSL (though the errors are cryptic as you said) and the flakes seemed recommended by everyone yet felt like an addon that was forgotten about (you needed to turn them on through some experimental flag IIRC?). I'll give it another shot some day, maybe it'll finally make sense. >Oh man, having a 25U rack sounds really fun. I have a moderate size cabinet I keep my server, desktop, a UPS, 10Gig switch, and my little fanless Home Assistant box. What's yours look like? * 2 UPSes (one for networking one for compute + storage) * a JBOD with about 400TB raw in ZFS RAID10 * a little intertech case with a supermicro board running TrueNAS (that connects to the JBOD) * 3 to 6 NUCs depending on the usage, all running Talos, rook-ceph cluster on the NVMEs, all NUCs have a Sonnet Solo 10G Thunderbolt NIC * 10 Gig unifi networking and a UDM Pro * misc other stuff like a zima blade, a pikvm, shelves, fans, ISP modem, etc I'm not necessarily thinking about downsizing but the NUCs have been acting up and I've gotten tired of replacing them or their drives so I thought I'd maybe build a new machine to rule them all in terms of compute and if I only want one host then k8s starts making less sense. Mini PCs are fine if you don't push them to the brim like I do. I'm a professional k8s engineer I guess, so on the software side most of this comes naturally at this point.
raybb 1y ago
Did you come across or consider using coolify at any point? I've been using it for over a year and quite enjoyed it for it's Heroku type ease of use and auto deployments from GitHub. https://coolify.io/ https://coolify.io/
- mirdaki 1y ago
  No I haven't heard of it before. I do like the idea though, especially for side projects. Thanks for sharing, I'll look more at it!
  - oulipo 1y ago
    Check out Dokploy too! https://dokploy.com/ https://dokploy.com/
    - _9y71 1y ago
      Their license is still ambiguous, and I don't like how they communicate with those who inquire about it.
colordrops 1y ago
Hi! Really excited by your work! I'm working on a similar project built on NixOS and curious what you thing. My goal is to have a small nearly zero-conf apple-device-like box that anyone can install by just plugging it into their modem then going through a web-based installation. It's still very nascent but I'm already running it at home. It is a hybrid router (think OPNSense/PFSense) + app server (nextcloud, synology, yunohost etc). All config is handled through a single Nix module. It automatically configures dynamic DNS, Letsencrypt TLS certs, and subdomains for each app. It's got built in ad blocking and headscale. I'm working on SSO at the moment. I'll take a look at your work and maybe steal some ideas. The project is currently self-hosted in my closet: https://homefree.host https://homefree.host
- mirdaki 1y ago
  Oh that sounds really rad! Certainly could have it's use cases. I really appreciate how NixOS enables projects like this. Best of luck with it!
ultra2d 1y ago
Do you use encrypted ZFS? I have dabbled before with FreeIPA and other VMs on a Debian host with ZFS. For simplicity, I switched to running Seafile with encrypted libraries on a VPS and back that up to a local server via ZFS send/receive. That local server switches itself on every night, updates, syncs and then goes into sleep again. For additional resiliency, I'm thinking of switching to ZFS on Linux desktop (currently Fedora), fully encrypted except for Steam. Then sync that every hour or so to another drive in the same machine, and sync less frequently to a local server. Since the dataset is already encrypted, I can either sync to an external drive or some cloud service. Another reason to do it like this is that storing a full photo archive within Seafile on a VPS is too costly.
- mirdaki 1y ago
  Yes! On top of the data safety features of ZFS, the fact you can encrypt a dataset and incremental send/receive is a fantastic ability
A4ET8a8uTh0_v2 1y ago
I appreciated the in depth look and while some ideas from your setup will take more time to implement, I just added flame for the dashboard and see how it fares with family.
- mirdaki 1y ago
  Thank you! It's all a journey, hope flame works well for you!

redrove 1y ago

How are you finding Nix for the homelab to be? Every time I try it I just end up confused, maybe next time will be the charm. The reason I ask is I homelab “hardcore”; i.e. I have a 25U rack and I run a small Kubernetes cluster and ceph via Talos Linux. Due to various reasons, including me running k8s in the lab for about 7 years now, I’ve been itching to change and consolidate and simplify, and every time i think about my requirements I somehow end up where you did: Nix and ZFS. All those services and problems are very very familiar to me, feel free to ask me questions back btw.

MisterKent 1y ago
I've been trying to switch my home cluster from Debian + K3s to Talos but keep running into issues. What does your persistent storage layer look like on Talos? How have you found it's hardware stability over the long term?
- esseph 1y ago
  Talos is the Linux kernel at heart, so.. just fine.
- redrove 1y ago
  >What does your persistent storage layer look like on Talos? Well, for its own storage: it's an immutable OS that you can configure via a single YAML file, it automatically provisions appropriate partitions for you, or you can even install the ZFS extension and have it use ZFS (no zfs on root though). For application/data storage there's a myriad of options to choose from[0]; after going back and forth a few times years ago with Longhorn and other solutions, I ended up at rook-ceph for PVCs and I've been using it for many years without any issues. If you don't have 10gig networking you can even do iSCSI from another host (or nvmeof via democratic-csi but that's quite esoteric). >How have you found it's hardware stability over the long term? It's Linux so pretty good! No complaints and everything just works. If something is down it's always me misconfiguring or a hardware failure. [0] https://www.talos.dev/v1.11/kubernetes-guides/configuration/storage/#storage-clusters https://www.talos.dev/v1.11/kubernetes-guides/configuration/...
  - avtar 1y ago
    > after going back and forth a few times years ago with Longhorn and other solutions, I ended up at rook-ceph for PVCs Curious to know what issues you ran into with Longhorn.
    - redrove 1y ago
      It was years ago but I recall high CPU usage being an issue in particular. In general it's just not as battle tested as ceph and I needed something more bulletproof. However I will say this: I'm sure that issue with the CPU usage was fixed (I was watching the GitHub issue) and you might not need your distributed FS to be CERN ready for your lab; AND the UI and built-in backups Longhorn offers are great for beginners so I'd suggest giving it a try if you don't already know you want ceph or OpenEBS Mayastor for the performance and so on.
      - avtar 1y ago
        Thanks! I haven’t had to implement replicated storage (still using EFS) but I was curious about Longhorn.
udev4096 1y ago
Honestly, I personally like the combination of keepalived+docker(swarm if needed)+rsync for syncing config files. keepalived uses VRRP, which creates a floating IP. It's extremely lightweight and works like a charm. You won't even notice the downtime, the switch to another server IP is instant
- cess11 1y ago
  Keepalived is great. Learning about it was one of the best things I got from building HA-aiming infra at a job once.
mirdaki 1y ago
I certainly didn't take to Nix the first few times I looked at it. The language itself is unusual and the error messages leave much to be desired. And the split around Flakes just complicates things further (though I do recommend using them, once you set it up, it's simple and the added reproducibility gives nice peace of mind) But once I fully understood how it's features really make it easy for you to recover from mistakes and how useful the package options available from nixpkgs are, I decided it was time to sink in and figure it out. Looking at other folks nix config on GitHub (especially for specific services you're wanting to use) is incredibly helpful (mine is also linked in the post) I certainly don't consider myself to be a nix expert, but the nice thing is you can do most things by using other examples and modifying them till you feel good about it. Then overtime you just get more familiar with and just grow your skill Oh man, having a 25U rack sounds really fun. I have a moderate size cabinet I keep my server, desktop, a UPS, 10Gig switch, and my little fanless Home Assistant box. What's yours look like? I should add it to the article, but one of my anti-requirements was anything in the realm of high availability. It's neat tech to play with, but I can deal with downtime for most things if the trade off is everything being much simpler. I've played a little bit with Kubernetes at work, but that is a whole ecosystem I've yet to tackle
- redrove 1y ago
  >The language itself is unusual and the error messages leave much to be desired. And the split around Flakes just complicates things further Those are my chief complaints as well, actually. I never quite got to the point where I grasped how all the bits fit together. I understand the DSL (though the errors are cryptic as you said) and the flakes seemed recommended by everyone yet felt like an addon that was forgotten about (you needed to turn them on through some experimental flag IIRC?). I'll give it another shot some day, maybe it'll finally make sense. >Oh man, having a 25U rack sounds really fun. I have a moderate size cabinet I keep my server, desktop, a UPS, 10Gig switch, and my little fanless Home Assistant box. What's yours look like? * 2 UPSes (one for networking one for compute + storage) * a JBOD with about 400TB raw in ZFS RAID10 * a little intertech case with a supermicro board running TrueNAS (that connects to the JBOD) * 3 to 6 NUCs depending on the usage, all running Talos, rook-ceph cluster on the NVMEs, all NUCs have a Sonnet Solo 10G Thunderbolt NIC * 10 Gig unifi networking and a UDM Pro * misc other stuff like a zima blade, a pikvm, shelves, fans, ISP modem, etc I'm not necessarily thinking about downsizing but the NUCs have been acting up and I've gotten tired of replacing them or their drives so I thought I'd maybe build a new machine to rule them all in terms of compute and if I only want one host then k8s starts making less sense. Mini PCs are fine if you don't push them to the brim like I do. I'm a professional k8s engineer I guess, so on the software side most of this comes naturally at this point.

raybb 1y ago

Did you come across or consider using coolify at any point? I've been using it for over a year and quite enjoyed it for it's Heroku type ease of use and auto deployments from GitHub. https://coolify.io/ https://coolify.io/

mirdaki 1y ago
No I haven't heard of it before. I do like the idea though, especially for side projects. Thanks for sharing, I'll look more at it!
- oulipo 1y ago
  Check out Dokploy too! https://dokploy.com/ https://dokploy.com/
  - _9y71 1y ago
    Their license is still ambiguous, and I don't like how they communicate with those who inquire about it.

colordrops 1y ago

Hi! Really excited by your work! I'm working on a similar project built on NixOS and curious what you thing. My goal is to have a small nearly zero-conf apple-device-like box that anyone can install by just plugging it into their modem then going through a web-based installation. It's still very nascent but I'm already running it at home. It is a hybrid router (think OPNSense/PFSense) + app server (nextcloud, synology, yunohost etc). All config is handled through a single Nix module. It automatically configures dynamic DNS, Letsencrypt TLS certs, and subdomains for each app. It's got built in ad blocking and headscale. I'm working on SSO at the moment. I'll take a look at your work and maybe steal some ideas. The project is currently self-hosted in my closet: https://homefree.host https://homefree.host

mirdaki 1y ago
Oh that sounds really rad! Certainly could have it's use cases. I really appreciate how NixOS enables projects like this. Best of luck with it!

ultra2d 1y ago

Do you use encrypted ZFS? I have dabbled before with FreeIPA and other VMs on a Debian host with ZFS. For simplicity, I switched to running Seafile with encrypted libraries on a VPS and back that up to a local server via ZFS send/receive. That local server switches itself on every night, updates, syncs and then goes into sleep again. For additional resiliency, I'm thinking of switching to ZFS on Linux desktop (currently Fedora), fully encrypted except for Steam. Then sync that every hour or so to another drive in the same machine, and sync less frequently to a local server. Since the dataset is already encrypted, I can either sync to an external drive or some cloud service. Another reason to do it like this is that storing a full photo archive within Seafile on a VPS is too costly.

mirdaki 1y ago
Yes! On top of the data safety features of ZFS, the fact you can encrypt a dataset and incremental send/receive is a fantastic ability

A4ET8a8uTh0_v2 1y ago

I appreciated the in depth look and while some ideas from your setup will take more time to implement, I just added flame for the dashboard and see how it fares with family.

mirdaki 1y ago
Thank you! It's all a journey, hope flame works well for you!