5 ms·

Rebuilding my homelab: Suffering as a service

106 points by xena 2y ago

PaulHoule 2y ago
No. Real suffering in the homelab is getting N 20 year old servers and swapping parts between them to get N-M servers that work. I feel like the project will be successful if I get all the drives wiped and I am within site of that although I discovered the wiping was going to be a process of triage: some drives did not spin up, one drive took 14 hours to wipe whereas a normal drive would take about 30 minutes. My collaborator will use the bad drives for target practice for a black powder rifle that shoots round balls. Noisy fans take the "home" out of the homelab; the machines are 64 bit Intel but top out at 4GB. The latest version of Ubuntu installs fine but the desktop struggles. I think I'm going to install the desktop Ubuntu again just to see if I can watch YouTube with it but the plan now is install the server edition and give it to my collaborator to run an occasional minecraft server, which might free up a (much more powerful) i3 machine to watch videos from my Jellyfin server on the TV downstairs, something the XBOX ONE oddly can't handle. (No patent licenses for codecs if it's a game console?) At least I dug out the old VGA-supporting monitors out of mothballs so I'll be ready to play around with the RISC-V and eZ80 SBCs I have which are, at the very least, a lot quieter.
- shrubble 2y ago
  That is retrocomputing, not homelabbing. Look into the sub-$200 Intel N100 based systems with 16GB RAM.
  - jjbinx007 2y ago
    I recently bought 2 of these and they are EXCELLENT! I can leave them running 24/7 without worrying about how much electricity they're using. The performance, flexibility and reliability far exceeds the Raspberry Pis that are confined the to cupboard and they're probably a fair bit faster than my old desktop PCs that I rarely switch on any more. I've gone uber-minimalist and only have NVME drives attached via USB-3. One's connected via ethernet, the other has a wifi connection. Personally I don't need any more and I've retired my old servers for now.
    - distances 2y ago
      > reliability far exceeds the Raspberry Pis In which ways? Most commenters say the exact opposite, that cheap N100 mini PCs are less reliable than Raspberry Pis. I'm just now trying to decide which way to go. Raspberry Pi 5 is definitely much much more interesting from the nerdy point of view, but would cost about the same as some cheap N100, for half the power. Though half the electricity usage too.
      - zamadatix 2y ago
        If you get away with a 1/2 GB RAM version I'd say the Pi 5 is worth it, otherwise cheap mini PCs are better. I've found mini PCs to be more reliable than the Pi's I've owned as well (including the 5), though a lot of that is mitigated if you go for an m.2 hat on the Pi. In general though, if you're thinking of using a Pi more like a regular computer there isn't really anything special about it that makes it worthwhile. Well, beyond "I get to tinker more" if you particularly like assembling the thing, picking out the exact case, finding your favorite m.2 hat, etc. Even if you enjoy doing all that by the time you're done you still end with a PC that is worse than one you could just buy but maybe with a claim of saying you saved $20 doing so. Where I like the Pi is use cases that don't quite fit with a normal mini PC like IP KVM.
        distances 2y ago
        My use case is Home Assistant, Pi-hole and such things. I would go for the 8 GB model and the NVMe hat, ending up somewhere around 170-180€. So practically the same price as N100. With RPi I do like the tinkering aspect, an actual community, and lower power consumption -- I'd prefer passive cooling but I can't see a way to do that with the NVMe hat. With N100 I would get double CPU power, RAM and storage. But I don't actually need that extra power, and the concept of buying a cheap pre-built PC is just a lot less appealing than something more nerdy. So I'll probably end up getting the RPi just for the, err, shall we call emotional reasons.
- Aurornis 2y ago
  I think we need a different word for collecting a lot of old, underpowered computers and tinkering endlessly. I understand the attraction of playing with old, cheap hardware. However, hardware has come so far that it’s easy to build a 16-core server with a lightly used AMD consumer chip and 64-128GB of RAM for under $1000. It will have more power and use far less energy than these clusters of old machines that I see people assembling. > Noisy fans take the "home" out of the homelab; Again, a completely unnecessary thing to suffer. If the goal is a homelab. It’s really easy to make a near-silent PC with modern parts and cooling that will outperform an entire rack of 20 year old PCs. Even 10G switches that are quiet or fan less are common. I get it. It can be fun. But I don’t think this is homelabbing.
fallingsquirrel 2y ago
The CoreOS diversion was interesting to read. I've been daily driving CoreOS+i3 for the past year (I might be the only one in the world). I thought having a tiny immutable base OS would make the system easier to manage over time, but unfortunately that hasn't been the case. It's been an adventure but I'm ready to give up and switch to something more vanilla.
- spicyusername 2y ago
  I had the same experience. Immutable OS are great when you have a team of people to manage the tooling and process complexity of deploying and managing them, end users who only want to use software running on those servers, not the servers themselves, and you are bringing servers up and down all the time, then you actually reap the benefits of standardization and infrastructure as code. But when it's just you, you want to interact directly with the OS, and it's just one device, it's just foot guns all day.
- rubenbe 2y ago
  I've keep running Fedora coreos on my home server. My biggest issue with it, is that it is very cloud oriented and doesn't seem to allow to rerun the provisioning config on an already existing machine. This turns the thing again into a stateful pet instead of a "one cow cattle". Although I do very much like the rollback feature which has allowed me temporarily roll back an update a couple of times
  - walterbell 2y ago
    > doesn't seem to allow to rerun the provisioning config on an already existing machine In theory, a generic existing machine could have been compromised by malware, in which case the configuration may not match the previously provisioned version. With OS launch integrity to guarantee absence of tampering, and prove that current=expected config+binaries, it could be feasible to rerun provisioning config.
- xena 2y ago
  I've wondered if that would be possible for a while, but I didn't imagine anyone would actually do that. What are the upsides and downsides of doing this? I'd love to read a writeup of how you did that and what you'll miss when you move away.
kkfx 2y ago
Well, a translation can be: a NixOS user tried some from the IT dark-modern age and find how much it suck, nothing extraordinary, unfortunately most people fails to even understand that some Big of IT have interest in crappy solutions because they allow commerce while good ones do not. Back then was the full-stack-virtualization era where a large flock of biped sheep say that VMWare "is the future, today" of course on x86 with the incredible overhead it have. For some times the psyco-PR-drug works, than paravirtualization start to be know with docker before, k8s after. Again the aforementioned flock run for the mania again ignoring the fact it's a nonsense. Ladies and gentlemen this solution have only ONE purpose sell pre-build crap for those who do no know how to work themselves, mostly the Dev part of DevOps who in mean have issues even deploying their own private personal desktop at home, not even a homelab. They do not care about the tremendous overhead, "hw it's relatively cheap and I'm rich enough to buy more", and they consider normal the abysmal level of complexity. So we see in-production docker images built by no-one-know who, perhaps with his/her ssh keys authorized "oups, I forgot to remove them" and so on. NixOS unfortunately have a terrible language, Guix System unfortunately it's mostly focused on HPC/academic usage instead of homelab/desktop usage, but that's the "modern IaC" since years. They are incredibly simpler and light than the modern crap pushed to sell pre-made images, offering "cloud services" of various kind and so on, yes Hetzner or OVH or Amazon do need to been able to sell so they do need these monsters, but users do need to AVOID them and re-learn how to use and deploy OSes and app, perhaps in a less archaic ways than the 80s style...
- PhilipRoman 2y ago
  >Big of IT have interest in crappy solutions because they allow commerce while good ones do not. This is really true, I don't want to go into specifics but I've witnessed a standards committee advertising how much "vendor added value" their product allows (which is fancy language for the standard being incomplete and unusable without extensions). Oh and don't forget arbitrary limits on capabilities for no good reason so that in a few years marketing teams can release a new version with a fancy nickname to keep the hype running.
  - kkfx 2y ago
    The big issue is that we can't go much further that way. That's why for instance China industries advance while we regress. Interoperability and diversity are keys to success, cutting them means cutting the branch we are sit on... These days many developers do not even know how to deploy their apps behind some third party APIs, finding a sysadmin it extremely hard and most are just "homelab guys with a bit of experience", how can we build something with this holes under our own feet? A stupid example: photovoltaic is now relatively popular and that's clear the sole reasonable usage we have so far is self-consumption, we also have LFP storage still high priced (respect of China) but at a price tag sufficient to buy some storage at home like a giant UPS for a home, that's actually 400V batteries, most common inverters recharge directly from their MPPTs, actually the very same batteries are in most BEV. Well NO DAMN SINGLE VENDOR offer DC-to-DC direct charge for cars, simply nobody have apparently thought of that. We even have a standard https://www.iso.org/standard/77845.html https://www.iso.org/standard/77845.html but nobody seems to have implemented it commercially. We have a significant set of IoT appliance, only very few offer at least an open standard protocol for integration and most of them offer only ModBUS as an open standard protocol, something from the '70s, nice for certain usage also today but way too limited for many other possible usage. MQTT is a complicated option, still valid, very few implement it. Two more easy open options exists (Kafka, Matter) but nobody seems to implement them. We have a gazillion of VoIP solution almost no one is really simple and ready to deploy for anyone. Mumble/MurMur are the easiest for voice, but they are just chat, no calling ability, GNU SIPWitch is a lightweight SIP alternative, but it's far from being comfy for SOHO usage and Astersik/Yate are simply too complex for most users, similarly the first voice + screen sharing was in 1968 "The Mother of All Demos" we still lack a broad comfy screen sharing + voice solution, we even lack an IPv6 global per host to ease anything. Long story short we waste gazillion of resources to maintain immense pile of crap following the classic https://xkcd.com/2347/ https://xkcd.com/2347/ creating fragile monsters and struggling to go past this sorry state.
sunshine-o 2y ago
So in summary a former NixOS user now use a preconfigured OS dedicated to run one single platform (k8s) and still suffer for having to tinker with everything. It would have been fun or wise if she had gone back to Nix in the end.
- xena 2y ago
  I still use Nix to build docker images, part of this is to see how bad the rest of the industry really is. It's slightly worse than I imagined it would be.
- srid 2y ago
  > a former NixOS user Why did the author stop using NixOS? This is the first time I'm hearing about a veteran NixOS user giving up on it.
  - shepherdjerred 2y ago
    It mostly seems due to some drama going on in the Nix community rather than some technical reason. https://xeiaso.net/blog/2024/much-ado-about-nothing/ https://xeiaso.net/blog/2024/much-ado-about-nothing/
    - xena 2y ago
      I would call the situation "a complete failure of management" instead of "drama".
      - shepherdjerred 2y ago
        I used "drama" because I'm completely detached from the Nix community, and I have no idea what's going on over there :)
    - matheusmoreira 2y ago
      Can't believe I'm witnessing another node/io.js thing in 2024.
khaki54 2y ago
[flagged]
chadsix 2y ago
This was a tremendous write-up. I appreciate the detail including your ingressd setup. I agree, though, that this is a pain. It's for this reason that we made Cloud Seeder [1] so you can have hands-free setup of your homelab and IPv6rs for painless ingress [2] </shameless> [1] https://github.com/ipv6rslimited/cloudseeder https://github.com/ipv6rslimited/cloudseeder [2] https://ipv6.rs https://ipv6.rs
- xena 2y ago
  I'd set up IPv6 on there, but the problem is that I use Flannel (which is IPv4 only) and my 8 gigabit fiber ISP only gives me IPv4 connectivity. I'll look into more details, but I've slightly given up on IPv6 for now. Maybe I'll set up Calico or something, but IPv6 seems to have been made artificially difficult by everything in the stack. I hate it.
  - miyuru 2y ago
    Can you add AAAA records to cdn.xeiaso.net as well. It seems to have the same IPv4 from fly.io as the main domain, but you forgot to add it to the CDN subdomain.
    - xena 2y ago
      Uhhhh, I thought I did that! I'm gonna go fix that, sorry!
      - xena 2y ago
        Did that fix it?
  - chadsix 2y ago
    > 8 gigabit fiber ISP only gives me IPv4 connectivity. Just like my ISP (but not 8 gbit!). Luckily, IPv6rs actually tunnels through IPv4 (or 6) and provides an IPv6 address. You don't need one to start! I don't know the ins and outs for flannel, but maybe you could setup IPv4 internally and use IPv6 (and an IPv4 Reverse Proxy) for the public internet? I agree, though, IPv6 on its own can be hard but thanks to WireGuard, tayga (NAT64), and nginx/caddy/etc. (reverse proxy), it's definitely quite usable!
shrubble 2y ago
My contrarian take is that eventually k8s will be recognized as the overcomplication it is; and better methods of managing less than 10,000 VMs will be researched and used.
- xena 2y ago
  I really hope that is the case too, but for now Kubernetes sucked all the oxygen out of the room for everything else :(
- xav0989 2y ago
  I switched my home lab to nomad, which I find much easier to wrangle, but we’ll see what happens with the IBM acquisition.
- Aurornis 2y ago
  People don’t use Kubernetes at home because it’s the easiest tool for the job. They use it in home labs because it’s a safe and easy environment to learn, practice, and explore. Most importantly: It’s a low-consequence environment. If you accidentally bring everything down at your homelab, the only person who suffers is you. You don’t get that degree of safety to explore and experiment when you’re using company resources.
  - snapplebobapple 2y ago
    Incorrect sir. I am way more concerned about my wife and kids as end users than i am about any employees
    - bbkane 2y ago
      You're more concerned about your home Plex server going down than the service you run at work?
      - ffsm8 2y ago
        Doubtful. But things like mail, documents, file storage, etc might be another story. It depends on what the person in question is hosting, and how it relates to the life of the SO. There are a lot of options around, just take a look at the awesome-selfhosted repo on github
wint3rmute 2y ago
After running NixOS for 6+ months on my homelab and also re-using part of the configuration on my work machine, I feel the same way as Xe each time I'm interacting with a non-declarative OS. There's just no simple way to share configuration between machines or to automagically clean things up after making changes. Ansible feels like a thin layer of ice upon a deep ocean of the OS state, hiding in a multitude of non-tracked configuration files. It is simply not enough to build a layer of YAML around an OS which is imperative by nature. Unfortunately, I can see the downsides of NixOS as well, being radically different from what we usually expect in a Linux distribution, adopting it in a already established environment will no doubt be hard. Steep learning curve, idiosyncracies of the Nix language (although after reading parts of the Nix thesis[1], I find it much more understandable and deeply thought out), just explaining Nix to people who don't have much experience with the functional way of doing things, let alone taking the functional approach all the way to defining an entire operating system - all of this sounds like a tough barrier to cross. And yet, the desire to keep things reproducible and declarative (not to mention going back in time) persists once you've had the taste of NixOS. [1] https://edolstra.github.io/pubs/phd-thesis.pdf https://edolstra.github.io/pubs/phd-thesis.pdf
- jt2190 2y ago
  I’m picking this nit: > When is a build reproducible? > “A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.” > Neither Nix or NixOS gives you these guarantees. This really makes me question whether all of the quirkiness of Nix is worth it if it can’t actually “pay off” with true reproducibility. [1] “NixOS is not reproducible (2022) https://linderud.dev/blog/nixos-is-not-reproducible/ https://linderud.dev/blog/nixos-is-not-reproducible/ [2] “non reproducible issues in NixOS” https://github.com/orgs/NixOS/projects/30 https://github.com/orgs/NixOS/projects/30
  - walterbell 2y ago
    Does Guix offer guarantees of build reproducibility?
  - 2y ago
moritonal 2y ago
Good place to tell people about CKD8S. It acts as an IaC translating TS to K3 Yalm greatly simplifying working in K8. Developers shouldn't be writing YAML.
- sgarland 2y ago
  Hot take: infrastructure shouldn't be created imperatively; you wind up with fun surprises too often. Hotter take: devs shouldn't be running infra at all, DevOps was a mistake. Return to specialized roles.
  - shepherdjerred 2y ago
    You don't have to use CDK8s in an imperative way. The benefit is you get _really_ good typing, which makes writing infra much easier than YAML.
- shepherdjerred 2y ago
  cdk8s* The project, afaik, is abandoned, but it does still work quite well. I use it for my homelab: https://github.com/shepherdjerred/servers/tree/main/cdk8s https://github.com/shepherdjerred/servers/tree/main/cdk8s
walterbell 2y ago
> I ran a poll on Mastodon to see what people wanted me to do. The results were overwhelmingly in favor of Rocky Linux. As an online "content creator", who am I to not give the people what they want? Truman would be proud! > the NAS. It has all of our media and backups on it. It runs Plex and a few other services, mostly managed by docker compose. Does the NAS run NixOS?
- xena 2y ago
  Yes, it runs NixOS and I am too cowardly to bother changing that any time soon. It's got everything on a giant ZFS array and most distros have poor ZFS support.
  - deadbunny 2y ago
    > and most distros have poor ZFS support. This is the complete opposite of my experience. For most "server" style distros (IE not arch/arch derivatives) you just install the zfs modules and forget about it. Ubuntu even has them pre baked on their kernel. Arch gets complicated because it's a rolling release so the zfs module supported kernel versions gets out of sync with the latest kernel version which can prevent system updates due to irresolvable requirements which can go on for days/weeks as they play catch-up with eachother but you can just install the lts kernel and mostly avoid that issue. For (mostly) every other distro they are a lot more coordinated in their releases so the zfs module will work with the newer kernel without any issues. Other than that I can't think of what poor support even means, once it's installed it works. You can even have zfs on root for most distros.
sgarland 2y ago
> What's not fine is how you prevent Ansible from running the same command over and over. You need to make a folder full of empty semaphore files that get touched when the command runs... > One of my patrons pointed out that I need to use Ansible conditionals in order to prevent these same commands from running over and over. Yes-ish. As I'm pretty sure OP figured out due to the pre-made roles comment, there exists a `community.general.dnf_config_manager` module that would handle this specific issue. As a general rule in Ansible, as ansible-lint [0] will tell you, is if you're using `ansible.builtin.{command, shell}`, there's a decent chance you're Doing It Wrong (TM). The biggest problem I have with Ansible (and I say this as someone who uses it extensively with Packer to build VM templates for Proxmox in my homelab) is that, like its underlying Python, there are a dozen ways to do anything. For example, if you wanted to perform a set of tasks depending on arbitrary conditions, you could: 0. Use `when` and rely on things like named host groups 1. Use `when` and rely on manually handling state throughout plays 2. Use handlers 3. Break down the tasks into logically-scoped roles, and manually call roles 4. Do #3, but rely on Ansible's role dependencies instead 5. Use semaphore files / `creates` as OP did 6. Probably something else [0]: https://github.com/ansible/ansible-lint https://github.com/ansible/ansible-lint
- notabee 2y ago
  It's true, the flexibility can be both a boon and a curse. There should be a little more "best practices" info out there that's not too prescriptive. It doesn't help that a lot of pre-made roles on Ansible Galaxy vary widely in style and quality. Certainly no one wants to inherit Ansible code that's nothing but shell and command modules, but sometimes those are crucial gap fillers when an idempotent module isn't available for the task or is missing needed functionality. And even then, specialized (as opposed to general use) modules are only idempotent within themselves and you still sometimes need to check and pass the state of things between tasks and stick that in a registered variable combined with conditionals if the multiple tasks are dependent on each other or require a specific ordering. I think a good generalized "best practice" is to keep those inter-task dependencies and conditionals to a minimum though. Small chunks or no chunks at all. It's always better to find a way to just run tasks independently with no knowledge of each other. The block module with "rescue" is useful for failing out a host gracefully if there's a bundle of finicky inter-dependent tasks that just have to run together though.
deleted 2y ago
[deleted]
sponaugle 2y ago
I really enjoy reading about what people do in their homelab. It has turned into platforms that encourages actual 'doing' in contrast to just 'thinking about it'. When I am interviewing to hire a DevOps or System Engineering person, I love it when they have an example of how they pushed their knowledge outside of their core job. Just because your company isn't doing something does not need to block you from trying it. There are some really neat things going on in the micro-homelab space as well - Using very small machines with very lower power footprint that opens up lots of possibilities. The N100 price/powerpoint has opened up more options along side the Raspberry PI and it friends. So cool! Then there is this guy: https://youtu.be/-b3t37SIyBs https://youtu.be/-b3t37SIyBs What does that nutjob run? Oh wait, I know! Keep on lab'ing and learning!
whazor 2y ago
For populating a homelab Kubernetes cluster, onedr0p has a very nice Flux template: https://github.com/onedr0p/cluster-template https://github.com/onedr0p/cluster-template

106 points by xena 2y ago

No. Real suffering in the homelab is getting N 20 year old servers and swapping parts between them to get N-M servers that work. I feel like the project will be successful if I get all the drives wiped and I am within site of that although I discovered the wiping was going to be a process of triage: some drives did not spin up, one drive took 14 hours to wipe whereas a normal drive would take about 30 minutes. My collaborator will use the bad drives for target practice for a black powder rifle that shoots round balls. Noisy fans take the "home" out of the homelab; the machines are 64 bit Intel but top out at 4GB. The latest version of Ubuntu installs fine but the desktop struggles. I think I'm going to install the desktop Ubuntu again just to see if I can watch YouTube with it but the plan now is install the server edition and give it to my collaborator to run an occasional minecraft server, which might free up a (much more powerful) i3 machine to watch videos from my Jellyfin server on the TV downstairs, something the XBOX ONE oddly can't handle. (No patent licenses for codecs if it's a game console?) At least I dug out the old VGA-supporting monitors out of mothballs so I'll be ready to play around with the RISC-V and eZ80 SBCs I have which are, at the very least, a lot quieter.

shrubble 2y ago
That is retrocomputing, not homelabbing. Look into the sub-$200 Intel N100 based systems with 16GB RAM.
- jjbinx007 2y ago
  I recently bought 2 of these and they are EXCELLENT! I can leave them running 24/7 without worrying about how much electricity they're using. The performance, flexibility and reliability far exceeds the Raspberry Pis that are confined the to cupboard and they're probably a fair bit faster than my old desktop PCs that I rarely switch on any more. I've gone uber-minimalist and only have NVME drives attached via USB-3. One's connected via ethernet, the other has a wifi connection. Personally I don't need any more and I've retired my old servers for now.
  - distances 2y ago
    > reliability far exceeds the Raspberry Pis In which ways? Most commenters say the exact opposite, that cheap N100 mini PCs are less reliable than Raspberry Pis. I'm just now trying to decide which way to go. Raspberry Pi 5 is definitely much much more interesting from the nerdy point of view, but would cost about the same as some cheap N100, for half the power. Though half the electricity usage too.
    - zamadatix 2y ago
      If you get away with a 1/2 GB RAM version I'd say the Pi 5 is worth it, otherwise cheap mini PCs are better. I've found mini PCs to be more reliable than the Pi's I've owned as well (including the 5), though a lot of that is mitigated if you go for an m.2 hat on the Pi. In general though, if you're thinking of using a Pi more like a regular computer there isn't really anything special about it that makes it worthwhile. Well, beyond "I get to tinker more" if you particularly like assembling the thing, picking out the exact case, finding your favorite m.2 hat, etc. Even if you enjoy doing all that by the time you're done you still end with a PC that is worse than one you could just buy but maybe with a claim of saying you saved $20 doing so. Where I like the Pi is use cases that don't quite fit with a normal mini PC like IP KVM.
      - distances 2y ago
        My use case is Home Assistant, Pi-hole and such things. I would go for the 8 GB model and the NVMe hat, ending up somewhere around 170-180€. So practically the same price as N100. With RPi I do like the tinkering aspect, an actual community, and lower power consumption -- I'd prefer passive cooling but I can't see a way to do that with the NVMe hat. With N100 I would get double CPU power, RAM and storage. But I don't actually need that extra power, and the concept of buying a cheap pre-built PC is just a lot less appealing than something more nerdy. So I'll probably end up getting the RPi just for the, err, shall we call emotional reasons.
Aurornis 2y ago
I think we need a different word for collecting a lot of old, underpowered computers and tinkering endlessly. I understand the attraction of playing with old, cheap hardware. However, hardware has come so far that it’s easy to build a 16-core server with a lightly used AMD consumer chip and 64-128GB of RAM for under $1000. It will have more power and use far less energy than these clusters of old machines that I see people assembling. > Noisy fans take the "home" out of the homelab; Again, a completely unnecessary thing to suffer. If the goal is a homelab. It’s really easy to make a near-silent PC with modern parts and cooling that will outperform an entire rack of 20 year old PCs. Even 10G switches that are quiet or fan less are common. I get it. It can be fun. But I don’t think this is homelabbing.

fallingsquirrel 2y ago

The CoreOS diversion was interesting to read. I've been daily driving CoreOS+i3 for the past year (I might be the only one in the world). I thought having a tiny immutable base OS would make the system easier to manage over time, but unfortunately that hasn't been the case. It's been an adventure but I'm ready to give up and switch to something more vanilla.

spicyusername 2y ago
I had the same experience. Immutable OS are great when you have a team of people to manage the tooling and process complexity of deploying and managing them, end users who only want to use software running on those servers, not the servers themselves, and you are bringing servers up and down all the time, then you actually reap the benefits of standardization and infrastructure as code. But when it's just you, you want to interact directly with the OS, and it's just one device, it's just foot guns all day.
rubenbe 2y ago
I've keep running Fedora coreos on my home server. My biggest issue with it, is that it is very cloud oriented and doesn't seem to allow to rerun the provisioning config on an already existing machine. This turns the thing again into a stateful pet instead of a "one cow cattle". Although I do very much like the rollback feature which has allowed me temporarily roll back an update a couple of times
- walterbell 2y ago
  > doesn't seem to allow to rerun the provisioning config on an already existing machine In theory, a generic existing machine could have been compromised by malware, in which case the configuration may not match the previously provisioned version. With OS launch integrity to guarantee absence of tampering, and prove that current=expected config+binaries, it could be feasible to rerun provisioning config.
xena 2y ago
I've wondered if that would be possible for a while, but I didn't imagine anyone would actually do that. What are the upsides and downsides of doing this? I'd love to read a writeup of how you did that and what you'll miss when you move away.

kkfx 2y ago

Well, a translation can be: a NixOS user tried some from the IT dark-modern age and find how much it suck, nothing extraordinary, unfortunately most people fails to even understand that some Big of IT have interest in crappy solutions because they allow commerce while good ones do not. Back then was the full-stack-virtualization era where a large flock of biped sheep say that VMWare "is the future, today" of course on x86 with the incredible overhead it have. For some times the psyco-PR-drug works, than paravirtualization start to be know with docker before, k8s after. Again the aforementioned flock run for the mania again ignoring the fact it's a nonsense. Ladies and gentlemen this solution have only ONE purpose sell pre-build crap for those who do no know how to work themselves, mostly the Dev part of DevOps who in mean have issues even deploying their own private personal desktop at home, not even a homelab. They do not care about the tremendous overhead, "hw it's relatively cheap and I'm rich enough to buy more", and they consider normal the abysmal level of complexity. So we see in-production docker images built by no-one-know who, perhaps with his/her ssh keys authorized "oups, I forgot to remove them" and so on. NixOS unfortunately have a terrible language, Guix System unfortunately it's mostly focused on HPC/academic usage instead of homelab/desktop usage, but that's the "modern IaC" since years. They are incredibly simpler and light than the modern crap pushed to sell pre-made images, offering "cloud services" of various kind and so on, yes Hetzner or OVH or Amazon do need to been able to sell so they do need these monsters, but users do need to AVOID them and re-learn how to use and deploy OSes and app, perhaps in a less archaic ways than the 80s style...

PhilipRoman 2y ago
>Big of IT have interest in crappy solutions because they allow commerce while good ones do not. This is really true, I don't want to go into specifics but I've witnessed a standards committee advertising how much "vendor added value" their product allows (which is fancy language for the standard being incomplete and unusable without extensions). Oh and don't forget arbitrary limits on capabilities for no good reason so that in a few years marketing teams can release a new version with a fancy nickname to keep the hype running.
- kkfx 2y ago
  The big issue is that we can't go much further that way. That's why for instance China industries advance while we regress. Interoperability and diversity are keys to success, cutting them means cutting the branch we are sit on... These days many developers do not even know how to deploy their apps behind some third party APIs, finding a sysadmin it extremely hard and most are just "homelab guys with a bit of experience", how can we build something with this holes under our own feet? A stupid example: photovoltaic is now relatively popular and that's clear the sole reasonable usage we have so far is self-consumption, we also have LFP storage still high priced (respect of China) but at a price tag sufficient to buy some storage at home like a giant UPS for a home, that's actually 400V batteries, most common inverters recharge directly from their MPPTs, actually the very same batteries are in most BEV. Well NO DAMN SINGLE VENDOR offer DC-to-DC direct charge for cars, simply nobody have apparently thought of that. We even have a standard https://www.iso.org/standard/77845.html https://www.iso.org/standard/77845.html but nobody seems to have implemented it commercially. We have a significant set of IoT appliance, only very few offer at least an open standard protocol for integration and most of them offer only ModBUS as an open standard protocol, something from the '70s, nice for certain usage also today but way too limited for many other possible usage. MQTT is a complicated option, still valid, very few implement it. Two more easy open options exists (Kafka, Matter) but nobody seems to implement them. We have a gazillion of VoIP solution almost no one is really simple and ready to deploy for anyone. Mumble/MurMur are the easiest for voice, but they are just chat, no calling ability, GNU SIPWitch is a lightweight SIP alternative, but it's far from being comfy for SOHO usage and Astersik/Yate are simply too complex for most users, similarly the first voice + screen sharing was in 1968 "The Mother of All Demos" we still lack a broad comfy screen sharing + voice solution, we even lack an IPv6 global per host to ease anything. Long story short we waste gazillion of resources to maintain immense pile of crap following the classic https://xkcd.com/2347/ https://xkcd.com/2347/ creating fragile monsters and struggling to go past this sorry state.

sunshine-o 2y ago

So in summary a former NixOS user now use a preconfigured OS dedicated to run one single platform (k8s) and still suffer for having to tinker with everything. It would have been fun or wise if she had gone back to Nix in the end.

xena 2y ago
I still use Nix to build docker images, part of this is to see how bad the rest of the industry really is. It's slightly worse than I imagined it would be.
srid 2y ago
> a former NixOS user Why did the author stop using NixOS? This is the first time I'm hearing about a veteran NixOS user giving up on it.
- shepherdjerred 2y ago
  It mostly seems due to some drama going on in the Nix community rather than some technical reason. https://xeiaso.net/blog/2024/much-ado-about-nothing/ https://xeiaso.net/blog/2024/much-ado-about-nothing/
  - xena 2y ago
    I would call the situation "a complete failure of management" instead of "drama".
    - shepherdjerred 2y ago
      I used "drama" because I'm completely detached from the Nix community, and I have no idea what's going on over there :)
  - matheusmoreira 2y ago
    Can't believe I'm witnessing another node/io.js thing in 2024.

khaki54 2y ago

[flagged]

chadsix 2y ago

This was a tremendous write-up. I appreciate the detail including your ingressd setup. I agree, though, that this is a pain. It's for this reason that we made Cloud Seeder [1] so you can have hands-free setup of your homelab and IPv6rs for painless ingress [2] </shameless> [1] https://github.com/ipv6rslimited/cloudseeder https://github.com/ipv6rslimited/cloudseeder [2] https://ipv6.rs https://ipv6.rs

xena 2y ago
I'd set up IPv6 on there, but the problem is that I use Flannel (which is IPv4 only) and my 8 gigabit fiber ISP only gives me IPv4 connectivity. I'll look into more details, but I've slightly given up on IPv6 for now. Maybe I'll set up Calico or something, but IPv6 seems to have been made artificially difficult by everything in the stack. I hate it.
- miyuru 2y ago
  Can you add AAAA records to cdn.xeiaso.net as well. It seems to have the same IPv4 from fly.io as the main domain, but you forgot to add it to the CDN subdomain.
  - xena 2y ago
    Uhhhh, I thought I did that! I'm gonna go fix that, sorry!
    - xena 2y ago
      Did that fix it?
- chadsix 2y ago
  > 8 gigabit fiber ISP only gives me IPv4 connectivity. Just like my ISP (but not 8 gbit!). Luckily, IPv6rs actually tunnels through IPv4 (or 6) and provides an IPv6 address. You don't need one to start! I don't know the ins and outs for flannel, but maybe you could setup IPv4 internally and use IPv6 (and an IPv4 Reverse Proxy) for the public internet? I agree, though, IPv6 on its own can be hard but thanks to WireGuard, tayga (NAT64), and nginx/caddy/etc. (reverse proxy), it's definitely quite usable!

shrubble 2y ago

My contrarian take is that eventually k8s will be recognized as the overcomplication it is; and better methods of managing less than 10,000 VMs will be researched and used.

xena 2y ago
I really hope that is the case too, but for now Kubernetes sucked all the oxygen out of the room for everything else :(
xav0989 2y ago
I switched my home lab to nomad, which I find much easier to wrangle, but we’ll see what happens with the IBM acquisition.
Aurornis 2y ago
People don’t use Kubernetes at home because it’s the easiest tool for the job. They use it in home labs because it’s a safe and easy environment to learn, practice, and explore. Most importantly: It’s a low-consequence environment. If you accidentally bring everything down at your homelab, the only person who suffers is you. You don’t get that degree of safety to explore and experiment when you’re using company resources.
- snapplebobapple 2y ago
  Incorrect sir. I am way more concerned about my wife and kids as end users than i am about any employees
  - bbkane 2y ago
    You're more concerned about your home Plex server going down than the service you run at work?
    - ffsm8 2y ago
      Doubtful. But things like mail, documents, file storage, etc might be another story. It depends on what the person in question is hosting, and how it relates to the life of the SO. There are a lot of options around, just take a look at the awesome-selfhosted repo on github

wint3rmute 2y ago

After running NixOS for 6+ months on my homelab and also re-using part of the configuration on my work machine, I feel the same way as Xe each time I'm interacting with a non-declarative OS. There's just no simple way to share configuration between machines or to automagically clean things up after making changes. Ansible feels like a thin layer of ice upon a deep ocean of the OS state, hiding in a multitude of non-tracked configuration files. It is simply not enough to build a layer of YAML around an OS which is imperative by nature. Unfortunately, I can see the downsides of NixOS as well, being radically different from what we usually expect in a Linux distribution, adopting it in a already established environment will no doubt be hard. Steep learning curve, idiosyncracies of the Nix language (although after reading parts of the Nix thesis[1], I find it much more understandable and deeply thought out), just explaining Nix to people who don't have much experience with the functional way of doing things, let alone taking the functional approach all the way to defining an entire operating system - all of this sounds like a tough barrier to cross. And yet, the desire to keep things reproducible and declarative (not to mention going back in time) persists once you've had the taste of NixOS. [1] https://edolstra.github.io/pubs/phd-thesis.pdf https://edolstra.github.io/pubs/phd-thesis.pdf

jt2190 2y ago
I’m picking this nit: > When is a build reproducible? > “A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.” > Neither Nix or NixOS gives you these guarantees. This really makes me question whether all of the quirkiness of Nix is worth it if it can’t actually “pay off” with true reproducibility. [1] “NixOS is not reproducible (2022) https://linderud.dev/blog/nixos-is-not-reproducible/ https://linderud.dev/blog/nixos-is-not-reproducible/ [2] “non reproducible issues in NixOS” https://github.com/orgs/NixOS/projects/30 https://github.com/orgs/NixOS/projects/30
- walterbell 2y ago
  Does Guix offer guarantees of build reproducibility?
- 2y ago

moritonal 2y ago

Good place to tell people about CKD8S. It acts as an IaC translating TS to K3 Yalm greatly simplifying working in K8. Developers shouldn't be writing YAML.

sgarland 2y ago
Hot take: infrastructure shouldn't be created imperatively; you wind up with fun surprises too often. Hotter take: devs shouldn't be running infra at all, DevOps was a mistake. Return to specialized roles.
- shepherdjerred 2y ago
  You don't have to use CDK8s in an imperative way. The benefit is you get _really_ good typing, which makes writing infra much easier than YAML.
shepherdjerred 2y ago
cdk8s* The project, afaik, is abandoned, but it does still work quite well. I use it for my homelab: https://github.com/shepherdjerred/servers/tree/main/cdk8s https://github.com/shepherdjerred/servers/tree/main/cdk8s

walterbell 2y ago

> I ran a poll on Mastodon to see what people wanted me to do. The results were overwhelmingly in favor of Rocky Linux. As an online "content creator", who am I to not give the people what they want? Truman would be proud! > the NAS. It has all of our media and backups on it. It runs Plex and a few other services, mostly managed by docker compose. Does the NAS run NixOS?

xena 2y ago
Yes, it runs NixOS and I am too cowardly to bother changing that any time soon. It's got everything on a giant ZFS array and most distros have poor ZFS support.
- deadbunny 2y ago
  > and most distros have poor ZFS support. This is the complete opposite of my experience. For most "server" style distros (IE not arch/arch derivatives) you just install the zfs modules and forget about it. Ubuntu even has them pre baked on their kernel. Arch gets complicated because it's a rolling release so the zfs module supported kernel versions gets out of sync with the latest kernel version which can prevent system updates due to irresolvable requirements which can go on for days/weeks as they play catch-up with eachother but you can just install the lts kernel and mostly avoid that issue. For (mostly) every other distro they are a lot more coordinated in their releases so the zfs module will work with the newer kernel without any issues. Other than that I can't think of what poor support even means, once it's installed it works. You can even have zfs on root for most distros.

sgarland 2y ago

> What's not fine is how you prevent Ansible from running the same command over and over. You need to make a folder full of empty semaphore files that get touched when the command runs... > One of my patrons pointed out that I need to use Ansible conditionals in order to prevent these same commands from running over and over. Yes-ish. As I'm pretty sure OP figured out due to the pre-made roles comment, there exists a `community.general.dnf_config_manager` module that would handle this specific issue. As a general rule in Ansible, as ansible-lint [0] will tell you, is if you're using `ansible.builtin.{command, shell}`, there's a decent chance you're Doing It Wrong (TM). The biggest problem I have with Ansible (and I say this as someone who uses it extensively with Packer to build VM templates for Proxmox in my homelab) is that, like its underlying Python, there are a dozen ways to do anything. For example, if you wanted to perform a set of tasks depending on arbitrary conditions, you could: 0. Use `when` and rely on things like named host groups 1. Use `when` and rely on manually handling state throughout plays 2. Use handlers 3. Break down the tasks into logically-scoped roles, and manually call roles 4. Do #3, but rely on Ansible's role dependencies instead 5. Use semaphore files / `creates` as OP did 6. Probably something else [0]: https://github.com/ansible/ansible-lint https://github.com/ansible/ansible-lint

notabee 2y ago
It's true, the flexibility can be both a boon and a curse. There should be a little more "best practices" info out there that's not too prescriptive. It doesn't help that a lot of pre-made roles on Ansible Galaxy vary widely in style and quality. Certainly no one wants to inherit Ansible code that's nothing but shell and command modules, but sometimes those are crucial gap fillers when an idempotent module isn't available for the task or is missing needed functionality. And even then, specialized (as opposed to general use) modules are only idempotent within themselves and you still sometimes need to check and pass the state of things between tasks and stick that in a registered variable combined with conditionals if the multiple tasks are dependent on each other or require a specific ordering. I think a good generalized "best practice" is to keep those inter-task dependencies and conditionals to a minimum though. Small chunks or no chunks at all. It's always better to find a way to just run tasks independently with no knowledge of each other. The block module with "rescue" is useful for failing out a host gracefully if there's a bundle of finicky inter-dependent tasks that just have to run together though.

deleted 2y ago

[deleted]

sponaugle 2y ago

I really enjoy reading about what people do in their homelab. It has turned into platforms that encourages actual 'doing' in contrast to just 'thinking about it'. When I am interviewing to hire a DevOps or System Engineering person, I love it when they have an example of how they pushed their knowledge outside of their core job. Just because your company isn't doing something does not need to block you from trying it. There are some really neat things going on in the micro-homelab space as well - Using very small machines with very lower power footprint that opens up lots of possibilities. The N100 price/powerpoint has opened up more options along side the Raspberry PI and it friends. So cool! Then there is this guy: https://youtu.be/-b3t37SIyBs https://youtu.be/-b3t37SIyBs What does that nutjob run? Oh wait, I know! Keep on lab'ing and learning!

whazor 2y ago

For populating a homelab Kubernetes cluster, onedr0p has a very nice Flux template: https://github.com/onedr0p/cluster-template https://github.com/onedr0p/cluster-template