6 ms·

Rebuilding My Homelab with Compose, Ruby, IPv6, and No Kubernetes

55 points by zrail 3mo ago

zrail 3mo ago
Hey, author here. This is a piece about moving away from kubernetes and toward something that I can actually maintain as a solo person who has a life outside of k9s. It's not really intended to be "anti-kubernetes", more like "kubernetes really is too hard for my purposes". IMO the best change that I've made has been to give deterministic IPv6 addresses to every container and then using those for ingress. I'm curious to hear where y'all think the line is between docker compose with Ruby glue and "Dear friend, you have built a Kubernetes".
- lloydatkinson 3mo ago
  Did you ever consider NixOS for your homelab?
  - zrail 3mo ago
    I haven't, mostly for personal reasons rather than technical ones.
- sierra1011 3mo ago
  For transparency, I work with K8s every day and run it with FluxCD as my homelab and have in various formats for a few years. Before that though, I had a single computer (a NUC if memory serves) with systemd running docker containers. Dead simple.
- evanphx 3mo ago
  The honest answer to your question about where the line is: it's when you start writing your own health checks, restart logic, and deployment orchestration on top of compose. That's the tipping point where you're maintaining infrastructure instead of just running apps. We built Miren for exactly this middle ground. Push to the cluster, it handles image building, TLS, and deployment on any Linux server. No Dockerfile, no compose, no Kubernetes. Autoscaling is on by default. It's not that Kubernetes is wrong, it's that there should be something between compose scripts and a full cluster. We're actively working on ways of deploying already ready services in docker files to the cluster, which would probably be what you're looking for initially unless the cluster you're doing a lot of development on the cluster.
  - henryteeare 3mo ago
    Why did you use AI to write this? EDIT: Only moderate snark intended.
ocd 3mo ago
I still don't know really what Kubernetes is for or why so many people outside specific environments are using it, but it's cool that you're using Ruby.
- LaurensBER 3mo ago
  Kubernetes makes complex things (e.g blue/green deployment, auto-scaling, failover) possible irrespective of the underlying cloud/hardware with a good and standardized API. It's absolutely overkill for small teams and homelabs (I run a cluster myself) but an absolute godsend if you do need the advanced functionality.
  - deleted 3mo ago
    [deleted]
  - randompartytime 3mo ago
    >Kubernetes makes complex things (e.g blue/green deployment, auto-scaling, failover) possible How?
  - pjmlp 3mo ago
    Somehow we were doing that with deployment scripts and VM management tooling before Kubernetes became a thing, and without having to deal with YAML spaghetti.
    - chuckadams 3mo ago
      Yes, and Kubernetes came around as another player in that ecosystem and became popular for a reason, largely so we didn't have to manage clusters with imperative non-idempotent scripts with no runtime introspection or self-healing. I've done light devops (lab scale, not enterprise) off and on since cfengine was a thing, and while I'm no fan of the explosion of YAML (there's a special place in hell for helm in particular for using text/template to generate yaml), I'll take the controller loop design any day over most of the alternatives. Just having a sane API alone is a godsend: you ever try scripting vSphere?
SMFloris 3mo ago
IMO, kubernetes is overkill for a small non-homogeneous home cluster. What I use and really recommend is using systemd +/- docker. It just becomes so darn simple. Do not go the compose route (that route is filled with sadness of the incomplete stacks because db container failed silently kind) - instead aim to decompose the compose files and write a separate systemd service file for each of them, you can then assign limits separately. I don't want to set anyone on the path ... but I use NixOs and this is so easy to do there.
- jbarberu 3mo ago
  As someone who one week ago switched from a Debian to NixOS setup, using docker compose, I'd be very interested in hearing more if you have any resources or tips to share. I was hoping to move over to running rootless containers, but so far my HA setup has proven to be a pita to get working.
- xienze 3mo ago
  > What I use and really recommend is using systemd +/- docker Even better, systemd+podman (=quadlets). Quadlets are great, you basically can declare a compose file as a systemd unit with all the good and bad that comes with that.
- maccard 3mo ago
  I disagree - I think k8s is absolutely the right choice at that level. We use ECS instead of k8s for small projects because the control plane is more than the actual service, but when you want container management, some sort of basic deployment management, service discovery, storage management and secret management, k8s is super simple to run and works pretty much out of the box. For a homelab, I'd expect you to want this. Building it all yourself may be fun (and that's a good reason to do it), but it's going to result in all the complexity with a lot of issues that k8s just solves for you.
  - zrail 3mo ago
    > I'd expect you to want this The thing is, I don't want any of that. At least, I don't want any of it to be dynamic. For my purposes the more I can determine at deploy time and make static the better. K8s is a stack of dozens-to-hundreds of control loops that are built around the idiosyncrasies of a particular distributed key-value store that is only moderately fit for purpose. This is all great fun if you want to learn about it (I did!) and/or if you get to be a user rather than an operator.
eqvinox 3mo ago
> Kubernetes is Too Hard. I built a system that I didn't actually know how to maintain without the time or energy necessary to dig myself out of trouble. Couldn't agree more. Unless your homelab's point is to learn Kubernetes, just keep it simple. Proxmox sounds good, or just QEMU, libvirt, lxc, Docker, podman, whatever. Install packages, not containers where possible. Shell scripts are fine where needed. If it works for you, that's it, end of discussion, don't spend time on "pretty" if it's not the thing you want to get into / enjoy / learn. (My "thing" is networking, I can assure you my homenet is beautiful. Couldn't give a rat's ass how & where my paperless is running tho. It runs. Done.)
- preisschild 3mo ago
  I enjoy using Kubernetes on bare metal nodes at home and took it over Proxmox, quite happy with it (not so happy with costs of hardware to expand the cluster though)
  - eqvinox 3mo ago
    If that works for you, great!
    - zrail 3mo ago
      Agree! There are a ton of people out there that are happily running k8s on their homelab and there is absolutely nothing wrong with that. I tried it and found it was just too much for me.
      - preisschild 3mo ago
        Tbf I do the exact same thing at work and love doing abstractions that "technically" makes maintainence easier. Kubernetes fortunately is extremely automatable.
MisterKent 3mo ago
Lot of kubernetes hate here, which is surprising. I run a little 3 node cluster and besides the hardware issues I had (long story), it has been rock solid and dead easy to setup. Talos + longhorn + fluxcd (optional), is super nice. And everything beyond that is additive and just works within the ecosystem. If anything, it helped keep my stuff alive during all the hardware issues a lot longer. I think like 5-6 years ago, kubernetes on baremetal was pretty painful. People should really give it another try, an LLM can probably set it up for you and fire off the docker compose to manifests in one shot. Or just follow the docs yourself, maybe a dozen commands to get a cluster running? All the enterprisey stuff makes it feel a lot more complex than it really is.
- dewey 3mo ago
  When you are only used to the Kubernetes at work I can understand how people dislike it. If you set it up yourself and start with minimal feature and not lots of annotations the config files become very simple and not more complicated than a docker compose file. It quickly realized that after just using the managed Kubernetes from Digital Ocean and deploying a side project there.
  - PunchyHamster 3mo ago
    Oh running stuff on k8s is great, but running k8s itself is just adding a lot of code to your stack and might be not so trivial to debug when something goes wrong. Then again tools to deploy it went a long way
    - dewey 3mo ago
      Agreed, personally I'd only do it through a hosted provider or maybe consider https://k3s.io https://k3s.io for a bit simpler setup. I'd also only do it if Kubernetes is something I'm already familiar with.
- pjmlp 3mo ago
  I got introduced to UNIX with Xenix, have used plenty of flavours including containers in HP-UX and Solaris before they became a thing in Linux, I have zero need to use Kubernetes at home. In fact, I also have zero needs for it at work directly, as I have become an advocate for serverless and managed runtimes, unless there is really a business need to control the whole infrastructure, including the Kubernetes cluster directly.
pjmlp 3mo ago
Most of the stuff hyperscalers use have no place in homelabs, unless you're training for a job application. So duh.
Havoc 3mo ago
Meanwhile I’m busy moving the other direction. More K8S. Main motivation is that I’ve got a lot of compute and memory but it’s spread across many smaller devices. Meaningfully leveraging that requires a way to coordinate… I do also have a classic Proxmox setup too though so can decide whether something should live in VM/LXC or k8s
wsdn 3mo ago
The deterministic IPv6 approach is fascinating. Have you run into any issues with ISP-provided IPv6 prefix delegation changing over time, or are you strictly using ULAs (Unique Local Addresses) internally to bypass that headache?
- zrail 3mo ago
  Hey thanks! That's my favorite recent addition. The host machines resolve a GUA from an ISP-provided prefix like normal. On top of that, I l33t'd my way to a fun ULA prefix. Each host's prefix, the stack network prefixes, and container addresses are deterministically derived from hashes of each component's name. As of today I haven't made any of the ULA stuff routable, although that was the original impetus for it. Clients see each host's GUA in DNS and generally route traffic through Caddy. Cross-host traffic is almost entirely through Tailscale. I run DNS refreshes at deploy time and hourly, but I haven't dealt with Comcast rotating my PD assignment yet. It's been stable for a couple years so I'm probably due.
pzmarzly 3mo ago
You might like https://uncloud.run/ https://uncloud.run/
pelasaco 3mo ago
I did the same move as you away from k8s to plain proxmox containers and VMs. Professionally i do work with k8s, and see the benefits of it (not always, but i see the use cases), but in my homelab it was consuming a lot of energy. Just dropping the whole k8s, made me save 1 kw energy when idle... I guess mainly because of the active API, shifting workload from different workloads and the whole machinery that happens behind the scenes..
- catdog 3mo ago
  On low power, mostly idle systems that's definitely a downside. K8s is always busy, probably largely because it's whole architecture is centered around reconciliation loops which contentiously compare the actual state to the desired state. It's not much but enough to prevent the CPU from entering its lowest power states.
  - chuckadams 3mo ago
    K8s uses watchers on etcd, not polling, so if there's no changes, there should be no activity. But just collecting metrics like CPU usage is probably a watcher on constantly changing data, like every second or so. So your control nodes are likely to stay warm most of the time, but there's also things like MicroK8s, which is made so you can stop the control plane and the worker containers keep running.
- trallnag 3mo ago
  I would have assumed that observability components (Prometheus, etc) dwarf the control plain when it comes to activity. At least with not extremely dynamic workloads. And you mention that it was due to shifting workloads. If the demands are static, nothing should shift in my experience?
  - pelasaco 2mo ago
    Yes, well I am not sure why, but my nodes, kept pushing load from node a, to node b, then back to a. Maybe updates, new certificates, or trying to optimize usage, etc..
hebetude 3mo ago
Unless you’re running multiple home servers, I can not reason any reason to use k8s. I’ll push any business to k8s but I would never bother with 1-3 home labs. Surprised this is even a hacker news topic of interest.
rtpg 3mo ago
I've recently reached for pyinfra of all things and found it straightforward enough... just an epsilon above a pile of bash scripts.
drusklo 3mo ago
The problem with using k8s on homelabs, is that a lot of the applications you would usually deploy, are not designed for it; having to manage a bunch of persistent volumes because most of your applications use sqlite is not very practical, and if the backend is sqlite, then you are probably running only one pod, so no real HA (if the pod goes down k8s will start a new one though), if you have to go through hoops to deploy an application that's not designed for it decreases its value. Having said that, I keep a k3s node running for learning purposes, and all my homemade apps live in k3s; it is nice to have the option to escalate my app from 1 to 100 running instances, in case I want to test something, with the press of a button.

55 points by zrail 3mo ago

zrail 3mo ago

Hey, author here. This is a piece about moving away from kubernetes and toward something that I can actually maintain as a solo person who has a life outside of k9s. It's not really intended to be "anti-kubernetes", more like "kubernetes really is too hard for my purposes". IMO the best change that I've made has been to give deterministic IPv6 addresses to every container and then using those for ingress. I'm curious to hear where y'all think the line is between docker compose with Ruby glue and "Dear friend, you have built a Kubernetes".

lloydatkinson 3mo ago
Did you ever consider NixOS for your homelab?
- zrail 3mo ago
  I haven't, mostly for personal reasons rather than technical ones.
sierra1011 3mo ago
For transparency, I work with K8s every day and run it with FluxCD as my homelab and have in various formats for a few years. Before that though, I had a single computer (a NUC if memory serves) with systemd running docker containers. Dead simple.
evanphx 3mo ago
The honest answer to your question about where the line is: it's when you start writing your own health checks, restart logic, and deployment orchestration on top of compose. That's the tipping point where you're maintaining infrastructure instead of just running apps. We built Miren for exactly this middle ground. Push to the cluster, it handles image building, TLS, and deployment on any Linux server. No Dockerfile, no compose, no Kubernetes. Autoscaling is on by default. It's not that Kubernetes is wrong, it's that there should be something between compose scripts and a full cluster. We're actively working on ways of deploying already ready services in docker files to the cluster, which would probably be what you're looking for initially unless the cluster you're doing a lot of development on the cluster.
- henryteeare 3mo ago
  Why did you use AI to write this? EDIT: Only moderate snark intended.

ocd 3mo ago

I still don't know really what Kubernetes is for or why so many people outside specific environments are using it, but it's cool that you're using Ruby.

LaurensBER 3mo ago
Kubernetes makes complex things (e.g blue/green deployment, auto-scaling, failover) possible irrespective of the underlying cloud/hardware with a good and standardized API. It's absolutely overkill for small teams and homelabs (I run a cluster myself) but an absolute godsend if you do need the advanced functionality.
- deleted 3mo ago
  [deleted]
- randompartytime 3mo ago
  >Kubernetes makes complex things (e.g blue/green deployment, auto-scaling, failover) possible How?
- pjmlp 3mo ago
  Somehow we were doing that with deployment scripts and VM management tooling before Kubernetes became a thing, and without having to deal with YAML spaghetti.
  - chuckadams 3mo ago
    Yes, and Kubernetes came around as another player in that ecosystem and became popular for a reason, largely so we didn't have to manage clusters with imperative non-idempotent scripts with no runtime introspection or self-healing. I've done light devops (lab scale, not enterprise) off and on since cfengine was a thing, and while I'm no fan of the explosion of YAML (there's a special place in hell for helm in particular for using text/template to generate yaml), I'll take the controller loop design any day over most of the alternatives. Just having a sane API alone is a godsend: you ever try scripting vSphere?

SMFloris 3mo ago

IMO, kubernetes is overkill for a small non-homogeneous home cluster. What I use and really recommend is using systemd +/- docker. It just becomes so darn simple. Do not go the compose route (that route is filled with sadness of the incomplete stacks because db container failed silently kind) - instead aim to decompose the compose files and write a separate systemd service file for each of them, you can then assign limits separately. I don't want to set anyone on the path ... but I use NixOs and this is so easy to do there.

jbarberu 3mo ago
As someone who one week ago switched from a Debian to NixOS setup, using docker compose, I'd be very interested in hearing more if you have any resources or tips to share. I was hoping to move over to running rootless containers, but so far my HA setup has proven to be a pita to get working.
xienze 3mo ago
> What I use and really recommend is using systemd +/- docker Even better, systemd+podman (=quadlets). Quadlets are great, you basically can declare a compose file as a systemd unit with all the good and bad that comes with that.
maccard 3mo ago
I disagree - I think k8s is absolutely the right choice at that level. We use ECS instead of k8s for small projects because the control plane is more than the actual service, but when you want container management, some sort of basic deployment management, service discovery, storage management and secret management, k8s is super simple to run and works pretty much out of the box. For a homelab, I'd expect you to want this. Building it all yourself may be fun (and that's a good reason to do it), but it's going to result in all the complexity with a lot of issues that k8s just solves for you.
- zrail 3mo ago
  > I'd expect you to want this The thing is, I don't want any of that. At least, I don't want any of it to be dynamic. For my purposes the more I can determine at deploy time and make static the better. K8s is a stack of dozens-to-hundreds of control loops that are built around the idiosyncrasies of a particular distributed key-value store that is only moderately fit for purpose. This is all great fun if you want to learn about it (I did!) and/or if you get to be a user rather than an operator.

eqvinox 3mo ago

> Kubernetes is Too Hard. I built a system that I didn't actually know how to maintain without the time or energy necessary to dig myself out of trouble. Couldn't agree more. Unless your homelab's point is to learn Kubernetes, just keep it simple. Proxmox sounds good, or just QEMU, libvirt, lxc, Docker, podman, whatever. Install packages, not containers where possible. Shell scripts are fine where needed. If it works for you, that's it, end of discussion, don't spend time on "pretty" if it's not the thing you want to get into / enjoy / learn. (My "thing" is networking, I can assure you my homenet is beautiful. Couldn't give a rat's ass how & where my paperless is running tho. It runs. Done.)

preisschild 3mo ago
I enjoy using Kubernetes on bare metal nodes at home and took it over Proxmox, quite happy with it (not so happy with costs of hardware to expand the cluster though)
- eqvinox 3mo ago
  If that works for you, great!
  - zrail 3mo ago
    Agree! There are a ton of people out there that are happily running k8s on their homelab and there is absolutely nothing wrong with that. I tried it and found it was just too much for me.
    - preisschild 3mo ago
      Tbf I do the exact same thing at work and love doing abstractions that "technically" makes maintainence easier. Kubernetes fortunately is extremely automatable.

MisterKent 3mo ago

Lot of kubernetes hate here, which is surprising. I run a little 3 node cluster and besides the hardware issues I had (long story), it has been rock solid and dead easy to setup. Talos + longhorn + fluxcd (optional), is super nice. And everything beyond that is additive and just works within the ecosystem. If anything, it helped keep my stuff alive during all the hardware issues a lot longer. I think like 5-6 years ago, kubernetes on baremetal was pretty painful. People should really give it another try, an LLM can probably set it up for you and fire off the docker compose to manifests in one shot. Or just follow the docs yourself, maybe a dozen commands to get a cluster running? All the enterprisey stuff makes it feel a lot more complex than it really is.

dewey 3mo ago
When you are only used to the Kubernetes at work I can understand how people dislike it. If you set it up yourself and start with minimal feature and not lots of annotations the config files become very simple and not more complicated than a docker compose file. It quickly realized that after just using the managed Kubernetes from Digital Ocean and deploying a side project there.
- PunchyHamster 3mo ago
  Oh running stuff on k8s is great, but running k8s itself is just adding a lot of code to your stack and might be not so trivial to debug when something goes wrong. Then again tools to deploy it went a long way
  - dewey 3mo ago
    Agreed, personally I'd only do it through a hosted provider or maybe consider https://k3s.io https://k3s.io for a bit simpler setup. I'd also only do it if Kubernetes is something I'm already familiar with.
pjmlp 3mo ago
I got introduced to UNIX with Xenix, have used plenty of flavours including containers in HP-UX and Solaris before they became a thing in Linux, I have zero need to use Kubernetes at home. In fact, I also have zero needs for it at work directly, as I have become an advocate for serverless and managed runtimes, unless there is really a business need to control the whole infrastructure, including the Kubernetes cluster directly.

pjmlp 3mo ago

Most of the stuff hyperscalers use have no place in homelabs, unless you're training for a job application. So duh.

Havoc 3mo ago

Meanwhile I’m busy moving the other direction. More K8S. Main motivation is that I’ve got a lot of compute and memory but it’s spread across many smaller devices. Meaningfully leveraging that requires a way to coordinate… I do also have a classic Proxmox setup too though so can decide whether something should live in VM/LXC or k8s

wsdn 3mo ago

The deterministic IPv6 approach is fascinating. Have you run into any issues with ISP-provided IPv6 prefix delegation changing over time, or are you strictly using ULAs (Unique Local Addresses) internally to bypass that headache?

zrail 3mo ago
Hey thanks! That's my favorite recent addition. The host machines resolve a GUA from an ISP-provided prefix like normal. On top of that, I l33t'd my way to a fun ULA prefix. Each host's prefix, the stack network prefixes, and container addresses are deterministically derived from hashes of each component's name. As of today I haven't made any of the ULA stuff routable, although that was the original impetus for it. Clients see each host's GUA in DNS and generally route traffic through Caddy. Cross-host traffic is almost entirely through Tailscale. I run DNS refreshes at deploy time and hourly, but I haven't dealt with Comcast rotating my PD assignment yet. It's been stable for a couple years so I'm probably due.

pzmarzly 3mo ago

You might like https://uncloud.run/ https://uncloud.run/

pelasaco 3mo ago

I did the same move as you away from k8s to plain proxmox containers and VMs. Professionally i do work with k8s, and see the benefits of it (not always, but i see the use cases), but in my homelab it was consuming a lot of energy. Just dropping the whole k8s, made me save 1 kw energy when idle... I guess mainly because of the active API, shifting workload from different workloads and the whole machinery that happens behind the scenes..

catdog 3mo ago
On low power, mostly idle systems that's definitely a downside. K8s is always busy, probably largely because it's whole architecture is centered around reconciliation loops which contentiously compare the actual state to the desired state. It's not much but enough to prevent the CPU from entering its lowest power states.
- chuckadams 3mo ago
  K8s uses watchers on etcd, not polling, so if there's no changes, there should be no activity. But just collecting metrics like CPU usage is probably a watcher on constantly changing data, like every second or so. So your control nodes are likely to stay warm most of the time, but there's also things like MicroK8s, which is made so you can stop the control plane and the worker containers keep running.
trallnag 3mo ago
I would have assumed that observability components (Prometheus, etc) dwarf the control plain when it comes to activity. At least with not extremely dynamic workloads. And you mention that it was due to shifting workloads. If the demands are static, nothing should shift in my experience?
- pelasaco 2mo ago
  Yes, well I am not sure why, but my nodes, kept pushing load from node a, to node b, then back to a. Maybe updates, new certificates, or trying to optimize usage, etc..

hebetude 3mo ago

Unless you’re running multiple home servers, I can not reason any reason to use k8s. I’ll push any business to k8s but I would never bother with 1-3 home labs. Surprised this is even a hacker news topic of interest.

rtpg 3mo ago

I've recently reached for pyinfra of all things and found it straightforward enough... just an epsilon above a pile of bash scripts.

drusklo 3mo ago

The problem with using k8s on homelabs, is that a lot of the applications you would usually deploy, are not designed for it; having to manage a bunch of persistent volumes because most of your applications use sqlite is not very practical, and if the backend is sqlite, then you are probably running only one pod, so no real HA (if the pod goes down k8s will start a new one though), if you have to go through hoops to deploy an application that's not designed for it decreases its value. Having said that, I keep a k3s node running for learning purposes, and all my homemade apps live in k3s; it is nice to have the option to escalate my app from 1 to 100 running instances, in case I want to test something, with the press of a button.