10 ms·

This is very well written, but also a marvel of overengineering in my opinion. It needs to - Set up a k3s cluster - Set up an ingress controller - Set up a c

This is very well written, but also a marvel of overengineering in my opinion. It needs to - Set up a k3s cluster - Set up an ingress controller - Set up a certificate manager - Set up a Github actions pipeline to build and push Docker images All of that to deploy a bunch of standard applications in a single server, with very low load. So now, not only do you have to worry about the applications themselves and their configuration, but you also have to worry about all the layers on top of them. And the guide sounds good and easy when you read it, but I wonder how much time will you spend implementing it, working out issues and quirks and adding new things compared to the time you'd need doing a standard installation with a regular Ubuntu, Debian, Fedora or whatever system. And the resource overhead is important too, specially in disk space (60G in system disk for a home server sounds like a lot to me). I don't see the advantage here, honestly. I see a lot of moving pieces and complexity added and the only thing you seem to get out of it is an automated deployment, which is not very useful for a home server and also doable with simpler tools (you could even do it with a set of basic scripts that install software and copy configuration files). Maybe it's more secure due to isolation, but for a simple home server you can (and should) protect yourself against 99% of the attacks by using 2FA for SSH, a firewall with only the ports you want opened (and possibly restricted to certain IP ranges) plus basic auth + HTTPS on Nginx with fail2ban.

threeseed 6y ago
If you were just running it without k8s you would still need to generate SSL certificates and setup Nginx (aka ingress) if you wanted secured inbound traffic. Also CI/CD is completely optional.
- gjulianm 6y ago
  I'd only need to get one certificate for Nginx and another for the SMTP server, which are two commands with Lets Encrypt. And setting up nginx as a reverse proxy for an application is as easy as adding two lines to a configuration file. It's far simpler, easier and better documented, and if any problems pop up it will also be easier to debug. You only need to know nginx to solve issues and add things: with the k3s setup it looks like you need to know nginx and Kubernetes.
- mnkypete 6y ago
  Or just use caddy and get SSL included within seconds...
rewtraw 6y ago
Totally agree that this is impressive, but overkill. I've been running a linux NAS with a few services (transmission, sab, nzbget, the other usual suspects) for over a decade now, starting with on-host services and later migrating to docker+compose (which greatly simplified the bootstrapping process). Recently migrated to TrueNAS to get an even more appliance-like setup, and was very impressed in how quickly I was up and running from scratch. I can have NextCloud deployed in about 10 seconds if I want. Maybe I'm just at the point where I don't want to have to tinker/micromanage my server, but the idea of running kubernetes for a home server gives me anxiety.
- gjulianm 6y ago
  Offtopic question now that you mentioned running a Linux NAS... Which hardware are you using? Because I'm interested in building a home NAS but I don't seem to find any small box ready to hold several disk that isn't ridiculously expensive.
  - trm42 6y ago
    I have had Fractal Design's Node 304 as a home server / NAS for the last 3,5 years. It's quite reasonably priced, the provided fans are quiet enough and can be replaced later on with Noctuas. It doesn't offer disk hotswap but otherwise it's quiet and I can suffer the downtime ;-) + can handle 6 3,5 HDDs :) https://www.fractal-design.com/products/cases/node/node-304/black/ https://www.fractal-design.com/products/cases/node/node-304/...
  - rubenbe 6y ago
    I'm running a Dell T140. It's not the cheapest, but has idrac "remote control bios". So I never have to sit next to the box. It houses 4 disks + you can add SSDs via PC and it has ECC RAM. Plus I added a couple of years "Next business day on site" warranty, so I'm pretty safe against long downtimes.
  - fuzzy2 6y ago
    I recently bought a Chenbro SR30169 for my backup storage. It’s super compact but also quite noisy. Instead, I’d recommend getting a µATX tower with sound dampening preinstalled. The extra interior space also makes installation and maintenance a lot easier.
throwaway894345 6y ago
I’m working on building out the same thing right now (except I’m running it on a RPI cluster in my office), and the objective is my own amusement and enjoyment, (which naturally tends to lead me toward overengineering) but also to learn Kubernetes better.
swinges 6y ago
You make a lot of great points here. Almost all of the features you call out are actually provided by an open source project called Kalm (https://kalm.dev https://kalm.dev). Certs, ingress, cicd integration etc. all come out of the box
- chimen 6y ago
  "by an open source project called..." - claim it. It's your only submission on this portal anyway.
MuffinFlavored 6y ago
> All of that to deploy a bunch of standard applications in a single server usage: deploy_app "commit message, push right to master!" deploy_app () { git add --all :/ git commit -m "$1" git push ssh user@server <<EOF cd app/ && git reset --hard && git clean -df && git fetch && git reset --hard origin/master && sh scripts/deploy.sh EOF } deploy.sh #!/bin/sh set -e # java sh scripts/rebuild.sh java-api # node.js sh scripts/rebuild.sh api sh scripts/rebuild.sh cron sh scripts/rebuild.sh ui sh scripts/rebuild.sh windows-api sh scripts/rebuild.sh ws # terraform sh scripts/apply.sh # nginx docker restart nginx # sql sh scripts/migrate-database.sh Rebuilds Docker images, taints the Terraform Docker resources
- davedx 6y ago
  I have something very, very similar for a new line of business app I've just started building, except with no docker or terraform (I use docker to run a local postgres, that's it).
- dirkc 6y ago
  looks like you're using terraform to provision your docker containers locally? What is the benefit over using docker-compose? Or do you also provision other things?
neurostimulant 6y ago
IMO K3S is quite low maintenance and easy to setup. You just need to run a one liner to install and run the cluster. My only gripes is the internal certs only has 1 year expiration and to rotate it you'll have to restart the cluster a few months before the expiration date (it won't rotate the cert unless you restart it near the expiration date). It would be great if the internal cert has long expiration time (RKE has 10 year expiry by default) or provides manual control for rotating internal certs.
- nisa 6y ago
  until something breaks - then you are busy parsing megabytes of logs and jump from one vague github issue to another - to be fair I've attempted to install in into an lxd container but debugging this complexity monster is nightmare.
flak48 6y ago
Reminds me of simpler times not too long ago, when we used to run an entire startup by simply rsyncing files to production via Gitlab CI after tests passed after every commit.
- iudqnolq 6y ago
  I'm doing that right now, but I can't figure out how to avoid downtime
aequitas 6y ago
> All of that to deploy a bunch of standard applications in a single server, with very low load > So now, not only do you have to worry about the applications themselves and their configuration, but you also have to worry about all the layers on top of them. The biggest benefit of K3s here imho is not scaling or performance but to have a standardized API to deploy standardized packages/containers/deployments (Docker/Helm) to. So instead of configuring and maintaining one of the many Linux flavours (and all those layers) out there I have one standard system to worry about. I have a K3s Rpi cluster running for a few months now. Setup was trivial and maintenance as well. If I have a problematic node I just remove it from the cluster, reflash the SD card with a new install of K3os and put it back in. No state or further configuration to worry about. All my previous homelab setups where either hand-crafted snowflakes or configuration managed by some tool (Puppet, Salt or Ansible). Each comes with its own problems but they all have the problem that they accumulate state over time and become to hard to manage.
- gjulianm 6y ago
  But you have a cluster, which is already different from a single node. There I see how a layer to manage several nodes starts making sense. But when you're managing a single server, you already have a standardized API to deploy standardized packages (APT/YUM + SystemD probably). It's just a different one. Of course when you deviate from that it starts getting messy, but that happens with everything.
  - aequitas 6y ago
    I also have a single node 'cluster' for the more bulkier (diskio) stuff that won't fit the raspi's. And it's nice to have the same API accross all my setups. Apart from the data (backups, photos, media, etc) that is stored on the disk there is nothing of state worth saving on the node. If my root disk crashes, I just install K3s again and apply all configurations from the yaml files on my workstation and K3s pull everything back up as it was before. As you said Linux does offer standardized packages, but they are not applications/deployments. To get them running, beyond installing the binary, still requires a lot of configuration. Nginx (proxy, tls), a database maybe, storage/LVM, firewall, etc. So you quickly run into tools like Puppet and Ansible to manage this. The disadvantage is that they don't reverse the changes they make. So if you want to try something out and deploy it with Ansible, there is no easy trivial way to undo it except for reverting all the changes individually. Also there is always the tempation to quickly tweak something by hand, forgetting to commit it to CM. With a system like K8s, everything (container, volume, ingress) is a declarative configuration and the 'system' works to converge to the state you declared. If you delete something, K8s will revert all changes. So there will be no lingering state or configuration left. Making everything way more managable imho.
jbob2000 6y ago
It blows my mind that people set up personal stuff like this when something like "git push heroku master" does the same thing.
- mplewis 6y ago
  For the cost of one Heroku Standard 2X worker, you can get a DigitalOcean managed k8s cluster with 5x 2GB, 1CPU nodes, and that's capable of running basically all of your hobbyist traffic - not just a single service. Heroku is nice, but it's not the same thing.
  - jbob2000 6y ago
    For hobbyist traffic, you don’t need a standard 2X though.

threeseed 6y ago

If you were just running it without k8s you would still need to generate SSL certificates and setup Nginx (aka ingress) if you wanted secured inbound traffic. Also CI/CD is completely optional.

gjulianm 6y ago
I'd only need to get one certificate for Nginx and another for the SMTP server, which are two commands with Lets Encrypt. And setting up nginx as a reverse proxy for an application is as easy as adding two lines to a configuration file. It's far simpler, easier and better documented, and if any problems pop up it will also be easier to debug. You only need to know nginx to solve issues and add things: with the k3s setup it looks like you need to know nginx and Kubernetes.
mnkypete 6y ago
Or just use caddy and get SSL included within seconds...

rewtraw 6y ago

Totally agree that this is impressive, but overkill. I've been running a linux NAS with a few services (transmission, sab, nzbget, the other usual suspects) for over a decade now, starting with on-host services and later migrating to docker+compose (which greatly simplified the bootstrapping process). Recently migrated to TrueNAS to get an even more appliance-like setup, and was very impressed in how quickly I was up and running from scratch. I can have NextCloud deployed in about 10 seconds if I want. Maybe I'm just at the point where I don't want to have to tinker/micromanage my server, but the idea of running kubernetes for a home server gives me anxiety.

gjulianm 6y ago
Offtopic question now that you mentioned running a Linux NAS... Which hardware are you using? Because I'm interested in building a home NAS but I don't seem to find any small box ready to hold several disk that isn't ridiculously expensive.
- trm42 6y ago
  I have had Fractal Design's Node 304 as a home server / NAS for the last 3,5 years. It's quite reasonably priced, the provided fans are quiet enough and can be replaced later on with Noctuas. It doesn't offer disk hotswap but otherwise it's quiet and I can suffer the downtime ;-) + can handle 6 3,5 HDDs :) https://www.fractal-design.com/products/cases/node/node-304/black/ https://www.fractal-design.com/products/cases/node/node-304/...
- rubenbe 6y ago
  I'm running a Dell T140. It's not the cheapest, but has idrac "remote control bios". So I never have to sit next to the box. It houses 4 disks + you can add SSDs via PC and it has ECC RAM. Plus I added a couple of years "Next business day on site" warranty, so I'm pretty safe against long downtimes.
- fuzzy2 6y ago
  I recently bought a Chenbro SR30169 for my backup storage. It’s super compact but also quite noisy. Instead, I’d recommend getting a µATX tower with sound dampening preinstalled. The extra interior space also makes installation and maintenance a lot easier.

throwaway894345 6y ago

I’m working on building out the same thing right now (except I’m running it on a RPI cluster in my office), and the objective is my own amusement and enjoyment, (which naturally tends to lead me toward overengineering) but also to learn Kubernetes better.

swinges 6y ago

You make a lot of great points here. Almost all of the features you call out are actually provided by an open source project called Kalm (https://kalm.dev https://kalm.dev). Certs, ingress, cicd integration etc. all come out of the box

chimen 6y ago
"by an open source project called..." - claim it. It's your only submission on this portal anyway.

MuffinFlavored 6y ago

> All of that to deploy a bunch of standard applications in a single server usage: deploy_app "commit message, push right to master!" deploy_app () { git add --all :/ git commit -m "$1" git push ssh user@server <<EOF cd app/ && git reset --hard && git clean -df && git fetch && git reset --hard origin/master && sh scripts/deploy.sh EOF } deploy.sh #!/bin/sh set -e # java sh scripts/rebuild.sh java-api # node.js sh scripts/rebuild.sh api sh scripts/rebuild.sh cron sh scripts/rebuild.sh ui sh scripts/rebuild.sh windows-api sh scripts/rebuild.sh ws # terraform sh scripts/apply.sh # nginx docker restart nginx # sql sh scripts/migrate-database.sh Rebuilds Docker images, taints the Terraform Docker resources

davedx 6y ago
I have something very, very similar for a new line of business app I've just started building, except with no docker or terraform (I use docker to run a local postgres, that's it).
dirkc 6y ago
looks like you're using terraform to provision your docker containers locally? What is the benefit over using docker-compose? Or do you also provision other things?

neurostimulant 6y ago

IMO K3S is quite low maintenance and easy to setup. You just need to run a one liner to install and run the cluster. My only gripes is the internal certs only has 1 year expiration and to rotate it you'll have to restart the cluster a few months before the expiration date (it won't rotate the cert unless you restart it near the expiration date). It would be great if the internal cert has long expiration time (RKE has 10 year expiry by default) or provides manual control for rotating internal certs.

nisa 6y ago
until something breaks - then you are busy parsing megabytes of logs and jump from one vague github issue to another - to be fair I've attempted to install in into an lxd container but debugging this complexity monster is nightmare.

flak48 6y ago

Reminds me of simpler times not too long ago, when we used to run an entire startup by simply rsyncing files to production via Gitlab CI after tests passed after every commit.

iudqnolq 6y ago
I'm doing that right now, but I can't figure out how to avoid downtime

aequitas 6y ago

> All of that to deploy a bunch of standard applications in a single server, with very low load > So now, not only do you have to worry about the applications themselves and their configuration, but you also have to worry about all the layers on top of them. The biggest benefit of K3s here imho is not scaling or performance but to have a standardized API to deploy standardized packages/containers/deployments (Docker/Helm) to. So instead of configuring and maintaining one of the many Linux flavours (and all those layers) out there I have one standard system to worry about. I have a K3s Rpi cluster running for a few months now. Setup was trivial and maintenance as well. If I have a problematic node I just remove it from the cluster, reflash the SD card with a new install of K3os and put it back in. No state or further configuration to worry about. All my previous homelab setups where either hand-crafted snowflakes or configuration managed by some tool (Puppet, Salt or Ansible). Each comes with its own problems but they all have the problem that they accumulate state over time and become to hard to manage.

gjulianm 6y ago
But you have a cluster, which is already different from a single node. There I see how a layer to manage several nodes starts making sense. But when you're managing a single server, you already have a standardized API to deploy standardized packages (APT/YUM + SystemD probably). It's just a different one. Of course when you deviate from that it starts getting messy, but that happens with everything.
- aequitas 6y ago
  I also have a single node 'cluster' for the more bulkier (diskio) stuff that won't fit the raspi's. And it's nice to have the same API accross all my setups. Apart from the data (backups, photos, media, etc) that is stored on the disk there is nothing of state worth saving on the node. If my root disk crashes, I just install K3s again and apply all configurations from the yaml files on my workstation and K3s pull everything back up as it was before. As you said Linux does offer standardized packages, but they are not applications/deployments. To get them running, beyond installing the binary, still requires a lot of configuration. Nginx (proxy, tls), a database maybe, storage/LVM, firewall, etc. So you quickly run into tools like Puppet and Ansible to manage this. The disadvantage is that they don't reverse the changes they make. So if you want to try something out and deploy it with Ansible, there is no easy trivial way to undo it except for reverting all the changes individually. Also there is always the tempation to quickly tweak something by hand, forgetting to commit it to CM. With a system like K8s, everything (container, volume, ingress) is a declarative configuration and the 'system' works to converge to the state you declared. If you delete something, K8s will revert all changes. So there will be no lingering state or configuration left. Making everything way more managable imho.

jbob2000 6y ago

It blows my mind that people set up personal stuff like this when something like "git push heroku master" does the same thing.

mplewis 6y ago
For the cost of one Heroku Standard 2X worker, you can get a DigitalOcean managed k8s cluster with 5x 2GB, 1CPU nodes, and that's capable of running basically all of your hobbyist traffic - not just a single service. Heroku is nice, but it's not the same thing.
- jbob2000 6y ago
  For hobbyist traffic, you don’t need a standard 2X though.