17 ms·

Managing my personnal servers in 2020 with K3s

412 points by shamallow 6y ago

rektide 6y ago
This is such a great write-up. I hope we continue to evolve the modern ops set-up on the metal, make it easy for folks to onboard into something that scales both big & high, and small & low. This is, imo, enormously good tech to learn, and I feel like many people are wasting their time learning "things that work for them" or that "aren't as complex" when those personal choices & investments will, in all due chance, not pay off elsewhere, will not be things other people are as likely to know or use or enjoy. And k3s is so easy to use, works so well, that I think many folks kind of cheat themselves out of a better experience when they pick something a little more legacy, like Docker-compose. Also noteably that this is just a first step, & we can get better & better at creating a wider system of services for the personal server from Kubernetes. For example I use the zalando postgres-operator, which lets me just ask for/apply a kubernetes object, and presto-chango, i have a postgres database with as many replicated instances as I want. The author here similarly enjoys having Let's Encrypt being ambient available. Managing more and more systems within Kubernetes will continue to scale the operational network effect of choosing tech like Kubernetes, tech that doesn't just run containers, but is an overaching cloud. Kubernetes is Desired State Management, a repository for all of your (small) cloud's state. I'd consider maybe replacing some of the hand-made Wireguard work done here with Kilo[1], which can either run as a Container Network Interface (CNI) plugin or can wrap your existing Kubernetes networking CNI provider (by default K3s uses Flannel). This will automate the process nicely, let you manage things like peers in Kubernetes easily. When the author connected the RPi to their existing cluster, that's exactly the sort of multi-cloud topology that Kilo is there to help you run & manage, and from inside of Kubernetes itself! Kilo rocks. Also worth noting that some of the latter half of this write-up is optional. Switching K3's own ingress out for nginx is the authors opinion, for example. You may/may not need a mail server. The write-up is pretty long; I think it's worth highlighting that the core of what's happening here, what others would need to do, is pretty short. I do enjoy that the author started the steps by running gpg & sops, to make keys to secure this all. This is pretty rigorous. It's good to see! I don't think all operators have to do this, but it showed that the author was taking it fairly seriously. For reference, I run a 3 node K3s cluster at home, a separate single K3s instance, and am planning on trying to convert my laptops & workstations over so that operationally I get the same kind of great observability & metrics & managability on them that I enjoy on the cluster. I'd like to cloud-nativize more of of my day to day computing experience, for consistency sake, & because I think uplifting many of the local things on my machine into pieces of a larger Cloud body of state will give me more flexibility & capabilities that I can enjoy playing with. I look forward to becoming less machine-centric, and more cross-machine cloud-centric. [1] https://github.com/squat/kilo https://github.com/squat/kilo
- e12e 6y ago
  > planning on trying to convert my laptops & workstations over Eh, ok? So set up a mesh vpn, like zerotier - when you close your notebook slack migrates to your workstation? (I know, you highlighted monitoring - but nothing stops you from running statsd or something on your laptop).
  - rektide 6y ago
    Kilo is easier to set up and integrates with the pattern of how I want to set up & run all my systems. I get WireGuard mesh "for free" because I run Kubernetes, thanks to Kilo. Your alternatives have no appeal, all involve manual work that I don't have to set up myself, because I am moving towards a better managed, better automated form of existence. Setting up little one off systems is an old dark terrible world. Same thing with monitoring. Yes, I can and do use ansible to go set up systemd daemons on my workstations, to run local prometheus & node-exporters. Today. But this is one of many things I have to go re-run every time I bring up a new node, new laptop. And if I change configuration, update my ansible scripts, then I have to go update all my nodes, go find each laptop, turn it on, re-run the script on it. If I want to add another admin to help me operate these systems, a sibling, a lover, they would have to go dig through my scripts to understand what I've done and how I've set things up in my environment. Kubernetes makes all of this not bad. It provides a centralized top down control of all my systems. If I had kubernetes, I add a node-exporter daemonset, and it'd be running, in a consistent fashion, on every node. If I need to change the configuration latter, I change the daemonset, and it changes on all my laptops, whenever they do boot up. Because it's Kubernetes, there's a practice, a standard way that other operators can see & understand & expect for how I have these concerns managed, it's easy for them to see what daemonsets are running, easy for them to see how they are configured, it'll all look bog standard to any operator. Nothing stops me from running statsd or whatever. But doing so is a pain in the ass to manage and maintain, and I can not bloody wait to be free of such rubbish unmanaged DIY computer-hugging. I want a better way of operating my pool of resources.
    - e12e 6y ago
      I'm still unconvinced "adding kubernetes" is likely to make simple setup simpler - but to each their own. But regarding kilo - doesn't sound like it makes much sense for laptop/road warrior? https://github.com/squat/kilo#step-4-ensure-nodes-have-public-ip https://github.com/squat/kilo#step-4-ensure-nodes-have-publi... > At least one node in each location must have an IP address that is routable from the other locations. If the locations are in different clouds or private networks, then this must be a public IP address. Makes it difficult to use you laptop from a coffee shop?
francis-io 6y ago
k3s looks like it removes some of the "moving parts" from Kubernetes, but for a single node setup, docker-compose might be simpler to manage.
- rektide 6y ago
  Docker-compose isn't going to help you with Let's Encrypt, you're going to need to keep resolving that problem with each app you have or find some other way to tackle it, because you've picked a way to deploy containers, and don't have any kind of centralized cloud system at your back. In my comments, I mention that the author could have used Kilo, which would have been a Kubernetes-native way to manage their WireGuard system, and to connect the Pi & their other systems to their existing K3S system. I agree that docker-compose might be simpler, but there's a very very limited realm of concerns that that will ever serve, where-as Kubernetes's / the Cloud Native ambition is to manage everything you would need in your cloud. Whatever you need, should, ideally, be managable within the same framework. DNS is another decent example, where Kubernetes will help you manage domain names, somewhat. Still work to be done there but there are some good starts. There's so many operators, all of which purport to let you manage these services in a "cloud native" way. We're still learning, getting better at it, but being able to manage all these thing semi-consistently, via the same tools, is a superpower. https://github.com/operator-framework/awesome-operators https://github.com/operator-framework/awesome-operators Also just the question of short term wins vs long term use. You will not use docker-compose at your job. More and more people are going to be using Kubernetes to manage a wider and wider variety of systems & services, making more and more capabilities managed by Kubernetes.
  - henryfjordan 6y ago
    If you run a single server, getting a Let's Encrypt cert is as easy as running a cron script. Then you just run a single instance of Nginx with the cert directory mounted as a volume. You will have to do a little extra work to maintain the nginx configuration to point to all your other containers, but it's generally just copy/pasting a block and changing the port to add a new service. Kubernetes is cool but the only reason to run it on a single instance is because you want to. Also docker-compose isn't used to host stuff in production very often but it is used to manage running local instances quite a bit. I wouldn't write it off as not worth learning.
an_opabinia 6y ago
A refreshing and straightforward, opinionated writeup. Cuts through the noise of the big cloud providers, who are ironically incentivized to keep things pretty complicated. Traefik isn't that complicated though, definitely worth learning.
- outworlder 6y ago
  > Traefik isn't that complicated though, definitely worth learning. Their documentation – specially for K8s – sucks ever since they bumped a major version. Lots of links to the old documentation that no longer work. Sometimes there's no documentation at all (middleware configuration on K8s, for instance).
  - StavrosK 6y ago
    Caddy inexplicably removed their v1 docs too. They provide a zipfile with the old docs site that you can download and browse locally, but I have no idea why they didn't make it browseable. I asked Matt Holt but got no response, so I deployed the docs myself: https://caddy-docs.netlify.app/v1/ https://caddy-docs.netlify.app/v1/
- GordonS 6y ago
  > Traefik isn't that complicated though, definitely worth learning. Hmm, not sure I fully agree here! I recently setup Traefik 2 for the first time, to front a bunch of containers. I didn't want to do anything particularly complex: expose services on 4 different ports, automatic Let's Encrypt certificates, strip a prefix for one of the services, enable the Traefik dashboard, and a few other bits and bobs. For context, I've configured nginx (and Apache before it) a few times over the years, and have an in-depth understanding of networking and HTTP. Still, I struggled to get things working - it was like whack-a-mole; I'd get something working, but something else would stop working! As someone else has noted, it's difficult to Google docs/samples, because Traefik 1.x is a thing, and is configured differently. I was configuring it in my Docker Compose file, and it's quite confusing - some stuff is added in the command line for the entrypoint, while other stuff is as labels. At some point I also realised that you can't quite configure quite everything in this way - some stuff, like TLS config, can only be done in files. The syntax for config in Compose files is pretty ugly too, and it's not always obvious - middleware config for example. I'm wondering if I might have made things easier for myself if I'd put all the config in YAML/TOML files instead, but I think the config for every other container was in the compose file, so it seemed more natural to do the same for Traefik. Anyway, next time I think I'll try config files, or maybe try Haproxy instead.
midrus 6y ago
LOL. I want to go back 10 years then. Or just use dokku. We've gone crazy guys.
- rektide 6y ago
  What seems crazy to me is that, a decade or two ago, * We had few standard ways of doing things. Everyone was cobbling together their own stack, figuring out their own ways to run a plethora of services needed to run & operate an internet connected box. * Each service or daemon needed to run your systems had it's own stand-alone management interfaces & systems. Nothing worked with anything else. There was no apiserver there to store the configurations you wanted. You had whatever scripts you wrote, checked in to source, then a big "do the scripts" button, and then you got a bunch of files written all over kingdom come to various hosts to do all the various tasks that supposedly would run your system, you hoped. * People with bare-metal had few tools to auto-scale or get resiliency for their systems. If you do have three machines, most operators managed them like pets, not cattle.
  - xorcist 6y ago
    Ten years ago was 2010. VMs had already taken over everything by then. Some were convinced that VM images would take over application distribution, because it solved all dependency problems. Others weren't convinced and pointed to problems with insight and integration. Resiliency or scaling wasn't any easier or harder than now. It's not 1985 we're talking about.
  - znpy 6y ago
    > * We had few standard ways of doing things. We still have few standard ways to do things. Yeah, docker. It's a thing. Then add kubernetes, you're really back to ten years ago. Unless you're running either stock GKE/EKS/AKS (thee different variants already), you don't really have a standard.
- tdeck 6y ago
  I actually still use dokku for my side projects and it works great. Very little learning curve, minimal headaches. Sure it doesn't scale out but that's not been a problem so far.
dsr_ 6y ago
"When you try to CRAM everything (mail, webserver, gitlab, pop3, imap, torrent, owncloud, munin, ...) into a single machine on Debian, you ultimately end-up activating unstable repository to get the latest version of packages and end-up with conflicting versions between softwares to the point that doing an apt-get update && apt-get upgrade is now your nemesis." This is not my experience. My main house server runs: mail: postfix, dovecot, clamav (SMTP, IMAP) web: nginx, certbot, pelican, smokeping, dokuwiki, ubooquity, rainloop, privoxy (personal pages, blog, traffic tracking, wiki, comic-book server, webmail, anti-ad proxy) git, postgresql, UPS monitoring, NTP, DNS, and DHCPd. Firewalling, more DNS and the other part of DHCPd failover is on the router. Package update is a breeze. The only time I waste the overhead of a virtual machine is when I'm testing out new configurations and don't want to break what I have. "just having the Kubernetes server components running add a 10% CPU on my Intel(R) Atom(TM) CPU C2338 @ 1.74GHz." Containerization is not a win here. Where's the second machine to fail over to?
- gavinray 6y ago
  Containerization and container orchestration platforms are only partly about scalability. The primary appeal for me is ease of deployment and reproducibility. This is why I develop everything in Docker Compose locally. Maybe the equivalent here would be something like Guix or Nix for declaratively writing the entire state of all the desired system packages and services + versions but honestly (without personal experience using these) they seem harder than containers.
  - tomberek 6y ago
    Nix/NixOS for this purpose is very nice.
  - ralmeida 6y ago
    Exactly! Other non-scalability concerns they address (specifically talking about Kubernetes here) is a primitive amount of monitoring/observability; no-downtime updates (rolling updates); liveness/readiness probes; primitive service discovery and load balancing; resiliency to any one single host failing (even if the total compute power could easily fit into a single bigger server).
bndw 6y ago
Great write up! I run a similar setup and documented the high level architecture here: https://bdw.to/personal-infrastructure.html https://bdw.to/personal-infrastructure.html
Sodman 6y ago
I love the k3s control-plane in the cloud and the raspberry pi worker nodes running from home (connected over VPN). Not a bad use for all of those raspberry pis we developers seem to accumulate over the years!
swsieber 6y ago
> When you try to CRAM everything (mail, webserver, gitlab, pop3, imap, torrent, owncloud, munin, ...) into a single machine on Debian, you ultimately end-up activating unstable repository to get the latest version of packages and end-up with conflicting versions between softwares to the point that doing an apt-get update && apt-get upgrade is now your nemesis. Has anyone here taken a look at bedrock linux? It lets you have multiple linux installations coexist and interoperate (different distros mainly, but probably different copies of debian is possible) I've been fascinated by it but never actually given it a try. From the bedrock linux introduction page: > Given someone already expended the effort to package the specific version of the specific piece of software a given user desires for one distro, the inability to use it with software from another distro seems wasteful. > Bedrock Linux provides a technical means to work around cross-distro compatibility limitations and, in many instances, resolve this limitation. https://bedrocklinux.org/introduction.html https://bedrocklinux.org/introduction.html
- ParadigmComplex 6y ago
  Bedrock Linux does indeed support workflows with multiple copies of Debian, and this does in practice often resolve the quoted concern of mixing parts from different Debian releases without having them step on each other problematically. Another use for having multiple Debian subsystems (what Bedrock calls "strata") is to serve as a low risk alternative to an in-place dist-upgrade: If anything doesn't work as desired in the new Debian release, just keep using the previous one until you get it working; once everything's migrated over, you've effectively dist-upgraded and can (if you wish) remove the old stratum. I'm Bedrock's founder and lead developer. If you have any questions about it I'm happy to answer them.
renewiltord 6y ago
Very cool. I use GKE with nginx as my ingress controller. The Google LB ingresses are too expensive for this sort of thing. Also appreciate the cert manager advice. Thank you!
twiclo 6y ago
I'll never understand people who run kubernetes, or even docker for that matter, on their personal servers
- k__ 6y ago
  I don't even understand 99% of the people who do it for their business workloads, but who am I to judge.
  - neurostimulant 6y ago
    Job security?
- yoyohello13 6y ago
  For fun?
  - k__ 6y ago
    The only answer I would accept! :D
- p_l 6y ago
  Amount of work I have to do on k3s vs amount of work I have do on "classic" setup... well, k3s is greatly simpler. And I can throw out some problematic distro idiosyncracies.
  - sercand 6y ago
    People are assuming k3s or managed Kubernetes (like GKE) is too complex or requires too much maintenance. I have bunch of applications running on raspberry pi and another ubuntu machine at home as classic setup and running GKE on production for our company. I can say that I spend more time on classic setup and simple things on Kubernetes like zero downtime deployment or auto cert with Let's Encrypt much harder on classic setup.
- deleted 6y ago
  [deleted]
candiddevmike 6y ago
I use Debian stable and systemd-nspawn, gives me the "virtual machine" experience (separate filesystem, init/systemd, network address, etc) via lightweight containers that are really easy to start, stop, and share files between. All managed by ansible. Once a month I bump versions, run ansible, and forget about it.
- packetlost 6y ago
  I really wanted to like systemd-nspawn, but ran into massive issues with poor to non-existent documentation, bugs (in particular with DNS between 'containers'), and usability issues. Also the inability to reasonably run non-systemd distros such as Alpine further killed my interest in it. Even distros like Ubuntu, which use systemd had to be modified to use the systemd network stack in order to function properly.
  - candiddevmike 6y ago
    Feel free to message me over email if you want some help (in my profile). I can share more details on my setup.
- GordonS 6y ago
  I haven't heard of nspawn, but it sounds pretty interesting - I will have a look myself of course, but are there any particular guides or references you can recommend?
jeromenerf 6y ago
Very nice post indeed, even though I chose years ago to stop at just running Debian stable, for personal stuff. Someone, maybe 10 years ago, said that at scale, one should treat servers like cattle not pets. At home, I feel it’s OK to treat your server as a pet. One self hosted server that does it all is not a 2020 anomaly, it’s just boring and effective :) I have spent a small part of my imposed 2020 free time simplifying my family digital cattle to a cow self hosted digital pets. I liberated myself both from gear and services I was maintaining for affective value or some professional distorsion; the openbsd systems I love but don’t use enough, the apple rmbp I never use, the VPSes for my personal services, ... donated, sold, resigned. It feels great.
- stevefan1999 6y ago
  why not a pet-cattle? you should be able to turn a pet into pet-cattle with ansible easily.
  - jeromenerf 6y ago
    I actually removed ansible from my setup as I got rid of redundancy. Good backup strategy and unattended-upgrades is all the automation I need. For my personal systems.
gregmac 6y ago
I recently rebuilt my home server from scratch, and after a bunch of research decided to run Proxmox [1] on it. I considered Kubernetes but two things really held me back: I've never installed/managed Kubernetes itself (and didn't really want to learn with my "production" home network), and I wanted to be able to run things other than Docker containers such as a Windows VM. I've used stuff like VMware and VirtualBox before and Proxmox is closer to those, but without the overhead of everything being a full VM. My old system was a "The Server" style setup as described here, running everything on a single Ubuntu bare metal machine (acting as a NAS, running Plex, Unifi, HomeAssistant, and a dozen other applications). It was getting old, had gone through some major OS upgrades and was about due for another. I used to do more "experimental" work on other old hardware I had laying around, but there's so much overhead involved that it often killed my motivation to learn something new. It's only been a couple months but I'm quite happy with Proxmox so far. I run most stuff in LXC containers with Alpine, though have at least one Ubuntu (Unifi Controller) and have played with some VMs (qemu): Hass-io [2] (which is designed for the Pi) and Windows 10 (just to try it). I actually made my Windows 10 image into a template, so I can quickly stand up an instance to try something out on Windows without hassle of reinstalling or worrying about breaking an actual PC. The LXC containers are very lightweight, and what I love is I can upgrade, experiment or swap out a service knowing I have good isolation from everything else -- just restore backup or revert to a snapshot. If I need to run the latest Debian Unstable to try some particular application out, I can do that without worry or having to setup another machine. Even porting everything over was very smooth, since I didn't have to worry about interaction between things or worry too much about permissions -- just install where it wants to go, whatever. (I used to try to keep my server setup in a consistent way, and it was often painful when some app had a different way of running or configuring itself or expected a different filesystem or user+group layout than I wanted). Now I just make symlinks in /root of each container and/or use Proxmox's "notes" feature to remind my future self where config files or other important things are located, and that's worked fine so far. What I can't attest to is what's involved in running it long-term: for example I haven't had to upgrade Proxmox itself, yet. On the plus side, it's removed an excuse preventing me from learning new things, and I donated/recycled a big pile of old hardware I had sitting in the basement ("just in case" I wanted to experiment on something). [1] https://www.proxmox.com/ https://www.proxmox.com/ [2] https://www.home-assistant.io/hassio/ https://www.home-assistant.io/hassio/
m0zg 6y ago
I consider this a form of procrastination, and I've been guilty of this myself. There are probably some unpleasant high-value tasks this person needs to be doing instead, so they convince themselves that this is high-value and do this because it's pleasant to them. You can actually weaponize this to do things that are "less unpleasant" but still worthwhile. Such tasks are much more palatable in the presence of another, more unpleasant task that you'd have to do instead. Story time: at Google the most unpleasant task employees have to do is writing their own (and their reports', if they have any) performance review, which results in a phenomenon called "perfcrastination", where a shit ton of work gets done just to postpone writing Perf until the last possible moment.
- outworlder 6y ago
  > Story time: at Google the most unpleasant task employees have to do is writing their own (and their reports', if they have any) performance review, which results in a phenomenon called "perfcrastination", where a shit ton of work gets done just to postpone writing Perf until the last possible moment. Oh, Google does 'self evaluations' too? That's reason enough to never even apply. That ranks among one of the worst ideas HR has ever had.
  - m0zg 6y ago
    Worse, you have to "apply for promotion" if you want to get promoted, and the decision is made by a remote committee of people who don't even know anything about what you did first hand - you have to beat your chest like an ape, and get endorsements from people a level or two up from where you are (which can be a big problem outside Mountain View, since there are fewer such people there). The whole process is utterly demeaning and demoralizing, especially if you're from cultures where beating one's chest is frowned upon (such as e.g. Russian). You routinely see people do ridiculously awesome work and then get turned down for promo. You also routinely see people promoted who just sat in the meetings with the right people and took the credit for "launching" stuff. Major part of the reason why I left.
  - allenu 6y ago
    Microsoft did that too. Now I'm at another company and we do it here as well. I'm not a fan of it, but having someone else write up your review sounds like a wonderful dream. The way I see it, you write your review, it goes up the management chain, they get to pick and choose what they agree with and now they have evidence since you admitted to doing it (or not, if it's missing in your self-eval), and then come back and justify why you get the bonus/raise or not using your own statements. Anyway, off-topic for this post.
Shorel 6y ago
From the K3s webpage: "We wanted an installation of Kubernetes that was half the size in terms of memory footprint. Kubernetes is a 10-letter word stylized as K8s. So something half as big as Kubernetes would be a 5-letter word stylized as K3s. There is no long form of K3s and no official pronunciation."
turing_complete 6y ago
I am curious, what are the benefits of running kubernetes on your personal servers? Seems like introducing unnecessary complexity.
- outworlder 6y ago
  You pay upfront for some more components. In exchange, you get a lot of benefits. 'classic' servers are only simpler if you can postpone dealing with their complexity.
  - beders 6y ago
    Classic servers are always simpler. Unless you need to manage a whole lot of them. It's like we suddenly have forgotten how to run several services on a single machine. ;)
  - kevindong 6y ago
    If you're running on a single-node machine and you don't NEED (or can't have it anyway since you're on a single machine/network/power "cluster") high availability, why bother with K8s over a much simpler vanilla Docker daemon that auto-restarts containers when they crash?
- mplewis 6y ago
  Kubernetes manages your containers and keeps them up at the scale you want. If you need to scale, you can. If your VPS goes down, you don't care, you just get a new one and plug it in. Your cluster was never down. If you want to move your cluster, you install k8s somewhere else and push the yaml.
  - beders 6y ago
    That is wishful thinking at best. If your software doesn't know what to do with the additional compute-power, nothing is going to happen. Your postgresql DB will not automagically scale. Your mail server won't. Your static web server either. Nor your backup software or image server, unless they were specifically made for that. And your bottleneck will still be the disk loading your actual data you want to serve. This should be considered an interesting case study but not a guide on how to actually build your own home servers. And "install k8s somewhere else and push the yaml" doesn't move your data.
imoverclocked 6y ago
"ssh ${HOST} '...; curl -sfL https://get.k3s.io https://get.k3s.io | sh -'" Why do we blindly run shell scripts from a website? I see this far more often than I should...
- wrkronmiller 6y ago
  As opposed to blindly running the precompiled binary you manually downloaded from the site?
  - bityard 6y ago
    You're right that from a purely security-oriented point of view there's not much difference. All code that you didn't write yourself can theoretically do anything it likes to your computer. (Unless sandboxed, of course.) But from a practical point of view in the context of convention, expectation, and trust, curlpipes and stand-alone binaries are totally different. Some of us have the battle scars of terrible shell-script installers gone wrong. At best, poorly-written shell script can deploy cruft to strange places on your disk or interfere with data and software managed by the OS package manager. At worst, such scripts have been known to destroy data and render the whole system a brick because the author was not sufficiently familiar with all of the systems that it might be run on. I don't remember the source now but one such installer effectively `rm -rf /` on the user's machine when a certain variable ended up undefined due to a bug elsewhere in the script. However, when I download a binary executable from a source that I trust, there's a certain expectation that the program will keep to itself and not interfere with the rest of my system or home directory beyond its own data and config files. Because _not_ doing so departs from convention and will result in a lot of angry users. Likewise I place a great deal of trust in the folks who package software for my OS because while mistakes can happen, their whole goal is a reliable and consistent system as a whole. Individual app developers care little about my system, they just want users to install their stuff. Curlpipes _look_ attractive to them ("just paste this into your terminal, type in your root password and you're ready to go!"), but we've seen many times over that cross-platform shell script installers are rarely their core competency.
    - 6y ago
gjulianm 6y ago
This is very well written, but also a marvel of overengineering in my opinion. It needs to - Set up a k3s cluster - Set up an ingress controller - Set up a certificate manager - Set up a Github actions pipeline to build and push Docker images All of that to deploy a bunch of standard applications in a single server, with very low load. So now, not only do you have to worry about the applications themselves and their configuration, but you also have to worry about all the layers on top of them. And the guide sounds good and easy when you read it, but I wonder how much time will you spend implementing it, working out issues and quirks and adding new things compared to the time you'd need doing a standard installation with a regular Ubuntu, Debian, Fedora or whatever system. And the resource overhead is important too, specially in disk space (60G in system disk for a home server sounds like a lot to me). I don't see the advantage here, honestly. I see a lot of moving pieces and complexity added and the only thing you seem to get out of it is an automated deployment, which is not very useful for a home server and also doable with simpler tools (you could even do it with a set of basic scripts that install software and copy configuration files). Maybe it's more secure due to isolation, but for a simple home server you can (and should) protect yourself against 99% of the attacks by using 2FA for SSH, a firewall with only the ports you want opened (and possibly restricted to certain IP ranges) plus basic auth + HTTPS on Nginx with fail2ban.
- threeseed 6y ago
  If you were just running it without k8s you would still need to generate SSL certificates and setup Nginx (aka ingress) if you wanted secured inbound traffic. Also CI/CD is completely optional.
  - gjulianm 6y ago
    I'd only need to get one certificate for Nginx and another for the SMTP server, which are two commands with Lets Encrypt. And setting up nginx as a reverse proxy for an application is as easy as adding two lines to a configuration file. It's far simpler, easier and better documented, and if any problems pop up it will also be easier to debug. You only need to know nginx to solve issues and add things: with the k3s setup it looks like you need to know nginx and Kubernetes.
dinkleberg 6y ago
I know it's popular to hate on K8s and containers as overkill for personal use and that they add tons of complexity. But for those of us who live and breathe Kubernetes every day professionally, it's honestly easier to do things this way. K3s gets you setup in minutes and then you can use kubectl + Helm to setup everything you want.
- bityard 6y ago
  Would you recommend k3s, kubectl, and helm for managing a single VPS? Asking for a friend...
  - rektide 6y ago
    The only downside is that k3s takes almost half a gig of ram. That does bump up the minimum VPS side a bit. It is overkill, but, in my mind, it is wonderful & greatly liberating overkill. It feels a little silly setting up a webserver the first time with a 45 line yaml file, but as you continue to make your way in the world, kubernetes is just such an amazingly pleasant top-down way of controlling systems, where it's so easy to see what you have, to see what is behaving and what isn't, and it's so easy to talk & share & communicate with other people about what you are doing versus trying to understand some handjammed nonsense a previous sysop scratched together 4 years ago. I'm rapidly planning out how to extend my Kubernetes life further. I've spent years writing Ansible scripts to automate setting up workstations, but I'd really like to switch over the bulk of the things I rely on to being more Kubernetes based. For example, I run prometheus's node-exporter to monitor the health of my systems. But if the config changes, or if I add a plugin, or if I start using a new laptop, I need to go re-run the script, and any other scripts. With Kubernetes, I can create a daemonset and it's fire and forget. It'll run on every node, and any changes I make will get applied to all nodes, whenever the node comes online or joins. Also my prometheus server wont need to be manually updated to point to these nodes when they get added. Quick note: K3s embed kubectl and helm in it; the install script symlinks helm and kubectl to the k3s binary. Also crictl for the container runtime tool. If you ask for k3s, you get these other things.
    - 6y ago
didip 6y ago
Does anyone have any articles highlighting the benefit of K3S? I am struggling to understand its pros and cons.
heavyset_go 6y ago
I'm curious about how much memory and CPU K3S uses on the manager and nodes in practice. I'm currently running Swarm at home because the overhead is minimal on managers and nodes when it comes to CPU and memory usage.
kevindong 6y ago
For my personal home server, I bought a Synology NAS which provides me with file sharing (via Samba), a built in VPN server, automatic RAID, easy dynamic DNS, Plex server, and support for Docker. Who needs Kubernetes (however stripped) down, when you're running a single-machine "cluster"? I write all of the services I want running as Docker containers and use a single docker-compose.yml to bring them up/down as needed. Specify `restart: unless-stopped` for each container and bam it's self healing too.
- justinlloyd 6y ago
  I understand the desire to tinker and learn. And still do a lot of it. But for my household infrastructure, I want it plug & play. A couple of Synology NAS, one for storage, one for backup & failover, a hardware firewall/reverse proxy/pihole container/wireguard container on a separate, hardened box. I sleep well at night and I don't have to explain to the wife why email/calendar/file sync/backup/notes/etc/etc isn't working today. I won't say it is for everyone, and if the OP is happy with his setup, so be it, but I wouldn't say it is a path that most should wander down if they place any kind of value on their personal time.
Zopieux 6y ago
I'm surprised no one mentioned using NixOS[0] for that kind of personal server thing, which is how I discovered it myself. Setting up unnecessary "control plane" daemons and services in an awful indent-nightmare Yaml DSL feels so clunky and error prone compared to writing a few hundred lines of Nix, which reproducibly builds your entire server image either live or on a boot medium. The result image can also be tested by launching a qemu VM with no additional code. The language itself provides some amount of syntactic validation. Thousands of fairly up-to-date packages[1] readily available and pre-compiled (but you can also build them if you prefer). Most sysadmin tooling already comes with specific, type-safe config parameters. Just override the defaults you don't like. Let's Encrypt support for an nginx vhost is a single "enableACME = true" line! While there are still some rough edges like secret sharing/storage, I'd encourage giving it a try before getting to the big, unnecessary guns. [0] https://nixos.org https://nixos.org [1] https://search.nixos.org/packages https://search.nixos.org/packages
mkj 6y ago
> Warning With everything installed, just having the Kubernetes server components running add a 10% CPU on my Intel(R) Atom(TM) CPU C2338 @ 1.74GHz Why is it that Kubernetes uses this much overhead? Surely it should be sleeping most of the time and occasionally reacting to events? Is the open source Go runtime less efficient than Google's internal one perhaps? (https://www.theregister.com/2020/08/10/google_scheduling_code_reaches_linux/ https://www.theregister.com/2020/08/10/google_scheduling_cod...)
- neurostimulant 6y ago
  Probably the old atom cpu. I got ~5% or less during idle on my vps server with k3s, and that's with various stuff running (nextcloud, postgres, postgis, mysql, pgadmin, phpmyadmin, wordpress, redis, memcache, several django apps + celery broker, etc). On the other hand, running kubernetes on mac immediately hog ~20% cpu, killing any hope for normal battery life.
- m3adow 6y ago
  I'm using my old home PC as server an K3s barely adds CPU load. I assume it's the 7 years old Atom processor. The memory memory footprint of K3s is recognizable on my 16 Gigs of RAM though.
mark242 6y ago
For those of you hitting ctrl-f "postfix": Postfix is very much not designed to run within a container. Unless you can absolutely 1000% guarantee that you will only ever have one instance of the postfix container running, you _will_ get data corruption and it is _very_ likely that you will lose email, because postfix does not support multiple instances sharing the same data/queue directory. A ReplicaSet, as given in the OP's yaml, does not suffice here because it is not guaranteed that the failing pod (which triggers a new pod to start) doesn't come back to life. That will almost guarantee your postfix installation to break especially as one postfix instance starts cleaning up files possibly during the time period where another postfix instance hitting the same directory is trying to access them. On edit: yes, Postfix 3.3 can be run as a container process. What I mean is running it within a standard container environment where you would expect multiple instances to run for failover etc.
hardwaresofton 6y ago
It's honestly not even that hard to run a full blown k8s cluster, I've been doing it for years, and even though I've made my share of mistakes along the way and downtime, I'm no longer scared of the k8s complexity boogeyman. Learn how it works from first principles, and the magic will disappear -- errors can still be daunting, but if you follow the concepts something normally jumps out at you. More than anything what gets people is the hypetrain and quickly moving libraries that just don't follow the "don't break user space" mantra these days. k8s is complex, but the complexity is for the most part essential. Just about every kubernetes concept maps to a thing you'd have to handle yourself if you ran one or more robust personal servers. It is a simple control loop, that you can run on only one machine -- one bit to check what should be running (etcd + apiserver), one bit to run the things (kubelet), and some bits to make sure they stay running (controller-manager and the other controllers). It is certainly more complex than Dokku[0] or Caprover[1] but that's because those are very focused on running applications -- Kubernetes aims to run more things, and for more use cases, but you don't have to know about all the use cases, just the ones that matter to you. All that said, I've been working on an orchestrator that simpler to operate than kubernetes, because I think kubernetes stumbled upon it's most powerful feature (operators) too late, and the answer to simplicity might actually be breaking kubernetes apart even more. [0]: https://github.com/dokku/dokku https://github.com/dokku/dokku [1]: https://github.com/CapRover/CapRover https://github.com/CapRover/CapRover
- GordonS 6y ago
  > All that said, I've been working on an orchestrator that simpler to operate than kubernetes Are you at a stage where you've published code yet? Curious to have a look!
  - hardwaresofton 6y ago
    Thanks for asking -- most of the ideas are in my head (I have a ton of long, scattered notes in Joplin), but I've started to transfer them to a repo. I'm starting with a repo of RFCs, which are really just me getting all my thoughts out there so they can be critiqued: https://gitlab.com/buhzaar/design https://gitlab.com/buhzaar/design I noted that I wanted it to be developed in the open and fretted a bit about whether I should finish the initial design first or not, but figured I should just put it out there, so people can look at it and comment if they want. The PoC to make sure this all works feels like a weekend of work (just because of how simple it is), but the upfront design is probably the hardest part. As you can see the design is so simple that the real work is defining the interfaces and leaving room for flexibility/protocol evolution.
sandGorgon 6y ago
we run k3s in production. its a much much nicer experience than even EKS. it integrates properly with spot instances, etc. I would pay money for k3s at this point.
lokimedes 6y ago
Just buy a Synology NAS or two and you have everything in a consistent system.
zelly 6y ago
The overhead of Kubernetes's runtime (Go) is nonzero. It's meant to run on Xenons which is going too far for a home server. You have the slow Go GC then you add on top of it a bunch of PHP and Ruby apps like Nextcloud, sprinkle a bit of unnecessary virtualization, and you've got a really slow machine that burns through more money on your electric bill than an equivalent EC2 would cost. Since everything I use at home is Linux, they're all "servers" already. They're all on a Wireguard network and share NFS mounts that I can access anywhere. No PHP, No Letsencrypt certificates, No Docker, No Kubernetes, No wasting electricity, No fiddling, No "git commiting every edit". My setup works just as well in a house with a static IP from an ISP as it could in a van with an antenna receiving 4G data.
j0057 6y ago
When I looked at the shutdown procedure for k3s I ran away in horror: it's basically firing off kill signals left and right and leaves all kinds of stuff laying around like bridges and temp state, making it hard to restart it if you don't want to reboot the server. The net result was the shutdown procedure for the server does not complete. As far as I can tell, this is a bit inherent to Kubernetes: it wants to be high-available and doesn't like shutting down the last node in the cluster, which is what a home server is. Edit to add: I'm having great fun with Podman instead. I don't need all the orchestration stuff or the autoscaling: one of each service is fine for a home server. Systemd can be the supervisor. I did a little fiddling to get service discovery with dynamic DNS, this could be improved. Using static routes on the router, I can route right into the right container network. This too could be improved: with IPv6 prefix delegation I could potentially route into the container from anywhere in the world.
fkin-hipsters 6y ago
On my home server I use OpenBSD as an orchestrator. It has cool features like processes and directories to isolate different things, it's pretty neat.
ohthehugemanate 6y ago
I have a very similar setup... but my k3s cluster is the graveyard for old devices. I love that my old devices are still in service, with services reasonably balanced between them. I live that each service I run has a simple configuration checked into a git repo. I love that when I want to run special workloads, I can add cloud based nodes. But if i was just running a single server I would have stuck with docker compose!
bucephale 6y ago
Adding home devices (rpi) to the remote cluster as k8s nodes through the use of wireguard seems strange to me as the the latency between nodes is supposed small within a k8s cluster. Do you think it will work ? Even after adding more of them ? It would be a nice feature.
justinholmes 6y ago
Truenas plus HA microk8s inside KVM VMs running in Mesos for myself.
parentheses 6y ago
k8s is all about automating resource allocation/topology among other things. this is a focus on what's outside of the scope of the processes that are part of the system. imo, personal servers rarely require this.
deleted 6y ago
[deleted]

412 points by shamallow 6y ago

This is such a great write-up. I hope we continue to evolve the modern ops set-up on the metal, make it easy for folks to onboard into something that scales both big & high, and small & low. This is, imo, enormously good tech to learn, and I feel like many people are wasting their time learning "things that work for them" or that "aren't as complex" when those personal choices & investments will, in all due chance, not pay off elsewhere, will not be things other people are as likely to know or use or enjoy. And k3s is so easy to use, works so well, that I think many folks kind of cheat themselves out of a better experience when they pick something a little more legacy, like Docker-compose. Also noteably that this is just a first step, & we can get better & better at creating a wider system of services for the personal server from Kubernetes. For example I use the zalando postgres-operator, which lets me just ask for/apply a kubernetes object, and presto-chango, i have a postgres database with as many replicated instances as I want. The author here similarly enjoys having Let's Encrypt being ambient available. Managing more and more systems within Kubernetes will continue to scale the operational network effect of choosing tech like Kubernetes, tech that doesn't just run containers, but is an overaching cloud. Kubernetes is Desired State Management, a repository for all of your (small) cloud's state. I'd consider maybe replacing some of the hand-made Wireguard work done here with Kilo[1], which can either run as a Container Network Interface (CNI) plugin or can wrap your existing Kubernetes networking CNI provider (by default K3s uses Flannel). This will automate the process nicely, let you manage things like peers in Kubernetes easily. When the author connected the RPi to their existing cluster, that's exactly the sort of multi-cloud topology that Kilo is there to help you run & manage, and from inside of Kubernetes itself! Kilo rocks. Also worth noting that some of the latter half of this write-up is optional. Switching K3's own ingress out for nginx is the authors opinion, for example. You may/may not need a mail server. The write-up is pretty long; I think it's worth highlighting that the core of what's happening here, what others would need to do, is pretty short. I do enjoy that the author started the steps by running gpg & sops, to make keys to secure this all. This is pretty rigorous. It's good to see! I don't think all operators have to do this, but it showed that the author was taking it fairly seriously. For reference, I run a 3 node K3s cluster at home, a separate single K3s instance, and am planning on trying to convert my laptops & workstations over so that operationally I get the same kind of great observability & metrics & managability on them that I enjoy on the cluster. I'd like to cloud-nativize more of of my day to day computing experience, for consistency sake, & because I think uplifting many of the local things on my machine into pieces of a larger Cloud body of state will give me more flexibility & capabilities that I can enjoy playing with. I look forward to becoming less machine-centric, and more cross-machine cloud-centric. [1] https://github.com/squat/kilo https://github.com/squat/kilo

e12e 6y ago
> planning on trying to convert my laptops & workstations over Eh, ok? So set up a mesh vpn, like zerotier - when you close your notebook slack migrates to your workstation? (I know, you highlighted monitoring - but nothing stops you from running statsd or something on your laptop).
- rektide 6y ago
  Kilo is easier to set up and integrates with the pattern of how I want to set up & run all my systems. I get WireGuard mesh "for free" because I run Kubernetes, thanks to Kilo. Your alternatives have no appeal, all involve manual work that I don't have to set up myself, because I am moving towards a better managed, better automated form of existence. Setting up little one off systems is an old dark terrible world. Same thing with monitoring. Yes, I can and do use ansible to go set up systemd daemons on my workstations, to run local prometheus & node-exporters. Today. But this is one of many things I have to go re-run every time I bring up a new node, new laptop. And if I change configuration, update my ansible scripts, then I have to go update all my nodes, go find each laptop, turn it on, re-run the script on it. If I want to add another admin to help me operate these systems, a sibling, a lover, they would have to go dig through my scripts to understand what I've done and how I've set things up in my environment. Kubernetes makes all of this not bad. It provides a centralized top down control of all my systems. If I had kubernetes, I add a node-exporter daemonset, and it'd be running, in a consistent fashion, on every node. If I need to change the configuration latter, I change the daemonset, and it changes on all my laptops, whenever they do boot up. Because it's Kubernetes, there's a practice, a standard way that other operators can see & understand & expect for how I have these concerns managed, it's easy for them to see what daemonsets are running, easy for them to see how they are configured, it'll all look bog standard to any operator. Nothing stops me from running statsd or whatever. But doing so is a pain in the ass to manage and maintain, and I can not bloody wait to be free of such rubbish unmanaged DIY computer-hugging. I want a better way of operating my pool of resources.
  - e12e 6y ago
    I'm still unconvinced "adding kubernetes" is likely to make simple setup simpler - but to each their own. But regarding kilo - doesn't sound like it makes much sense for laptop/road warrior? https://github.com/squat/kilo#step-4-ensure-nodes-have-public-ip https://github.com/squat/kilo#step-4-ensure-nodes-have-publi... > At least one node in each location must have an IP address that is routable from the other locations. If the locations are in different clouds or private networks, then this must be a public IP address. Makes it difficult to use you laptop from a coffee shop?

francis-io 6y ago

k3s looks like it removes some of the "moving parts" from Kubernetes, but for a single node setup, docker-compose might be simpler to manage.

rektide 6y ago
Docker-compose isn't going to help you with Let's Encrypt, you're going to need to keep resolving that problem with each app you have or find some other way to tackle it, because you've picked a way to deploy containers, and don't have any kind of centralized cloud system at your back. In my comments, I mention that the author could have used Kilo, which would have been a Kubernetes-native way to manage their WireGuard system, and to connect the Pi & their other systems to their existing K3S system. I agree that docker-compose might be simpler, but there's a very very limited realm of concerns that that will ever serve, where-as Kubernetes's / the Cloud Native ambition is to manage everything you would need in your cloud. Whatever you need, should, ideally, be managable within the same framework. DNS is another decent example, where Kubernetes will help you manage domain names, somewhat. Still work to be done there but there are some good starts. There's so many operators, all of which purport to let you manage these services in a "cloud native" way. We're still learning, getting better at it, but being able to manage all these thing semi-consistently, via the same tools, is a superpower. https://github.com/operator-framework/awesome-operators https://github.com/operator-framework/awesome-operators Also just the question of short term wins vs long term use. You will not use docker-compose at your job. More and more people are going to be using Kubernetes to manage a wider and wider variety of systems & services, making more and more capabilities managed by Kubernetes.
- henryfjordan 6y ago
  If you run a single server, getting a Let's Encrypt cert is as easy as running a cron script. Then you just run a single instance of Nginx with the cert directory mounted as a volume. You will have to do a little extra work to maintain the nginx configuration to point to all your other containers, but it's generally just copy/pasting a block and changing the port to add a new service. Kubernetes is cool but the only reason to run it on a single instance is because you want to. Also docker-compose isn't used to host stuff in production very often but it is used to manage running local instances quite a bit. I wouldn't write it off as not worth learning.

an_opabinia 6y ago

A refreshing and straightforward, opinionated writeup. Cuts through the noise of the big cloud providers, who are ironically incentivized to keep things pretty complicated. Traefik isn't that complicated though, definitely worth learning.

outworlder 6y ago
> Traefik isn't that complicated though, definitely worth learning. Their documentation – specially for K8s – sucks ever since they bumped a major version. Lots of links to the old documentation that no longer work. Sometimes there's no documentation at all (middleware configuration on K8s, for instance).
- StavrosK 6y ago
  Caddy inexplicably removed their v1 docs too. They provide a zipfile with the old docs site that you can download and browse locally, but I have no idea why they didn't make it browseable. I asked Matt Holt but got no response, so I deployed the docs myself: https://caddy-docs.netlify.app/v1/ https://caddy-docs.netlify.app/v1/
GordonS 6y ago
> Traefik isn't that complicated though, definitely worth learning. Hmm, not sure I fully agree here! I recently setup Traefik 2 for the first time, to front a bunch of containers. I didn't want to do anything particularly complex: expose services on 4 different ports, automatic Let's Encrypt certificates, strip a prefix for one of the services, enable the Traefik dashboard, and a few other bits and bobs. For context, I've configured nginx (and Apache before it) a few times over the years, and have an in-depth understanding of networking and HTTP. Still, I struggled to get things working - it was like whack-a-mole; I'd get something working, but something else would stop working! As someone else has noted, it's difficult to Google docs/samples, because Traefik 1.x is a thing, and is configured differently. I was configuring it in my Docker Compose file, and it's quite confusing - some stuff is added in the command line for the entrypoint, while other stuff is as labels. At some point I also realised that you can't quite configure quite everything in this way - some stuff, like TLS config, can only be done in files. The syntax for config in Compose files is pretty ugly too, and it's not always obvious - middleware config for example. I'm wondering if I might have made things easier for myself if I'd put all the config in YAML/TOML files instead, but I think the config for every other container was in the compose file, so it seemed more natural to do the same for Traefik. Anyway, next time I think I'll try config files, or maybe try Haproxy instead.

midrus 6y ago

LOL. I want to go back 10 years then. Or just use dokku. We've gone crazy guys.

rektide 6y ago
What seems crazy to me is that, a decade or two ago, * We had few standard ways of doing things. Everyone was cobbling together their own stack, figuring out their own ways to run a plethora of services needed to run & operate an internet connected box. * Each service or daemon needed to run your systems had it's own stand-alone management interfaces & systems. Nothing worked with anything else. There was no apiserver there to store the configurations you wanted. You had whatever scripts you wrote, checked in to source, then a big "do the scripts" button, and then you got a bunch of files written all over kingdom come to various hosts to do all the various tasks that supposedly would run your system, you hoped. * People with bare-metal had few tools to auto-scale or get resiliency for their systems. If you do have three machines, most operators managed them like pets, not cattle.
- xorcist 6y ago
  Ten years ago was 2010. VMs had already taken over everything by then. Some were convinced that VM images would take over application distribution, because it solved all dependency problems. Others weren't convinced and pointed to problems with insight and integration. Resiliency or scaling wasn't any easier or harder than now. It's not 1985 we're talking about.
- znpy 6y ago
  > * We had few standard ways of doing things. We still have few standard ways to do things. Yeah, docker. It's a thing. Then add kubernetes, you're really back to ten years ago. Unless you're running either stock GKE/EKS/AKS (thee different variants already), you don't really have a standard.
tdeck 6y ago
I actually still use dokku for my side projects and it works great. Very little learning curve, minimal headaches. Sure it doesn't scale out but that's not been a problem so far.

dsr_ 6y ago

"When you try to CRAM everything (mail, webserver, gitlab, pop3, imap, torrent, owncloud, munin, ...) into a single machine on Debian, you ultimately end-up activating unstable repository to get the latest version of packages and end-up with conflicting versions between softwares to the point that doing an apt-get update && apt-get upgrade is now your nemesis." This is not my experience. My main house server runs: mail: postfix, dovecot, clamav (SMTP, IMAP) web: nginx, certbot, pelican, smokeping, dokuwiki, ubooquity, rainloop, privoxy (personal pages, blog, traffic tracking, wiki, comic-book server, webmail, anti-ad proxy) git, postgresql, UPS monitoring, NTP, DNS, and DHCPd. Firewalling, more DNS and the other part of DHCPd failover is on the router. Package update is a breeze. The only time I waste the overhead of a virtual machine is when I'm testing out new configurations and don't want to break what I have. "just having the Kubernetes server components running add a 10% CPU on my Intel(R) Atom(TM) CPU C2338 @ 1.74GHz." Containerization is not a win here. Where's the second machine to fail over to?

gavinray 6y ago
Containerization and container orchestration platforms are only partly about scalability. The primary appeal for me is ease of deployment and reproducibility. This is why I develop everything in Docker Compose locally. Maybe the equivalent here would be something like Guix or Nix for declaratively writing the entire state of all the desired system packages and services + versions but honestly (without personal experience using these) they seem harder than containers.
- tomberek 6y ago
  Nix/NixOS for this purpose is very nice.
- ralmeida 6y ago
  Exactly! Other non-scalability concerns they address (specifically talking about Kubernetes here) is a primitive amount of monitoring/observability; no-downtime updates (rolling updates); liveness/readiness probes; primitive service discovery and load balancing; resiliency to any one single host failing (even if the total compute power could easily fit into a single bigger server).

bndw 6y ago

Great write up! I run a similar setup and documented the high level architecture here: https://bdw.to/personal-infrastructure.html https://bdw.to/personal-infrastructure.html

Sodman 6y ago

I love the k3s control-plane in the cloud and the raspberry pi worker nodes running from home (connected over VPN). Not a bad use for all of those raspberry pis we developers seem to accumulate over the years!

swsieber 6y ago

> When you try to CRAM everything (mail, webserver, gitlab, pop3, imap, torrent, owncloud, munin, ...) into a single machine on Debian, you ultimately end-up activating unstable repository to get the latest version of packages and end-up with conflicting versions between softwares to the point that doing an apt-get update && apt-get upgrade is now your nemesis. Has anyone here taken a look at bedrock linux? It lets you have multiple linux installations coexist and interoperate (different distros mainly, but probably different copies of debian is possible) I've been fascinated by it but never actually given it a try. From the bedrock linux introduction page: > Given someone already expended the effort to package the specific version of the specific piece of software a given user desires for one distro, the inability to use it with software from another distro seems wasteful. > Bedrock Linux provides a technical means to work around cross-distro compatibility limitations and, in many instances, resolve this limitation. https://bedrocklinux.org/introduction.html https://bedrocklinux.org/introduction.html

ParadigmComplex 6y ago
Bedrock Linux does indeed support workflows with multiple copies of Debian, and this does in practice often resolve the quoted concern of mixing parts from different Debian releases without having them step on each other problematically. Another use for having multiple Debian subsystems (what Bedrock calls "strata") is to serve as a low risk alternative to an in-place dist-upgrade: If anything doesn't work as desired in the new Debian release, just keep using the previous one until you get it working; once everything's migrated over, you've effectively dist-upgraded and can (if you wish) remove the old stratum. I'm Bedrock's founder and lead developer. If you have any questions about it I'm happy to answer them.

renewiltord 6y ago

Very cool. I use GKE with nginx as my ingress controller. The Google LB ingresses are too expensive for this sort of thing. Also appreciate the cert manager advice. Thank you!

twiclo 6y ago

I'll never understand people who run kubernetes, or even docker for that matter, on their personal servers

k__ 6y ago
I don't even understand 99% of the people who do it for their business workloads, but who am I to judge.
- neurostimulant 6y ago
  Job security?
yoyohello13 6y ago
For fun?
- k__ 6y ago
  The only answer I would accept! :D
p_l 6y ago
Amount of work I have to do on k3s vs amount of work I have do on "classic" setup... well, k3s is greatly simpler. And I can throw out some problematic distro idiosyncracies.
- sercand 6y ago
  People are assuming k3s or managed Kubernetes (like GKE) is too complex or requires too much maintenance. I have bunch of applications running on raspberry pi and another ubuntu machine at home as classic setup and running GKE on production for our company. I can say that I spend more time on classic setup and simple things on Kubernetes like zero downtime deployment or auto cert with Let's Encrypt much harder on classic setup.
deleted 6y ago
[deleted]

candiddevmike 6y ago

I use Debian stable and systemd-nspawn, gives me the "virtual machine" experience (separate filesystem, init/systemd, network address, etc) via lightweight containers that are really easy to start, stop, and share files between. All managed by ansible. Once a month I bump versions, run ansible, and forget about it.

packetlost 6y ago
I really wanted to like systemd-nspawn, but ran into massive issues with poor to non-existent documentation, bugs (in particular with DNS between 'containers'), and usability issues. Also the inability to reasonably run non-systemd distros such as Alpine further killed my interest in it. Even distros like Ubuntu, which use systemd had to be modified to use the systemd network stack in order to function properly.
- candiddevmike 6y ago
  Feel free to message me over email if you want some help (in my profile). I can share more details on my setup.
GordonS 6y ago
I haven't heard of nspawn, but it sounds pretty interesting - I will have a look myself of course, but are there any particular guides or references you can recommend?

jeromenerf 6y ago

Very nice post indeed, even though I chose years ago to stop at just running Debian stable, for personal stuff. Someone, maybe 10 years ago, said that at scale, one should treat servers like cattle not pets. At home, I feel it’s OK to treat your server as a pet. One self hosted server that does it all is not a 2020 anomaly, it’s just boring and effective :) I have spent a small part of my imposed 2020 free time simplifying my family digital cattle to a cow self hosted digital pets. I liberated myself both from gear and services I was maintaining for affective value or some professional distorsion; the openbsd systems I love but don’t use enough, the apple rmbp I never use, the VPSes for my personal services, ... donated, sold, resigned. It feels great.

stevefan1999 6y ago
why not a pet-cattle? you should be able to turn a pet into pet-cattle with ansible easily.
- jeromenerf 6y ago
  I actually removed ansible from my setup as I got rid of redundancy. Good backup strategy and unattended-upgrades is all the automation I need. For my personal systems.

gregmac 6y ago

I recently rebuilt my home server from scratch, and after a bunch of research decided to run Proxmox [1] on it. I considered Kubernetes but two things really held me back: I've never installed/managed Kubernetes itself (and didn't really want to learn with my "production" home network), and I wanted to be able to run things other than Docker containers such as a Windows VM. I've used stuff like VMware and VirtualBox before and Proxmox is closer to those, but without the overhead of everything being a full VM. My old system was a "The Server" style setup as described here, running everything on a single Ubuntu bare metal machine (acting as a NAS, running Plex, Unifi, HomeAssistant, and a dozen other applications). It was getting old, had gone through some major OS upgrades and was about due for another. I used to do more "experimental" work on other old hardware I had laying around, but there's so much overhead involved that it often killed my motivation to learn something new. It's only been a couple months but I'm quite happy with Proxmox so far. I run most stuff in LXC containers with Alpine, though have at least one Ubuntu (Unifi Controller) and have played with some VMs (qemu): Hass-io [2] (which is designed for the Pi) and Windows 10 (just to try it). I actually made my Windows 10 image into a template, so I can quickly stand up an instance to try something out on Windows without hassle of reinstalling or worrying about breaking an actual PC. The LXC containers are very lightweight, and what I love is I can upgrade, experiment or swap out a service knowing I have good isolation from everything else -- just restore backup or revert to a snapshot. If I need to run the latest Debian Unstable to try some particular application out, I can do that without worry or having to setup another machine. Even porting everything over was very smooth, since I didn't have to worry about interaction between things or worry too much about permissions -- just install where it wants to go, whatever. (I used to try to keep my server setup in a consistent way, and it was often painful when some app had a different way of running or configuring itself or expected a different filesystem or user+group layout than I wanted). Now I just make symlinks in /root of each container and/or use Proxmox's "notes" feature to remind my future self where config files or other important things are located, and that's worked fine so far. What I can't attest to is what's involved in running it long-term: for example I haven't had to upgrade Proxmox itself, yet. On the plus side, it's removed an excuse preventing me from learning new things, and I donated/recycled a big pile of old hardware I had sitting in the basement ("just in case" I wanted to experiment on something). [1] https://www.proxmox.com/ https://www.proxmox.com/ [2] https://www.home-assistant.io/hassio/ https://www.home-assistant.io/hassio/

m0zg 6y ago

I consider this a form of procrastination, and I've been guilty of this myself. There are probably some unpleasant high-value tasks this person needs to be doing instead, so they convince themselves that this is high-value and do this because it's pleasant to them. You can actually weaponize this to do things that are "less unpleasant" but still worthwhile. Such tasks are much more palatable in the presence of another, more unpleasant task that you'd have to do instead. Story time: at Google the most unpleasant task employees have to do is writing their own (and their reports', if they have any) performance review, which results in a phenomenon called "perfcrastination", where a shit ton of work gets done just to postpone writing Perf until the last possible moment.

outworlder 6y ago
> Story time: at Google the most unpleasant task employees have to do is writing their own (and their reports', if they have any) performance review, which results in a phenomenon called "perfcrastination", where a shit ton of work gets done just to postpone writing Perf until the last possible moment. Oh, Google does 'self evaluations' too? That's reason enough to never even apply. That ranks among one of the worst ideas HR has ever had.
- m0zg 6y ago
  Worse, you have to "apply for promotion" if you want to get promoted, and the decision is made by a remote committee of people who don't even know anything about what you did first hand - you have to beat your chest like an ape, and get endorsements from people a level or two up from where you are (which can be a big problem outside Mountain View, since there are fewer such people there). The whole process is utterly demeaning and demoralizing, especially if you're from cultures where beating one's chest is frowned upon (such as e.g. Russian). You routinely see people do ridiculously awesome work and then get turned down for promo. You also routinely see people promoted who just sat in the meetings with the right people and took the credit for "launching" stuff. Major part of the reason why I left.
- allenu 6y ago
  Microsoft did that too. Now I'm at another company and we do it here as well. I'm not a fan of it, but having someone else write up your review sounds like a wonderful dream. The way I see it, you write your review, it goes up the management chain, they get to pick and choose what they agree with and now they have evidence since you admitted to doing it (or not, if it's missing in your self-eval), and then come back and justify why you get the bonus/raise or not using your own statements. Anyway, off-topic for this post.

Shorel 6y ago

From the K3s webpage: "We wanted an installation of Kubernetes that was half the size in terms of memory footprint. Kubernetes is a 10-letter word stylized as K8s. So something half as big as Kubernetes would be a 5-letter word stylized as K3s. There is no long form of K3s and no official pronunciation."

turing_complete 6y ago

I am curious, what are the benefits of running kubernetes on your personal servers? Seems like introducing unnecessary complexity.

outworlder 6y ago
You pay upfront for some more components. In exchange, you get a lot of benefits. 'classic' servers are only simpler if you can postpone dealing with their complexity.
- beders 6y ago
  Classic servers are always simpler. Unless you need to manage a whole lot of them. It's like we suddenly have forgotten how to run several services on a single machine. ;)
- kevindong 6y ago
  If you're running on a single-node machine and you don't NEED (or can't have it anyway since you're on a single machine/network/power "cluster") high availability, why bother with K8s over a much simpler vanilla Docker daemon that auto-restarts containers when they crash?
mplewis 6y ago
Kubernetes manages your containers and keeps them up at the scale you want. If you need to scale, you can. If your VPS goes down, you don't care, you just get a new one and plug it in. Your cluster was never down. If you want to move your cluster, you install k8s somewhere else and push the yaml.
- beders 6y ago
  That is wishful thinking at best. If your software doesn't know what to do with the additional compute-power, nothing is going to happen. Your postgresql DB will not automagically scale. Your mail server won't. Your static web server either. Nor your backup software or image server, unless they were specifically made for that. And your bottleneck will still be the disk loading your actual data you want to serve. This should be considered an interesting case study but not a guide on how to actually build your own home servers. And "install k8s somewhere else and push the yaml" doesn't move your data.

imoverclocked 6y ago

"ssh ${HOST} '...; curl -sfL https://get.k3s.io https://get.k3s.io | sh -'" Why do we blindly run shell scripts from a website? I see this far more often than I should...

wrkronmiller 6y ago
As opposed to blindly running the precompiled binary you manually downloaded from the site?
- bityard 6y ago
  You're right that from a purely security-oriented point of view there's not much difference. All code that you didn't write yourself can theoretically do anything it likes to your computer. (Unless sandboxed, of course.) But from a practical point of view in the context of convention, expectation, and trust, curlpipes and stand-alone binaries are totally different. Some of us have the battle scars of terrible shell-script installers gone wrong. At best, poorly-written shell script can deploy cruft to strange places on your disk or interfere with data and software managed by the OS package manager. At worst, such scripts have been known to destroy data and render the whole system a brick because the author was not sufficiently familiar with all of the systems that it might be run on. I don't remember the source now but one such installer effectively `rm -rf /` on the user's machine when a certain variable ended up undefined due to a bug elsewhere in the script. However, when I download a binary executable from a source that I trust, there's a certain expectation that the program will keep to itself and not interfere with the rest of my system or home directory beyond its own data and config files. Because _not_ doing so departs from convention and will result in a lot of angry users. Likewise I place a great deal of trust in the folks who package software for my OS because while mistakes can happen, their whole goal is a reliable and consistent system as a whole. Individual app developers care little about my system, they just want users to install their stuff. Curlpipes _look_ attractive to them ("just paste this into your terminal, type in your root password and you're ready to go!"), but we've seen many times over that cross-platform shell script installers are rarely their core competency.
  - 6y ago

gjulianm 6y ago

This is very well written, but also a marvel of overengineering in my opinion. It needs to - Set up a k3s cluster - Set up an ingress controller - Set up a certificate manager - Set up a Github actions pipeline to build and push Docker images All of that to deploy a bunch of standard applications in a single server, with very low load. So now, not only do you have to worry about the applications themselves and their configuration, but you also have to worry about all the layers on top of them. And the guide sounds good and easy when you read it, but I wonder how much time will you spend implementing it, working out issues and quirks and adding new things compared to the time you'd need doing a standard installation with a regular Ubuntu, Debian, Fedora or whatever system. And the resource overhead is important too, specially in disk space (60G in system disk for a home server sounds like a lot to me). I don't see the advantage here, honestly. I see a lot of moving pieces and complexity added and the only thing you seem to get out of it is an automated deployment, which is not very useful for a home server and also doable with simpler tools (you could even do it with a set of basic scripts that install software and copy configuration files). Maybe it's more secure due to isolation, but for a simple home server you can (and should) protect yourself against 99% of the attacks by using 2FA for SSH, a firewall with only the ports you want opened (and possibly restricted to certain IP ranges) plus basic auth + HTTPS on Nginx with fail2ban.

threeseed 6y ago
If you were just running it without k8s you would still need to generate SSL certificates and setup Nginx (aka ingress) if you wanted secured inbound traffic. Also CI/CD is completely optional.
- gjulianm 6y ago
  I'd only need to get one certificate for Nginx and another for the SMTP server, which are two commands with Lets Encrypt. And setting up nginx as a reverse proxy for an application is as easy as adding two lines to a configuration file. It's far simpler, easier and better documented, and if any problems pop up it will also be easier to debug. You only need to know nginx to solve issues and add things: with the k3s setup it looks like you need to know nginx and Kubernetes.

dinkleberg 6y ago

I know it's popular to hate on K8s and containers as overkill for personal use and that they add tons of complexity. But for those of us who live and breathe Kubernetes every day professionally, it's honestly easier to do things this way. K3s gets you setup in minutes and then you can use kubectl + Helm to setup everything you want.

bityard 6y ago
Would you recommend k3s, kubectl, and helm for managing a single VPS? Asking for a friend...
- rektide 6y ago
  The only downside is that k3s takes almost half a gig of ram. That does bump up the minimum VPS side a bit. It is overkill, but, in my mind, it is wonderful & greatly liberating overkill. It feels a little silly setting up a webserver the first time with a 45 line yaml file, but as you continue to make your way in the world, kubernetes is just such an amazingly pleasant top-down way of controlling systems, where it's so easy to see what you have, to see what is behaving and what isn't, and it's so easy to talk & share & communicate with other people about what you are doing versus trying to understand some handjammed nonsense a previous sysop scratched together 4 years ago. I'm rapidly planning out how to extend my Kubernetes life further. I've spent years writing Ansible scripts to automate setting up workstations, but I'd really like to switch over the bulk of the things I rely on to being more Kubernetes based. For example, I run prometheus's node-exporter to monitor the health of my systems. But if the config changes, or if I add a plugin, or if I start using a new laptop, I need to go re-run the script, and any other scripts. With Kubernetes, I can create a daemonset and it's fire and forget. It'll run on every node, and any changes I make will get applied to all nodes, whenever the node comes online or joins. Also my prometheus server wont need to be manually updated to point to these nodes when they get added. Quick note: K3s embed kubectl and helm in it; the install script symlinks helm and kubectl to the k3s binary. Also crictl for the container runtime tool. If you ask for k3s, you get these other things.
  - 6y ago

didip 6y ago

Does anyone have any articles highlighting the benefit of K3S? I am struggling to understand its pros and cons.

heavyset_go 6y ago

I'm curious about how much memory and CPU K3S uses on the manager and nodes in practice. I'm currently running Swarm at home because the overhead is minimal on managers and nodes when it comes to CPU and memory usage.

kevindong 6y ago

For my personal home server, I bought a Synology NAS which provides me with file sharing (via Samba), a built in VPN server, automatic RAID, easy dynamic DNS, Plex server, and support for Docker. Who needs Kubernetes (however stripped) down, when you're running a single-machine "cluster"? I write all of the services I want running as Docker containers and use a single docker-compose.yml to bring them up/down as needed. Specify `restart: unless-stopped` for each container and bam it's self healing too.

justinlloyd 6y ago
I understand the desire to tinker and learn. And still do a lot of it. But for my household infrastructure, I want it plug & play. A couple of Synology NAS, one for storage, one for backup & failover, a hardware firewall/reverse proxy/pihole container/wireguard container on a separate, hardened box. I sleep well at night and I don't have to explain to the wife why email/calendar/file sync/backup/notes/etc/etc isn't working today. I won't say it is for everyone, and if the OP is happy with his setup, so be it, but I wouldn't say it is a path that most should wander down if they place any kind of value on their personal time.

Zopieux 6y ago

I'm surprised no one mentioned using NixOS[0] for that kind of personal server thing, which is how I discovered it myself. Setting up unnecessary "control plane" daemons and services in an awful indent-nightmare Yaml DSL feels so clunky and error prone compared to writing a few hundred lines of Nix, which reproducibly builds your entire server image either live or on a boot medium. The result image can also be tested by launching a qemu VM with no additional code. The language itself provides some amount of syntactic validation. Thousands of fairly up-to-date packages[1] readily available and pre-compiled (but you can also build them if you prefer). Most sysadmin tooling already comes with specific, type-safe config parameters. Just override the defaults you don't like. Let's Encrypt support for an nginx vhost is a single "enableACME = true" line! While there are still some rough edges like secret sharing/storage, I'd encourage giving it a try before getting to the big, unnecessary guns. [0] https://nixos.org https://nixos.org [1] https://search.nixos.org/packages https://search.nixos.org/packages

mkj 6y ago

> Warning With everything installed, just having the Kubernetes server components running add a 10% CPU on my Intel(R) Atom(TM) CPU C2338 @ 1.74GHz Why is it that Kubernetes uses this much overhead? Surely it should be sleeping most of the time and occasionally reacting to events? Is the open source Go runtime less efficient than Google's internal one perhaps? (https://www.theregister.com/2020/08/10/google_scheduling_code_reaches_linux/ https://www.theregister.com/2020/08/10/google_scheduling_cod...)

neurostimulant 6y ago
Probably the old atom cpu. I got ~5% or less during idle on my vps server with k3s, and that's with various stuff running (nextcloud, postgres, postgis, mysql, pgadmin, phpmyadmin, wordpress, redis, memcache, several django apps + celery broker, etc). On the other hand, running kubernetes on mac immediately hog ~20% cpu, killing any hope for normal battery life.
m3adow 6y ago
I'm using my old home PC as server an K3s barely adds CPU load. I assume it's the 7 years old Atom processor. The memory memory footprint of K3s is recognizable on my 16 Gigs of RAM though.

mark242 6y ago

For those of you hitting ctrl-f "postfix": Postfix is very much not designed to run within a container. Unless you can absolutely 1000% guarantee that you will only ever have one instance of the postfix container running, you _will_ get data corruption and it is _very_ likely that you will lose email, because postfix does not support multiple instances sharing the same data/queue directory. A ReplicaSet, as given in the OP's yaml, does not suffice here because it is not guaranteed that the failing pod (which triggers a new pod to start) doesn't come back to life. That will almost guarantee your postfix installation to break especially as one postfix instance starts cleaning up files possibly during the time period where another postfix instance hitting the same directory is trying to access them. On edit: yes, Postfix 3.3 can be run as a container process. What I mean is running it within a standard container environment where you would expect multiple instances to run for failover etc.

hardwaresofton 6y ago

It's honestly not even that hard to run a full blown k8s cluster, I've been doing it for years, and even though I've made my share of mistakes along the way and downtime, I'm no longer scared of the k8s complexity boogeyman. Learn how it works from first principles, and the magic will disappear -- errors can still be daunting, but if you follow the concepts something normally jumps out at you. More than anything what gets people is the hypetrain and quickly moving libraries that just don't follow the "don't break user space" mantra these days. k8s is complex, but the complexity is for the most part essential. Just about every kubernetes concept maps to a thing you'd have to handle yourself if you ran one or more robust personal servers. It is a simple control loop, that you can run on only one machine -- one bit to check what should be running (etcd + apiserver), one bit to run the things (kubelet), and some bits to make sure they stay running (controller-manager and the other controllers). It is certainly more complex than Dokku[0] or Caprover[1] but that's because those are very focused on running applications -- Kubernetes aims to run more things, and for more use cases, but you don't have to know about all the use cases, just the ones that matter to you. All that said, I've been working on an orchestrator that simpler to operate than kubernetes, because I think kubernetes stumbled upon it's most powerful feature (operators) too late, and the answer to simplicity might actually be breaking kubernetes apart even more. [0]: https://github.com/dokku/dokku https://github.com/dokku/dokku [1]: https://github.com/CapRover/CapRover https://github.com/CapRover/CapRover

GordonS 6y ago
> All that said, I've been working on an orchestrator that simpler to operate than kubernetes Are you at a stage where you've published code yet? Curious to have a look!
- hardwaresofton 6y ago
  Thanks for asking -- most of the ideas are in my head (I have a ton of long, scattered notes in Joplin), but I've started to transfer them to a repo. I'm starting with a repo of RFCs, which are really just me getting all my thoughts out there so they can be critiqued: https://gitlab.com/buhzaar/design https://gitlab.com/buhzaar/design I noted that I wanted it to be developed in the open and fretted a bit about whether I should finish the initial design first or not, but figured I should just put it out there, so people can look at it and comment if they want. The PoC to make sure this all works feels like a weekend of work (just because of how simple it is), but the upfront design is probably the hardest part. As you can see the design is so simple that the real work is defining the interfaces and leaving room for flexibility/protocol evolution.

sandGorgon 6y ago

we run k3s in production. its a much much nicer experience than even EKS. it integrates properly with spot instances, etc. I would pay money for k3s at this point.

lokimedes 6y ago

Just buy a Synology NAS or two and you have everything in a consistent system.

zelly 6y ago

The overhead of Kubernetes's runtime (Go) is nonzero. It's meant to run on Xenons which is going too far for a home server. You have the slow Go GC then you add on top of it a bunch of PHP and Ruby apps like Nextcloud, sprinkle a bit of unnecessary virtualization, and you've got a really slow machine that burns through more money on your electric bill than an equivalent EC2 would cost. Since everything I use at home is Linux, they're all "servers" already. They're all on a Wireguard network and share NFS mounts that I can access anywhere. No PHP, No Letsencrypt certificates, No Docker, No Kubernetes, No wasting electricity, No fiddling, No "git commiting every edit". My setup works just as well in a house with a static IP from an ISP as it could in a van with an antenna receiving 4G data.

j0057 6y ago

When I looked at the shutdown procedure for k3s I ran away in horror: it's basically firing off kill signals left and right and leaves all kinds of stuff laying around like bridges and temp state, making it hard to restart it if you don't want to reboot the server. The net result was the shutdown procedure for the server does not complete. As far as I can tell, this is a bit inherent to Kubernetes: it wants to be high-available and doesn't like shutting down the last node in the cluster, which is what a home server is. Edit to add: I'm having great fun with Podman instead. I don't need all the orchestration stuff or the autoscaling: one of each service is fine for a home server. Systemd can be the supervisor. I did a little fiddling to get service discovery with dynamic DNS, this could be improved. Using static routes on the router, I can route right into the right container network. This too could be improved: with IPv6 prefix delegation I could potentially route into the container from anywhere in the world.

fkin-hipsters 6y ago

On my home server I use OpenBSD as an orchestrator. It has cool features like processes and directories to isolate different things, it's pretty neat.

ohthehugemanate 6y ago

I have a very similar setup... but my k3s cluster is the graveyard for old devices. I love that my old devices are still in service, with services reasonably balanced between them. I live that each service I run has a simple configuration checked into a git repo. I love that when I want to run special workloads, I can add cloud based nodes. But if i was just running a single server I would have stuck with docker compose!

bucephale 6y ago

Adding home devices (rpi) to the remote cluster as k8s nodes through the use of wireguard seems strange to me as the the latency between nodes is supposed small within a k8s cluster. Do you think it will work ? Even after adding more of them ? It would be a nice feature.

justinholmes 6y ago

Truenas plus HA microk8s inside KVM VMs running in Mesos for myself.

parentheses 6y ago

k8s is all about automating resource allocation/topology among other things. this is a focus on what's outside of the scope of the processes that are part of the system. imo, personal servers rarely require this.

deleted 6y ago

[deleted]