13 ms·

K3s – Lightweight Kubernetes

254 points by kristianpaul 3y ago

ge96 3y ago
keees? kiots maybe
- cudder 3y ago
  Kubes?
  - sesm 3y ago
    Kyuss
- bantunes 3y ago
  "Kes" ?
- verdverm 3y ago
  > We wanted an installation of Kubernetes that was half the size in terms of memory footprint. Kubernetes is a 10 letter word stylized as k8s. So something half as big as Kubernetes would be a 5 letter word stylized as K3s. There is neither a long-form of K3s nor official pronunciation.
  - dheera 3y ago
    I always prefer to spell it kUbernEats.
  - JohnMakin 3y ago
    So wouldn't it be K4s? I can't figure out how they got 3.
    - maxbond 3y ago
      They're counting the "k" and the "s" to get to 5 total. It's a bit more convoluted than 4 = 8 / 2, it's 3 = (8 + 2) / 2 - 2.
    - worksonmine 3y ago
      Cut the 8 in half and you get 3, not mathematically but visually.
      - ge96 3y ago
        KԐS
    - verdverm 3y ago
      len(kubernetes) == 10 -> k8s half(kubernetes) == 5 -> k3s
- V99 3y ago
  The real origin of the name is kubernetes -> k[8 letters]s -> k8s -> people pronounce as "kates" -> k[3 letters]s -> k3s.
- deleted 3y ago
  [deleted]
rmelton 3y ago
K3s is fantastic especially for local development with Kubernetes when orchestrated using k3d. This is what we use for most of our internal K8s testing at OpenC3.
- doctoboggan 3y ago
  Yes, I recently moved my Dev team to K3D as well. Previously we were using AWS clusters that were spun up on demand every time someone needed to test something. the time needed to do this grew and grew overtime eventually coming close to 30 to 40 minutes. With k3d testing locally can be done in a minute. Plus I learned a ton about k8s while doing this.
mpsprd 3y ago
Can this tool help to simplify self hosting implems? K3s was recommended to me to replace my personal pile of systemd units starting docker compose configs and manual reverse proxy configs. Im am completely oblivious to how k8s works.
- proxysna 3y ago
  Start with hashicorp nomad. Less of a learning curve and easy to set up. Traefik can do automatic discovery of services in nomad. Definitely better than a bunch of systemd units.
- hhh 3y ago
  K3s/k8s may help with that, but it's going to be a learning curve. I personally moved onto k8s for a similar reason, but it was a learning curve
  - thelastparadise 3y ago
    It's a learning curve, but it is consistent, reliable, and standardized. Kubernetes has become an interface, independent of implementation. It's much like how POSIX is a standard and there are many implementations. And yes, POSIX has a learning curve too, but I'd rather learn that than anything proprietary, or non standard/rapidly changing.
    - moondev 3y ago
      No doubt. The ability to scale is great, but the best thing about Kubernetes is really the API.
- dewey 3y ago
  Check out https://kamal-deploy.org https://kamal-deploy.org, it just hit 1.0 and 37signals moved their whole Kubernetes stack to it. I was playing around with it recently for side projects and I think it's a nice fit for simpler products like that.
- thelastparadise 3y ago
imachine1980_ 3y ago
Why use this over kubernetes, if it's the same but less memory why not only use k3s ans drop k8s
- cogman10 3y ago
  There's no real "over kubernetes". K3s is a kubernetes implementation. You have to think of k8s as more of a specification of what can be done with multiple implementations. k3s, kind, microk8s, minikube are all implementations of kubernetes that you might want to try out. k8s is much like the original IBM pc or a linux distribution. There are a bunch of interchangeable parts that you may land on depending on your needs. Why use this over something else? I've found k3s to be really simple to setup and upgrade, which is why I use it for my home lab.
  - Takennickname 3y ago
    So all of those can be used to host a highly available multi physical node cluster?
    - cogman10 3y ago
      k3s and microk8s definitely support multiple node setup. I'm not sure about kind or minikube (Maybe? You'd have to look into the docs). How ready they are and how well they'll scale out will depend a lot on how the controllers are setup. k3s, at least, supports multiple controller nodes on setup. Where things really get squirrelly, at least with k3s, is storage and ingresses. k3s supports both but in a super robust fashion.
    - ianburrell 3y ago
      It makes sense to use full Kubernetes in that case. The lightweight distributions are good for single instances or limited multiple nodes. Like homelabs, edge nodes, or local development. But hosting application in data center makes sense to use full Kubernetes.
      - 3y ago
vault 3y ago
forget about it if you want to run it on an IPv6-only VPS, it's not ready yet
- thelastparadise 3y ago
  There's not any common reason I can think of to go ipv6 on the backend. KISS --if there's enough IPs in a private v4 range, use them. If you need/want to serve v6 clients, put a v6 load balancer in front. You're likely going to need that lb anyway.
  - yjftsjthsd-h 3y ago
    I think the use case was running on a cheap server that actually doesn't have ipv4 connectivity
    - vault 3y ago
      correct
  - dmm 3y ago
    > KISS ipv6 and routable ips is way simpler than patching layer 2 networks together.
- cwayne 3y ago
  K3s absolutely supports ipv6-only
  - vault 3y ago
    this tutorial doesn't work on an IPv6-only VPS https://www.jeffgeerling.com/blog/2022/quick-hello-world-http-deployment-testing-k3s-and-traefik https://www.jeffgeerling.com/blog/2022/quick-hello-world-htt...
rcarmo 3y ago
If anyone wants a ready-to-go Azure template to play around, here you go: https://github.com/rcarmo/azure-k3s-cluster https://github.com/rcarmo/azure-k3s-cluster (I tweak this every now and then since I both used it as a training sample for my customers/peers and as a way to run my own batch processes as cheap as possible)
diggan 3y ago
"Lightweight Kubernetes" and then a graph involving 2 different nodes with 10+ services running on them. Nomad seems to be a truly "lightweight Kubernetes" and for the small amount of time I've been using it, it seems to do it's job really well and it's easy to understand all the moving pieces without spending countless of hours reading docs and source code. Although, it's hard to recommend Nomad for future uses as it sadly stopped being FOSS :/
- q3k 3y ago
  I mean, 2 nodes and 10 services is very light for what Kubernetes is designed to scale to.
- worksonmine 3y ago
  The amount of services running is not an indication of how lightweight the underlying tech is. K3S is for orchestration across nodes, and by design you can't run services on the master for security reasons unless you manually change it. Are you trying to say that 10 services and 2 nodes would be less on Nomad?
  - tarruda 3y ago
    > K3S is for orchestration across nodes, and by design you can't run services on the master for security reasons unless you manually change it. That has not been my experience with k3s. Unlike normal distributions, k3s servers do allow workload to run on them unless you manually disable. From https://docs.k3s.io/datastore/ha-embedded https://docs.k3s.io/datastore/ha-embedded : " An HA K3s cluster with embedded etcd is composed of: - Three or more server nodes that will serve the Kubernetes API and run other control plane services, as well as host the embedded etcd datastore. - Optional: Zero or more agent nodes that are designated to run your apps and services " So you can have a full HA k3s cluster with only 3 server nodes and 0 agent(worker) nodes.
    - sleepybrett 3y ago
      This is not true of k8s in general, maybe certain on prem 'distributions' enforce this, i'm not sure. Sometimes kubernetes admins/operators who are running their own control planes and etcd will run etcd on non k8s controlled nodes, or they will run it on the same nodes that run their control plane services (scheduler, apiserver, controller manager, etc). Often those nodes will be tainted in such a way that they don't get overloaded by a noisy neighbor 'customer pod', but that's in no way 'by design'.. perhaps by convention or best practice.
inssein 3y ago
Loving K3S so far, I've got a "homelab" of 4 RPis running it and so far it has been pretty seamless experience.
- rigelina 3y ago
  This was my introduction to K3s as well. RPis running K3s with enough resources left over to actually do some small tasks. I hosted a small data pipeline that analyzed trading data from an MMO. It was as fun as it was impractical, and I learned quite a bit.
BossingAround 3y ago
Interestingly, SUSE now owns Rancher [0], so k3s has been backed by a large company for some time now. I've never tried k3s, but I have always thought it's probably the most loved-by-its-users version of Kubernetes. [0] https://www.suse.com/news/suse-completes-rancher-acquisition/ https://www.suse.com/news/suse-completes-rancher-acquisition...
- mfer 3y ago
  If you want to easily try it out, we baked it into Rancher Desktop [1]. RD let's you run Kubernetes (via k3s) and containers on your desktop. You pick the k8s version. [1] https://rancherdesktop.io/ https://rancherdesktop.io/ Disclosure, I work for SUSE and started Rancher Desktop.
  - willaaam 3y ago
    I just made an account here to say thank you. Rancher desktop is amazing on my M2 mac, it's easy to use, solves a bunch of challenges from k8s to docker and has been surprisingly reliable for a young product.
  - withinboredom 3y ago
    I've honestly been scared to switch from Docker Desktop to Rancher Desktop on Windows. Like do I have to switch or do they play nice together? Do I need to install Docker in WSL to use it, or kubectl, or is it like Docker Desktop where it "just works." So many questions...
    - mfer 3y ago
      You can have Docker Desktop and Rancher Desktop installed on the same Windows machine. Last I checked, you can't run them both at the same time due to docker socket locations (assuming you run Rancher Desktop using Moby). Rancher Desktop will provide the CLI tools to work with it. The idea is "it just works". It's a desktop app so it installs what you need.
  - cortesoft 3y ago
FourSigma 3y ago
Could someone please explain the difference between K0s[1] and K3s? They seem to both target the same minimalist K8s segment. [1]https://k0sproject.io/ https://k0sproject.io/
- r-bar 3y ago
  K3S includes some extras that make it nice for working in small local clusters, but are not part of the standard k8s codebase. * Traefik daemonset as a load balancer * Helm controller that lets you apply helm manifests without the helm command line * Upgrade controller * Sqlite as the default backing store for the k8s API * Their own local storage provisioner K0S has a lot of the same goals: be light weight and self contained in a single binary. But K0S tries to be as vanilla as possible. Choosing between the two it comes down to your use case. Do you want light weight and compatible (k0s), or lightweight and convenient (k3s)? Edit: formatting
  - FourSigma 3y ago
    Thank you! Great details. Definitely want convenient.
  - nullify88 3y ago
    What you've listed for k3s is mostly included in k0s. I wouldnt go far to say k0s isnt convenient. * A helm controller is included in k0s * Etcd is bundled and bootstrapped automatically which I perfer because I dont want the overhead of the translation that Kine does. Although Kine is available for a non-etcd datastore if that is preferred. * Upgrade controller is included (autopilot). * They have a local storage provider based on OpenEBS. * Ingress is missing, but due to the built in helm controller that can be boot strapped upon cluster initialisation. Overall, together with k0sctl and its declarative configuration it is easier to deploy k0s than it was k3s.
    - pas 3y ago
      Can you please elaborate on the "kine" overhead?
slig 3y ago
Anyone running K3S on Hetzner's new ARM64 servers?
onionisafruit 3y ago
K3s' go.mod[0] is insane. [0] https://github.com/k3s-io/k3s/blob/master/go.mod https://github.com/k3s-io/k3s/blob/master/go.mod
- sleepybrett 3y ago
  Every kubernetes project's go.mod is quite robust. Kubernetes is a very large codebase.
- pests 3y ago
  My only worry is the 10+ domains (involving 4+ major companies) being used for import paths. I know there are solutions but that would be a mess to figure out 10 years from now when half those links are broken or changed subtly.
sidcool 3y ago
We have been using K3S in production now for 2 years. And it's working like a charm.
j_m_b 3y ago
Is the overhead of Kubernetes worth it enough to make it smaller, presumably to run in less resource intensive environments? Docker compose fits this bill nicely and is much more simple.
- efrecon 3y ago
  Compose is for single node setups. K3s runs in multi-nodes clusters.
  - Xiol32 3y ago
    Also easier to scale out from your single starter k3s node as you grow.
- speedgoose 3y ago
  Kubernetes is offering much more features than Docker compose. And then you have all the kubernetes operators and other extensions that makes it very complex and powerful. It’s fine to use docker compose if it does what you need. If you start hacking weird things using Docker compose, it’s perhaps time to suffer and learn kubernetes.
betaby 3y ago
Too bad k3os was abandoned. And while k3s sounds easy, it's not after even a slightly larger scale. If one willing to have in-house k8s today I would recommend https://deckhouse.io/ https://deckhouse.io/ (I'm not affiliated with them)
- yjftsjthsd-h 3y ago
  > Too bad k3os was abandoned. Agreed:( Someday I'd like to see how hard it would be to use microos to build a replacement cheaply (cheap in terms of maintenance; ideally it would just be a combustion config or so that deployed k3s on vanilla microos) > And while k3s sounds easy, it's not after even a slightly larger scale. How does it fall apart?
  - betaby 3y ago
    > How does it fall apart? Agent nodes failing to join after reboots or long network partitioning. Manual re-joining works. Additionally, hard to debug issues with the supplied Traefic.
dmd 3y ago
I believe Truenas SCALE's plugin system is built on k3s.
worldsayshi 3y ago
vCluster is an interesting related technology. It spins up k3s inside Kubernetes. Handy if you want a lightweight way to manage multiple staging environments. https://www.vcluster.com/docs/what-are-virtual-clusters https://www.vcluster.com/docs/what-are-virtual-clusters
einstand 3y ago
My favorite presentation about K3s is when Darren Shepherd explains all the black magic (=all the custom patches) which made it possible: https://www.youtube.com/watch?v=k58WnbKmjdA https://www.youtube.com/watch?v=k58WnbKmjdA
- rcarmo 3y ago
  this is highly recommended.
praveenhm 3y ago
I've been closely following the discussion on k3s and Kubernetes in general. I recently acquired an M1 Mac Ultra, and I'm curious about the best options available for running Kubernetes locally on it. Does anyone have experience or recommendations in this area? I've heard about a few tools but wanted to gather insights from this knowledgeable community.
- maxekman 3y ago
  You should try out OrbStack: https://docs.orbstack.dev/kubernetes/ https://docs.orbstack.dev/kubernetes/ I switched to it completely, it’s very convenient to have both fast (-est on Mac) Docker support and a really smooth VM setup for running occasional Linux tools (such as Yocto in my case). Edit: added some background info to my recommendation.
  - praveenhm 3y ago
    Thanks for the orbstack recommendation, I am using it for docker containers, It is really fast and lightweight, I will try out Kubernetes.
- deleted 3y ago
  [deleted]
- nunez 3y ago
  If you're just messing around, just use kind (https://kind.sigs.k8s.io https://kind.sigs.k8s.io) or minikube if you want VMs (https://minikube.sigs.k8s.io https://minikube.sigs.k8s.io). Both work on ARM-based platforms. You can also use k3s; it's hella easy to get started with and it works great.
  - sklarsa 3y ago
    If you go the kind route, be sure to not use Docker for mac and instead opt for podman which is much more resource efficient. Now that I've switched over to podman, my computer doesn't sound like it's about to blast off when I'm running clusters.
nunez 3y ago
There are lighter Kubernetes "distributions" (kind, minikube), but what makes k3s special is that it's (a) packaged as two binaries that provide all of the Kubernetes components in one, and (b) it's 100% suitable for production use. Lots of teams are using K3s to run Kubernetes at the edge and in IoT applications, and with good reason. It's a fantastic Kubernetes distribution that's well-maintained, easy to get going with and well-documented. (Ironically, if you look at the first commits to kubernetes/kubernetes, the Kubernetes components were originally shipped as a single binary. They decided to break them up later to simplify releasing, but the k3s monolith lives on.)
nurettin 3y ago
I wish kubernetes could install and manage services through systemd.
- rcarmo 3y ago
  Well, have a look a this: https://github.com/virtual-kubelet/systemk https://github.com/virtual-kubelet/systemk
intangible 3y ago
I've been using a 3 nuc (actually Ryzen devices) k3s on SuSE MicroOS https://microos.opensuse.org/ https://microos.opensuse.org/ for my homelab for a while, and I really like it. They made some really nice decisions on which parts of k8s to trim down and which Networking / LB / Ingress to use. The option to use sqlite in place of etcd on an even lighter single node setup makes it super interesting for even lighter weight homelab container environment setups. I even use it with Longhorn https://longhorn.io/ https://longhorn.io/ for shared block storage on the mini cluster. If anyone uses it with MicroOS, just make sure you switch to kured https://kured.dev/ https://kured.dev/ for the transactional-updates reboot method. I'd love to compare it against Talos https://www.talos.dev/ https://www.talos.dev/ but Talos's lack of support for a persistent storage partition (only separate storage device https://github.com/siderolabs/talos/issues/4041 https://github.com/siderolabs/talos/issues/4041 ) really hurts most small home / office usage I'd want to try.
- organsnyder 3y ago
  Funny: I've been running a Talos cluster for the past six months, and just today decided to look into k3s. Talos has a lot of really nice things, but I have found that the lack of shell access can be frustrating at times when trying to troubleshoot.
  - davkan 3y ago
    Curious, what have you run into that you couldn’t troubleshoot with a privileged pod with host networking?
    - organsnyder 3y ago
      I had a situation where the etcd cluster got hosed, making it basically impossible (at least with the ways I know) to interact with the k8s API at all. So I didn't have any way to get a privileged pod running.
      - davkan 3y ago
samcat116 3y ago
K3s is great but I'll also shout out that RKE2 is almost as simple to install as K3s but its full Kubernetes.
maxthegeek1 3y ago
k3s is my preferred way to run k8s locally, for the purpose of developing helm charts locally. It's way more lightweight than minikube, and with k3d it's way easier to use as well.
bfrog 3y ago
Lightweight is a misnomer. Running even the most basic of tasks requires an insanely complex setup with vxlan, unintelligible iptables and bridge rules, all taking up a great deal of resources on something like an embedded x86/arm devices
nullify88 3y ago
If you want something lighter than k3s, k0s could be worth a try. I recently switched to k0s and noticed the controller, and the various kube processes consume less cpu resources compared to when I was running k3s.
umvi 3y ago
Kubernetes seems like an "acquired taste". Either you use it a lot (likely at work) and are therefore comfortable using it outside of work as your one-size-fits-all solution to everything, or else you don't use it a lot and see it as a complexity behemoth that hides all of your system behind esoteric CLIs and config files that just make everything a nightmare to manage.
- sleepybrett 3y ago
  I find it very acceptable on my local servers that run my mail, web, mastodon, etc servers. I prefer the configuration in kubenetes templates instead of spread all over my filesystem in /etc (config in one place and fucking unit files in another), /var/lib, /opt, etc.
  - lillecarl 3y ago
    I think you'd love NixOS!
    - sleepybrett 3y ago
      I think I work with kubernetes absolutely every day at work and it's doing just fine. I've used Nix for some builds but that's really it.
  - umvi 3y ago
    Right, you've acquired the taste. But why should people who haven't acquired the taste do this? To me seems like a mountain of unneeded complexity compared to just running a local webserver directly in a docker container with "docker run --restart always <image>".
    - sleepybrett 3y ago
      Turns out to run the stuff I run I need more than 1 machine :).. luckily a couple of years back when my job was dissolving a dept and had a stack of uneeded NUCs headed for recycling (pretty nice ones, i've added a little ram but that's it) I tossed them in a box and took them home instead. I've also added a couple of PI/Arm nodes just for kicks.
secondcoming 3y ago
How is K3s pronounced?
- gitpusher 3y ago
  from their docs: https://docs.k3s.io/ https://docs.k3s.io/ > We wanted an installation of Kubernetes that was half the size in terms of memory footprint. Kubernetes is a 10-letter word stylized as K8s. So something half as big as Kubernetes would be a 5-letter word stylized as K3s. There is no long form of K3s and no official pronunciation.
Difwif 3y ago
k3s users: I tried using k3s a few years back and on a fresh single node setup I was seeing ridiculous system usage with 0 deployments. Something like 20-30% cpu load. The same system with a k8s deployment using kubeadm was 3-5% without a deployment. There's this issue that goes back to 2019 that got closed without any real resolution. I moved on but I'm curious if anyone else has measured this? https://github.com/k3s-io/k3s/issues/294 https://github.com/k3s-io/k3s/issues/294
infogulch 3y ago
I've been eyeing Kairos [1] which is an OS lifecycle management system for k3s which looks like a nice way to deploy k3s. [1]: https://github.com/kairos-io/kairos https://github.com/kairos-io/kairos
selljamhere 3y ago
I had a great k3s experience with a silly, over engineered weekend project to automate my fog machine with motion sensors. I connected motion sensors to battery & wifi enabled RPis, built a remote circuit to control the fog machine, and ran a k3s cluster with NATS to bring it all together. 10/10 would do again. https://blog.apartment304.com/fog-machine-madness/ https://blog.apartment304.com/fog-machine-madness/
- ddejohn 3y ago
  No pictures of fog machine and spooky graveyard -- 4/10
  - selljamhere 3y ago
    Updated.
    - ddejohn 3y ago
      Awesome. 10/10.
feldrim 3y ago
If I remember correctly, the new F5 BIG IP instances also internally use k3s under the hood. That's probably it's efficient resource usage. But this is also an evidence, at least for me, on the stability of k3s.
iamdbtoo 3y ago
A great companion to k3s is k3sup. https://github.com/alexellis/k3sup https://github.com/alexellis/k3sup
koito17 3y ago
I've been running k3s on my home server and it's been painless to set up compared to other options (e.g. kubeadm) while also being very lightweight. For single-node setups, it defaults to Kine instead of etcd, using SQLite as the database. This removes a significant chunk of overhead for dev clusters and running in tiny devices. It also has Traefik set up with sane defaults, and the local path provisioner is also pretty good, too. But recently I've moved to Longhorn since I plan to eventually scale past 1 node. My only complaint about Longhorn is that applications that are write-heavy and delete old data (e.g. Prometheus with short retention) will require aggressive trimming (e.g. trim once a day) to keep the actual size of volumes down. Besides that, Longhorn makes backups to S3-compatible storage very effortless and you get RWX volumes, too! Regarding k3s itself, you can persist modifications to the way some components (e.g. Traefik) are installed through a HelmChartConfig CRD. This is what I personally use so I can use Traefik to route SSH traffic for Forgejo. Another nice thing is that although components like kube-proxy are baked in to the single k3s binary, you can still scrape metrics with Prometheus provided that you expose their endpoints somewhere on your cluster network.
mariuz 3y ago
K3s and hetzner related hetzner-k3s : This is a CLI tool to quickly create and manage Kubernetes clusters in Hetzner Cloud using the lightweight Kubernetes distribution k3s from Rancher. https://github.com/vitobotta/hetzner-k3s https://github.com/vitobotta/hetzner-k3s Kubernetes on Hetzner Cloud the easiest way https://vitobotta.com/2023/01/07/kubernetes-on-hetzner-cloud-the-easiest-way/ https://vitobotta.com/2023/01/07/kubernetes-on-hetzner-cloud...
mad_vill 3y ago
If you want to get an HA k3s cluster going quickly on your cloud-provider of choice: https://github.com/cluster-api-provider-k3s/cluster-api-k3s https://github.com/cluster-api-provider-k3s/cluster-api-k3s
ramdac 3y ago
how do you pronounce it? kuberthreeties?
lakomen 3y ago
Oh yeah I always wanted to ask someone who's in the deep with kubernetes. How do you migrate services and storage from one cluster to another?

254 points by kristianpaul 3y ago

ge96 3y ago

keees? kiots maybe

cudder 3y ago
Kubes?
- sesm 3y ago
  Kyuss
bantunes 3y ago
"Kes" ?
verdverm 3y ago
> We wanted an installation of Kubernetes that was half the size in terms of memory footprint. Kubernetes is a 10 letter word stylized as k8s. So something half as big as Kubernetes would be a 5 letter word stylized as K3s. There is neither a long-form of K3s nor official pronunciation.
- dheera 3y ago
  I always prefer to spell it kUbernEats.
- JohnMakin 3y ago
  So wouldn't it be K4s? I can't figure out how they got 3.
  - maxbond 3y ago
    They're counting the "k" and the "s" to get to 5 total. It's a bit more convoluted than 4 = 8 / 2, it's 3 = (8 + 2) / 2 - 2.
  - worksonmine 3y ago
    Cut the 8 in half and you get 3, not mathematically but visually.
    - ge96 3y ago
      KԐS
  - verdverm 3y ago
    len(kubernetes) == 10 -> k8s half(kubernetes) == 5 -> k3s
V99 3y ago
The real origin of the name is kubernetes -> k[8 letters]s -> k8s -> people pronounce as "kates" -> k[3 letters]s -> k3s.
deleted 3y ago
[deleted]

rmelton 3y ago

K3s is fantastic especially for local development with Kubernetes when orchestrated using k3d. This is what we use for most of our internal K8s testing at OpenC3.

doctoboggan 3y ago
Yes, I recently moved my Dev team to K3D as well. Previously we were using AWS clusters that were spun up on demand every time someone needed to test something. the time needed to do this grew and grew overtime eventually coming close to 30 to 40 minutes. With k3d testing locally can be done in a minute. Plus I learned a ton about k8s while doing this.

mpsprd 3y ago

Can this tool help to simplify self hosting implems? K3s was recommended to me to replace my personal pile of systemd units starting docker compose configs and manual reverse proxy configs. Im am completely oblivious to how k8s works.

proxysna 3y ago
Start with hashicorp nomad. Less of a learning curve and easy to set up. Traefik can do automatic discovery of services in nomad. Definitely better than a bunch of systemd units.
hhh 3y ago
K3s/k8s may help with that, but it's going to be a learning curve. I personally moved onto k8s for a similar reason, but it was a learning curve
- thelastparadise 3y ago
  It's a learning curve, but it is consistent, reliable, and standardized. Kubernetes has become an interface, independent of implementation. It's much like how POSIX is a standard and there are many implementations. And yes, POSIX has a learning curve too, but I'd rather learn that than anything proprietary, or non standard/rapidly changing.
  - moondev 3y ago
    No doubt. The ability to scale is great, but the best thing about Kubernetes is really the API.
dewey 3y ago
Check out https://kamal-deploy.org https://kamal-deploy.org, it just hit 1.0 and 37signals moved their whole Kubernetes stack to it. I was playing around with it recently for side projects and I think it's a nice fit for simpler products like that.
thelastparadise 3y ago

imachine1980_ 3y ago

Why use this over kubernetes, if it's the same but less memory why not only use k3s ans drop k8s

cogman10 3y ago
There's no real "over kubernetes". K3s is a kubernetes implementation. You have to think of k8s as more of a specification of what can be done with multiple implementations. k3s, kind, microk8s, minikube are all implementations of kubernetes that you might want to try out. k8s is much like the original IBM pc or a linux distribution. There are a bunch of interchangeable parts that you may land on depending on your needs. Why use this over something else? I've found k3s to be really simple to setup and upgrade, which is why I use it for my home lab.
- Takennickname 3y ago
  So all of those can be used to host a highly available multi physical node cluster?
  - cogman10 3y ago
    k3s and microk8s definitely support multiple node setup. I'm not sure about kind or minikube (Maybe? You'd have to look into the docs). How ready they are and how well they'll scale out will depend a lot on how the controllers are setup. k3s, at least, supports multiple controller nodes on setup. Where things really get squirrelly, at least with k3s, is storage and ingresses. k3s supports both but in a super robust fashion.
  - ianburrell 3y ago
    It makes sense to use full Kubernetes in that case. The lightweight distributions are good for single instances or limited multiple nodes. Like homelabs, edge nodes, or local development. But hosting application in data center makes sense to use full Kubernetes.
    - 3y ago

vault 3y ago

forget about it if you want to run it on an IPv6-only VPS, it's not ready yet

thelastparadise 3y ago
There's not any common reason I can think of to go ipv6 on the backend. KISS --if there's enough IPs in a private v4 range, use them. If you need/want to serve v6 clients, put a v6 load balancer in front. You're likely going to need that lb anyway.
- yjftsjthsd-h 3y ago
  I think the use case was running on a cheap server that actually doesn't have ipv4 connectivity
  - vault 3y ago
    correct
- dmm 3y ago
  > KISS ipv6 and routable ips is way simpler than patching layer 2 networks together.
cwayne 3y ago
K3s absolutely supports ipv6-only
- vault 3y ago
  this tutorial doesn't work on an IPv6-only VPS https://www.jeffgeerling.com/blog/2022/quick-hello-world-http-deployment-testing-k3s-and-traefik https://www.jeffgeerling.com/blog/2022/quick-hello-world-htt...

rcarmo 3y ago

If anyone wants a ready-to-go Azure template to play around, here you go: https://github.com/rcarmo/azure-k3s-cluster https://github.com/rcarmo/azure-k3s-cluster (I tweak this every now and then since I both used it as a training sample for my customers/peers and as a way to run my own batch processes as cheap as possible)

diggan 3y ago

"Lightweight Kubernetes" and then a graph involving 2 different nodes with 10+ services running on them. Nomad seems to be a truly "lightweight Kubernetes" and for the small amount of time I've been using it, it seems to do it's job really well and it's easy to understand all the moving pieces without spending countless of hours reading docs and source code. Although, it's hard to recommend Nomad for future uses as it sadly stopped being FOSS :/

q3k 3y ago
I mean, 2 nodes and 10 services is very light for what Kubernetes is designed to scale to.
worksonmine 3y ago
The amount of services running is not an indication of how lightweight the underlying tech is. K3S is for orchestration across nodes, and by design you can't run services on the master for security reasons unless you manually change it. Are you trying to say that 10 services and 2 nodes would be less on Nomad?
- tarruda 3y ago
  > K3S is for orchestration across nodes, and by design you can't run services on the master for security reasons unless you manually change it. That has not been my experience with k3s. Unlike normal distributions, k3s servers do allow workload to run on them unless you manually disable. From https://docs.k3s.io/datastore/ha-embedded https://docs.k3s.io/datastore/ha-embedded : " An HA K3s cluster with embedded etcd is composed of: - Three or more server nodes that will serve the Kubernetes API and run other control plane services, as well as host the embedded etcd datastore. - Optional: Zero or more agent nodes that are designated to run your apps and services " So you can have a full HA k3s cluster with only 3 server nodes and 0 agent(worker) nodes.
  - sleepybrett 3y ago
    This is not true of k8s in general, maybe certain on prem 'distributions' enforce this, i'm not sure. Sometimes kubernetes admins/operators who are running their own control planes and etcd will run etcd on non k8s controlled nodes, or they will run it on the same nodes that run their control plane services (scheduler, apiserver, controller manager, etc). Often those nodes will be tainted in such a way that they don't get overloaded by a noisy neighbor 'customer pod', but that's in no way 'by design'.. perhaps by convention or best practice.

inssein 3y ago

Loving K3S so far, I've got a "homelab" of 4 RPis running it and so far it has been pretty seamless experience.

rigelina 3y ago
This was my introduction to K3s as well. RPis running K3s with enough resources left over to actually do some small tasks. I hosted a small data pipeline that analyzed trading data from an MMO. It was as fun as it was impractical, and I learned quite a bit.

BossingAround 3y ago

Interestingly, SUSE now owns Rancher [0], so k3s has been backed by a large company for some time now. I've never tried k3s, but I have always thought it's probably the most loved-by-its-users version of Kubernetes. [0] https://www.suse.com/news/suse-completes-rancher-acquisition/ https://www.suse.com/news/suse-completes-rancher-acquisition...

mfer 3y ago
If you want to easily try it out, we baked it into Rancher Desktop [1]. RD let's you run Kubernetes (via k3s) and containers on your desktop. You pick the k8s version. [1] https://rancherdesktop.io/ https://rancherdesktop.io/ Disclosure, I work for SUSE and started Rancher Desktop.
- willaaam 3y ago
  I just made an account here to say thank you. Rancher desktop is amazing on my M2 mac, it's easy to use, solves a bunch of challenges from k8s to docker and has been surprisingly reliable for a young product.
- withinboredom 3y ago
  I've honestly been scared to switch from Docker Desktop to Rancher Desktop on Windows. Like do I have to switch or do they play nice together? Do I need to install Docker in WSL to use it, or kubectl, or is it like Docker Desktop where it "just works." So many questions...
  - mfer 3y ago
    You can have Docker Desktop and Rancher Desktop installed on the same Windows machine. Last I checked, you can't run them both at the same time due to docker socket locations (assuming you run Rancher Desktop using Moby). Rancher Desktop will provide the CLI tools to work with it. The idea is "it just works". It's a desktop app so it installs what you need.
- cortesoft 3y ago

FourSigma 3y ago

Could someone please explain the difference between K0s[1] and K3s? They seem to both target the same minimalist K8s segment. [1]https://k0sproject.io/ https://k0sproject.io/

r-bar 3y ago
K3S includes some extras that make it nice for working in small local clusters, but are not part of the standard k8s codebase. * Traefik daemonset as a load balancer * Helm controller that lets you apply helm manifests without the helm command line * Upgrade controller * Sqlite as the default backing store for the k8s API * Their own local storage provisioner K0S has a lot of the same goals: be light weight and self contained in a single binary. But K0S tries to be as vanilla as possible. Choosing between the two it comes down to your use case. Do you want light weight and compatible (k0s), or lightweight and convenient (k3s)? Edit: formatting
- FourSigma 3y ago
  Thank you! Great details. Definitely want convenient.
- nullify88 3y ago
  What you've listed for k3s is mostly included in k0s. I wouldnt go far to say k0s isnt convenient. * A helm controller is included in k0s * Etcd is bundled and bootstrapped automatically which I perfer because I dont want the overhead of the translation that Kine does. Although Kine is available for a non-etcd datastore if that is preferred. * Upgrade controller is included (autopilot). * They have a local storage provider based on OpenEBS. * Ingress is missing, but due to the built in helm controller that can be boot strapped upon cluster initialisation. Overall, together with k0sctl and its declarative configuration it is easier to deploy k0s than it was k3s.
  - pas 3y ago
    Can you please elaborate on the "kine" overhead?

slig 3y ago

Anyone running K3S on Hetzner's new ARM64 servers?

onionisafruit 3y ago

K3s' go.mod[0] is insane. [0] https://github.com/k3s-io/k3s/blob/master/go.mod https://github.com/k3s-io/k3s/blob/master/go.mod

sleepybrett 3y ago
Every kubernetes project's go.mod is quite robust. Kubernetes is a very large codebase.
pests 3y ago
My only worry is the 10+ domains (involving 4+ major companies) being used for import paths. I know there are solutions but that would be a mess to figure out 10 years from now when half those links are broken or changed subtly.

sidcool 3y ago

We have been using K3S in production now for 2 years. And it's working like a charm.

j_m_b 3y ago

Is the overhead of Kubernetes worth it enough to make it smaller, presumably to run in less resource intensive environments? Docker compose fits this bill nicely and is much more simple.

efrecon 3y ago
Compose is for single node setups. K3s runs in multi-nodes clusters.
- Xiol32 3y ago
  Also easier to scale out from your single starter k3s node as you grow.
speedgoose 3y ago
Kubernetes is offering much more features than Docker compose. And then you have all the kubernetes operators and other extensions that makes it very complex and powerful. It’s fine to use docker compose if it does what you need. If you start hacking weird things using Docker compose, it’s perhaps time to suffer and learn kubernetes.

betaby 3y ago

Too bad k3os was abandoned. And while k3s sounds easy, it's not after even a slightly larger scale. If one willing to have in-house k8s today I would recommend https://deckhouse.io/ https://deckhouse.io/ (I'm not affiliated with them)

yjftsjthsd-h 3y ago
> Too bad k3os was abandoned. Agreed:( Someday I'd like to see how hard it would be to use microos to build a replacement cheaply (cheap in terms of maintenance; ideally it would just be a combustion config or so that deployed k3s on vanilla microos) > And while k3s sounds easy, it's not after even a slightly larger scale. How does it fall apart?
- betaby 3y ago
  > How does it fall apart? Agent nodes failing to join after reboots or long network partitioning. Manual re-joining works. Additionally, hard to debug issues with the supplied Traefic.

dmd 3y ago

I believe Truenas SCALE's plugin system is built on k3s.

worldsayshi 3y ago

vCluster is an interesting related technology. It spins up k3s inside Kubernetes. Handy if you want a lightweight way to manage multiple staging environments. https://www.vcluster.com/docs/what-are-virtual-clusters https://www.vcluster.com/docs/what-are-virtual-clusters

einstand 3y ago

My favorite presentation about K3s is when Darren Shepherd explains all the black magic (=all the custom patches) which made it possible: https://www.youtube.com/watch?v=k58WnbKmjdA https://www.youtube.com/watch?v=k58WnbKmjdA

rcarmo 3y ago
this is highly recommended.

praveenhm 3y ago

I've been closely following the discussion on k3s and Kubernetes in general. I recently acquired an M1 Mac Ultra, and I'm curious about the best options available for running Kubernetes locally on it. Does anyone have experience or recommendations in this area? I've heard about a few tools but wanted to gather insights from this knowledgeable community.

maxekman 3y ago
You should try out OrbStack: https://docs.orbstack.dev/kubernetes/ https://docs.orbstack.dev/kubernetes/ I switched to it completely, it’s very convenient to have both fast (-est on Mac) Docker support and a really smooth VM setup for running occasional Linux tools (such as Yocto in my case). Edit: added some background info to my recommendation.
- praveenhm 3y ago
  Thanks for the orbstack recommendation, I am using it for docker containers, It is really fast and lightweight, I will try out Kubernetes.
deleted 3y ago
[deleted]
nunez 3y ago
If you're just messing around, just use kind (https://kind.sigs.k8s.io https://kind.sigs.k8s.io) or minikube if you want VMs (https://minikube.sigs.k8s.io https://minikube.sigs.k8s.io). Both work on ARM-based platforms. You can also use k3s; it's hella easy to get started with and it works great.
- sklarsa 3y ago
  If you go the kind route, be sure to not use Docker for mac and instead opt for podman which is much more resource efficient. Now that I've switched over to podman, my computer doesn't sound like it's about to blast off when I'm running clusters.

nunez 3y ago

There are lighter Kubernetes "distributions" (kind, minikube), but what makes k3s special is that it's (a) packaged as two binaries that provide all of the Kubernetes components in one, and (b) it's 100% suitable for production use. Lots of teams are using K3s to run Kubernetes at the edge and in IoT applications, and with good reason. It's a fantastic Kubernetes distribution that's well-maintained, easy to get going with and well-documented. (Ironically, if you look at the first commits to kubernetes/kubernetes, the Kubernetes components were originally shipped as a single binary. They decided to break them up later to simplify releasing, but the k3s monolith lives on.)

nurettin 3y ago

I wish kubernetes could install and manage services through systemd.

rcarmo 3y ago
Well, have a look a this: https://github.com/virtual-kubelet/systemk https://github.com/virtual-kubelet/systemk

intangible 3y ago

I've been using a 3 nuc (actually Ryzen devices) k3s on SuSE MicroOS https://microos.opensuse.org/ https://microos.opensuse.org/ for my homelab for a while, and I really like it. They made some really nice decisions on which parts of k8s to trim down and which Networking / LB / Ingress to use. The option to use sqlite in place of etcd on an even lighter single node setup makes it super interesting for even lighter weight homelab container environment setups. I even use it with Longhorn https://longhorn.io/ https://longhorn.io/ for shared block storage on the mini cluster. If anyone uses it with MicroOS, just make sure you switch to kured https://kured.dev/ https://kured.dev/ for the transactional-updates reboot method. I'd love to compare it against Talos https://www.talos.dev/ https://www.talos.dev/ but Talos's lack of support for a persistent storage partition (only separate storage device https://github.com/siderolabs/talos/issues/4041 https://github.com/siderolabs/talos/issues/4041 ) really hurts most small home / office usage I'd want to try.

organsnyder 3y ago
Funny: I've been running a Talos cluster for the past six months, and just today decided to look into k3s. Talos has a lot of really nice things, but I have found that the lack of shell access can be frustrating at times when trying to troubleshoot.
- davkan 3y ago
  Curious, what have you run into that you couldn’t troubleshoot with a privileged pod with host networking?
  - organsnyder 3y ago
    I had a situation where the etcd cluster got hosed, making it basically impossible (at least with the ways I know) to interact with the k8s API at all. So I didn't have any way to get a privileged pod running.
    - davkan 3y ago

samcat116 3y ago

K3s is great but I'll also shout out that RKE2 is almost as simple to install as K3s but its full Kubernetes.

maxthegeek1 3y ago

k3s is my preferred way to run k8s locally, for the purpose of developing helm charts locally. It's way more lightweight than minikube, and with k3d it's way easier to use as well.

bfrog 3y ago

Lightweight is a misnomer. Running even the most basic of tasks requires an insanely complex setup with vxlan, unintelligible iptables and bridge rules, all taking up a great deal of resources on something like an embedded x86/arm devices

nullify88 3y ago

If you want something lighter than k3s, k0s could be worth a try. I recently switched to k0s and noticed the controller, and the various kube processes consume less cpu resources compared to when I was running k3s.

umvi 3y ago

Kubernetes seems like an "acquired taste". Either you use it a lot (likely at work) and are therefore comfortable using it outside of work as your one-size-fits-all solution to everything, or else you don't use it a lot and see it as a complexity behemoth that hides all of your system behind esoteric CLIs and config files that just make everything a nightmare to manage.

sleepybrett 3y ago
I find it very acceptable on my local servers that run my mail, web, mastodon, etc servers. I prefer the configuration in kubenetes templates instead of spread all over my filesystem in /etc (config in one place and fucking unit files in another), /var/lib, /opt, etc.
- lillecarl 3y ago
  I think you'd love NixOS!
  - sleepybrett 3y ago
    I think I work with kubernetes absolutely every day at work and it's doing just fine. I've used Nix for some builds but that's really it.
- umvi 3y ago
  Right, you've acquired the taste. But why should people who haven't acquired the taste do this? To me seems like a mountain of unneeded complexity compared to just running a local webserver directly in a docker container with "docker run --restart always <image>".
  - sleepybrett 3y ago
    Turns out to run the stuff I run I need more than 1 machine :).. luckily a couple of years back when my job was dissolving a dept and had a stack of uneeded NUCs headed for recycling (pretty nice ones, i've added a little ram but that's it) I tossed them in a box and took them home instead. I've also added a couple of PI/Arm nodes just for kicks.

secondcoming 3y ago

How is K3s pronounced?

gitpusher 3y ago
from their docs: https://docs.k3s.io/ https://docs.k3s.io/ > We wanted an installation of Kubernetes that was half the size in terms of memory footprint. Kubernetes is a 10-letter word stylized as K8s. So something half as big as Kubernetes would be a 5-letter word stylized as K3s. There is no long form of K3s and no official pronunciation.

Difwif 3y ago

k3s users: I tried using k3s a few years back and on a fresh single node setup I was seeing ridiculous system usage with 0 deployments. Something like 20-30% cpu load. The same system with a k8s deployment using kubeadm was 3-5% without a deployment. There's this issue that goes back to 2019 that got closed without any real resolution. I moved on but I'm curious if anyone else has measured this? https://github.com/k3s-io/k3s/issues/294 https://github.com/k3s-io/k3s/issues/294

infogulch 3y ago

I've been eyeing Kairos [1] which is an OS lifecycle management system for k3s which looks like a nice way to deploy k3s. [1]: https://github.com/kairos-io/kairos https://github.com/kairos-io/kairos

selljamhere 3y ago

I had a great k3s experience with a silly, over engineered weekend project to automate my fog machine with motion sensors. I connected motion sensors to battery & wifi enabled RPis, built a remote circuit to control the fog machine, and ran a k3s cluster with NATS to bring it all together. 10/10 would do again. https://blog.apartment304.com/fog-machine-madness/ https://blog.apartment304.com/fog-machine-madness/

ddejohn 3y ago
No pictures of fog machine and spooky graveyard -- 4/10
- selljamhere 3y ago
  Updated.
  - ddejohn 3y ago
    Awesome. 10/10.

feldrim 3y ago

If I remember correctly, the new F5 BIG IP instances also internally use k3s under the hood. That's probably it's efficient resource usage. But this is also an evidence, at least for me, on the stability of k3s.

iamdbtoo 3y ago

A great companion to k3s is k3sup. https://github.com/alexellis/k3sup https://github.com/alexellis/k3sup

koito17 3y ago

I've been running k3s on my home server and it's been painless to set up compared to other options (e.g. kubeadm) while also being very lightweight. For single-node setups, it defaults to Kine instead of etcd, using SQLite as the database. This removes a significant chunk of overhead for dev clusters and running in tiny devices. It also has Traefik set up with sane defaults, and the local path provisioner is also pretty good, too. But recently I've moved to Longhorn since I plan to eventually scale past 1 node. My only complaint about Longhorn is that applications that are write-heavy and delete old data (e.g. Prometheus with short retention) will require aggressive trimming (e.g. trim once a day) to keep the actual size of volumes down. Besides that, Longhorn makes backups to S3-compatible storage very effortless and you get RWX volumes, too! Regarding k3s itself, you can persist modifications to the way some components (e.g. Traefik) are installed through a HelmChartConfig CRD. This is what I personally use so I can use Traefik to route SSH traffic for Forgejo. Another nice thing is that although components like kube-proxy are baked in to the single k3s binary, you can still scrape metrics with Prometheus provided that you expose their endpoints somewhere on your cluster network.

mariuz 3y ago

K3s and hetzner related hetzner-k3s : This is a CLI tool to quickly create and manage Kubernetes clusters in Hetzner Cloud using the lightweight Kubernetes distribution k3s from Rancher. https://github.com/vitobotta/hetzner-k3s https://github.com/vitobotta/hetzner-k3s Kubernetes on Hetzner Cloud the easiest way https://vitobotta.com/2023/01/07/kubernetes-on-hetzner-cloud-the-easiest-way/ https://vitobotta.com/2023/01/07/kubernetes-on-hetzner-cloud...

mad_vill 3y ago

If you want to get an HA k3s cluster going quickly on your cloud-provider of choice: https://github.com/cluster-api-provider-k3s/cluster-api-k3s https://github.com/cluster-api-provider-k3s/cluster-api-k3s

ramdac 3y ago

how do you pronounce it? kuberthreeties?

lakomen 3y ago

Oh yeah I always wanted to ask someone who's in the deep with kubernetes. How do you migrate services and storage from one cluster to another?